Collect the following information when high CPU consumption is with IBM Java process:
Enable garbage collection trace to see whether Java garbage collection is thrashing if possible. If you want to enable Java garbage collection trace on IBM WebSphere Application Server, please refer to the following document: Enabling verbose garbage collection (verbosegc) in WebSphere application Server
Run the following command:
top -d delaytime -c -b > top.log
Where delaytime is the number of seconds to delay. This must be 60 seconds or greater, depending on how soon the failure is expected.
Create a script file, vmstat.sh with the following content:
#vmstat.sh
#output file name
VMSTAT_LOG=$1
LIMIT=288
#sleep for 5 miniutes
SLEEP_TIME=300
while true
do
i=0
echo >$VMSTAT_LOG
while [ $i -le "$LIMIT" ];
do
date >> $VMSTAT_LOG;
vmstat 5 12 >> $VMSTAT_LOG;
i=`expr $i + 1`;
sleep $SLEEP_TIME;
done
done
Create a script, ps.sh with the following content:
#ps.sh
#output file name
PS_LOG=$1
LIMIT=288
#sleep for 5 miniutes
SLEEP_TIME=300
while true
do
i=0
echo >$PS_LOG
while [ $i -le "$LIMIT" ];
do
date >> $PS_LOG;
ps -eLf >> $PS_LOG;
i=`expr $i + 1`;
sleep $SLEEP_TIME;
done
done
Run the scripts:
./ps.sh ps_eLf.log
./vmstat.sh vmstat.log
Notes: . The scripts ps.sh and vmstat.sh, as provided, roll over every 24 hours. . You might need to modify the scripts to meet your needs. . The preceding scripts will run forever. After the error condition is reached, you will have to terminate them.
When high CPU consumption occurs, collect the following logs:
netstat -an > netstat1.out
If the Web server is remote, run the following on the Web server system:
netstat -an > netstatwebserver1.out
Run the following:
kill -3 [PID_of_problem_JVM]
The kill -3 commands create javacore*.txt files
Note: If you are not able to determine which JVM process is experiencing the high CPU usage then you should issue the kill -3 PID for each of the JVM processes.
Wait two minutes.
Run the following:
kill -3 [PID_of_problem_JVM]
Wait two minutes.
Run the following:
kill -3 [PID_of_problem_JVM]
Wait two minutes.
Run the following:
netstat -an > netstat2.out
If the Web server is remote, run the following on the Web server system:
netstat -an > netstatwebserver2.out
If you are unable to generate javacore files, then perform the following:
kill -11 [PID_of_problem_JVM]
WARNING: kill -11 will terminate the JVM process, produce a core file, and possibly a javacore.
Review all output files and collect the following files for IBM Performance Analysis Tool for Java for Linux
ps_eLf.log
javacore*.txt files
Courtesy:http://wasissues.blogspot.com/