MustGather:
Performance, hang, or high CPU issues with WebSphere Application Server on AIX
If you are experiencing performance, hang, or high CPU issues with WebSphere Application Server on AIX, this MustGather will assist you in collecting the data necessary to diagnose and resolve the issue.
To improve the accuracy of complete data collection, IBM recommends you use the automated data collectors
within IBM Support Assistant. Not only will the automated collector
gather the equivalent of the manual process, it will also provide a
secure file transfer of the collection to IBM.
Collecting data using the IBM Support Assistant Data Collector
Go to the ISA Data Collector site for WebSphere Application Server.
- Start the ISA Data Collector:
- To run online, select the option to collect from this system using the current browser and click Start Collection.
- To run locally, select the option to collect from this or another system using a downloadable utility. Download and extract the zip file to your WAS_HOME directory. From a command line, run isadc.[sh|bat] or launch index.html to use the web interface.
- Select the JDK > Hang / High CPU / Performance Problem collector and click Start.
- Follow the prompts to automatically submit the collected data to IBM Support
Collecting data using the WAIT Data Collector
This data collection method should only be used with WebSphere Application Server 6.1 and later.
If you have not already done so, enable verboseGC and restart the server.
At the time of the problem, run the script linked below with the following command:
If you have not already done so, enable verboseGC and restart the server.
At the time of the problem, run the script linked below with the following command:
- ./perfMustGather.sh [PID]
This command will create a file named waitData.tar.gz which will contain all of the diagnostic data needed, server logs and javacores included. This script should be ?executed as the root user?. As with any script, you may need to add execute permissions before executing the script (chmod).?
In the above command, [PID] is the Process ID of the problematic JVM(s). If specifying multiple Process IDs they should each be separated by a space.?
Download link: hwrps://wait.researchlabs.ibm.com/wait/public/perfMustGather.sh
Send the waitData.tar.gz output file to IBM Support: "Exchanging information with IBM Support"
Collecting data manually
Complete the following three steps:
(1) Collecting the required data:
If you have not already done so, enable verboseGC and restart the server.
At the time of the problem, run one of the awrached scripts with the appropriate command:
These scripts will create a file named aixperf_RESULTS.tar.gz and three javacores. These scripts should be executed as the root user. As with any script, you may need to add execute permissions before executing the script (chmod).
In the above commands, [PID] is the Process ID of the problematic JVM(s). If specifying multiple Process IDs they should each be separated by a space.
(2) Collecting log files:
Collect the server logs (SystemOut.log, native_stderr.log,...) from the problematic server(s):
<profile_root> /logs/ server_name /*
(3) Submiwring required data:
Zip/Tar all the files gathered:
Send the results to IBM Support: "Exchanging information with IBM Support"
Frequently Asked Questions:
If asked to do so:
The preceding data is used to troubleshoot most of these type of issues; however, in certain situations Support may require additional data. Only collect the following data if asked to do so by IBM Support.
(1) Collecting the required data:
If you have not already done so, enable verboseGC and restart the server.
At the time of the problem, run one of the awrached scripts with the appropriate command:
- For WebSphere Application Server 6.1 and later:
./aixperf.sh [PID]
For WebSphere Application Server6.0 and earlier:
./aixperf60.sh [PID]
Note: Both aixperf60.sh and dbxtrace_aix.sh should be in the same directory and dbx must be installed to diagnose high CPU issues.
These scripts will create a file named aixperf_RESULTS.tar.gz and three javacores. These scripts should be executed as the root user. As with any script, you may need to add execute permissions before executing the script (chmod).
In the above commands, [PID] is the Process ID of the problematic JVM(s). If specifying multiple Process IDs they should each be separated by a space.
(2) Collecting log files:
Collect the server logs (SystemOut.log, native_stderr.log,...) from the problematic server(s):
<profile_root> /logs/ server_name /*
(3) Submiwring required data:
Zip/Tar all the files gathered:
- aixperf_RESULTS.tar.gz
- three javacores
- server logs (SystemOut.log, native_stderr.log,...)
Send the results to IBM Support: "Exchanging information with IBM Support"
Frequently Asked Questions:
- What is the impact of enabling verboseGC?
VerboseGC data is critical to diagnosing these issues. This can be enabled on production systems because it has a negligible impact on performance (< 2%).
What is the aixperf_RESULTS.tar.gz file and where can I find it?
The aixperf_RESULTS.tar.gz file is created while running the aixperf.sh script and contains output from the commands called by the script. It will be created in the directory from which you execute the script.
What are 'javacores' and where do I find them?
Javacores are snapshots of the JVM activity and are essential to troubleshooting these issues. These files will usually be found in the <profile_root>. If you do not find the files here, you can search your entire system for them using the following command:
find / -name "*javacore*"
If asked to do so:
The preceding data is used to troubleshoot most of these type of issues; however, in certain situations Support may require additional data. Only collect the following data if asked to do so by IBM Support.
- System core
- You can collect a system core by executing the following command:
gencore [PID] core.[PID]
Process the system core file using the instructions in How to process a core dump using jextract on the IBM SDK on Windows, Linux, and AIX
Monitor process sizes and paging usage
- The aixmon.sh script will collect data every 5 minutes until it is stopped manually. Run the following command before the issue occurs to start the script:
./aixmon.sh
This will create two files: ps_mon.out and vmstat_mon.out.
Exchanging data with IBM Support
To diagnose or identify a problem, it is sometimes necessary to provide
Technical Support with data and information from your system. In
addition, Technical Support might also need to provide you with tools or
utilities to be used in problem determination. You can submit files
using one of following methods to help speed problem diagnosis:
- IBM Support Assistant (ISA)
- Service Request (SR)
- FTP to the Enhanced Customer Data Repository (ECuRep)
No comments:
Post a Comment