This document contains useful commands and code snippets for troubleshooting (Java) server applications in production.
The following assumptions are made.
- The
$pidvariable contains the process ID (PID) to the Java process in question $JAVA_HOMEis set to a JDK installation- You are using Oracle's HotSpot JVM
- You are logged in as the same user as the Java process you are troubleshooting is running as
- You are on a Linux machine (most stuff works on *NIX but is untested)
The commands are generally safe to run in production unless otherwise stated.
$JAVA_HOME/bin/jps -mlvV
Performance impact is usually minimal, although all threads must reach a safepoint to be able to dump
$JAVA_HOME/bin/jstack $pid
Take a thread dump every 5 seconds and save to a timestamped file
while true;do echo -n ".";$JAVA_HOME/bin/jstack $pid > /tmp/dump_${pid}_$(date '+%H_%M_%S').txt; sleep 5;done
Take a thread dump every 5 seconds whenever the CPU usage on the whole machine is over 50%.
while true
do echo -n .
if [ "$(mpstat 1 1|grep '^Average' |awk '{print $3}'|cut -d '.' -f 1)" -gt "50" ];then
echo dumping
$JAVA_HOME/bin/jstack $pid > /tmp/dump_${pid}_$(date '+%H_%M_%S').txt
sleep 5
fi
done
Show heap usage by generation and GC duration in seconds accumulated since the JVM started.
$JAVA_HOME/bin/jstat -gc $pid 3s
If you prefer percentages instead of bytes:
$JAVA_HOME/bin/jstat -gcutil $pid 3s
Print the cause of the current and the last GC.
$JAVA_HOME/bin/jstat -gccause $pid 3s
Show a class histogram with class names, number of instances and total size on the heap.
$JAVA_HOME/bin/jmap -histo $pid
Show only live objects (NB: this triggers a full GC):
$JAVA_HOME/bin/jmap -histo:live $pid
Tip: This is a way to explicitly trigger a full GC even though explicit GC has been disabled on the target JVM.
Generate a full heap dump (NB: this will freeze the JVM for the full duration).
$JAVA_HOME/bin/jmap -dump:format=b,file=/tmp/heap_dump_$pid.bin $pid
Java threads are mapped to native threads on Linux. Use top -H to show threads in top.
top -p $pid -H
The PIDs of the threads can be converted to hexadecimal and matched to the nid= entries in a thread dump. The following command can be used to convert PIDs to hexadecimal on the fly.
top -p $pid -H -d 3 -b | sed -r 's/^ *([0-9]+) (.*)/printf "%x \2" \1/e'
Top is used in batch mode which makes it append the output to the console.
Dump both thread stacks and CPU usage by thread with PIDs hex-converted to the same file.
export fn=/tmp/dump_${pid}_$(date +"%H_%M_%S"); $JAVA_HOME/bin/jstack $pid > $fn && top -p $pid -H -d 2 -b -n 2 | sed -r 's/^ *([0-9]+) (.*)/printf "%x \2" \1/e' >> $fn
Show number of total compile tasks and information about the most recent task (size, type and method).
$JAVA_HOME/bin/jstat -printcompilation $pid 3s
$JAVA_HOME/bin/jcmd $pid PerfCounter.print
Uses a lowercase c (Perfcounter.print) on older JVMs.
$JAVA_HOME/bin/jcmd $pid VM.system_properties
$JAVA_HOME/bin/jcmd $pid VM.flags
Show all listen ports:
ss -ln
Show which process is listening on port 8989 (requires root):
ss -lnp "sport = :8989"
Show 4 packets as ascii that is sent or received on port 10003 on interface bond0.
tcpdump -A -p -c 4 -i bond0 'port 10003'
tcpdump is not ubiquitous.
Network utilization on interface bond0.
nethogs bond0
nethogs is not ubiquitous.
iostat -x 2
lsof -p $pid
df -h
du -hs <path>
free -h
top -ba -n 1
tail -n0 -F <file> |grep --line-buffered <regex> | <cmd>
ps -eL|wc -l