< back to blog

Linux Troubleshooting Cheatsheet: strace, htop, lsof, tcpdump, iftop & sysdig

Phil Rzewski
Phil Rzewski
@
Linux Troubleshooting Cheatsheet: strace, htop, lsof, tcpdump, iftop & sysdig
Published:
April 13, 2016
Table of contents
This is the block containing the component that will be injected inside the Rich Text. You can hide this block if you want.
This is the block containing the component that will be injected inside the Rich Text. You can hide this block if you want.

This Sysdig cheatsheet is a great guide of command-lines linux admins can use to get insights into their servers. Whether you've been an admin for one month or 20 years you've definitely used one if not all of these tools to troubleshoot an issue. Because we love Sysdig (naturally!) we also included a translation for each of these common operations into the sysdig command line or csysdig.Rather than attempt covering all options from manpages (which would have made for boring coverage of many esoteric, rarely-used switches), we've started from examples referenced at the most popular web pages you'd find when you search for terms like "strace examples", "htop examples", and so forth.Do you have favorites that aren't listed here? Let us know and we'll include them in future articles.

strace

There's one subtle difference between strace and sysdig that will be apparent in many of these side-by-side comparisons: Many of the simplest strace examples include command-lines that are executed and traced as a "one-shot" operation. On the other hand, Sysdig has a somewhat different philosophy, in that it either watches live events from afar as they happen, or analyzes capture data previously saved to a file. Thankfully, Sysdig's rich filtering options provide the knobs to watch for specific one-shot executions, as you'll soon see.

OperationstracesysdigNote
Trace the execution of a commandstrace who sysdig proc.name=who Whereas strace runs the who command shown here as a one-shot, Sysdig is watching for the execution of who. Use Sysdig's filtering to further isolate a specific run, e.g.:sysdig proc.name=who and proc.ppid=534This watches for a who that's about to be run in a shell that you've determined to have PID of 534.
Trace only when certain/specific system calls are madestrace -e open whostrace -e trace=open,read whosysdig evt.type=open and proc.name=whosysdig "evt.type in (open,read) and proc.name=who"
Save a trace to a filestrace -o output.txt who sysdig -w output.scap proc.name=who With strace, the file produced contains the same text you'd have viewed on the screen if run interactively. With Sysdig, you get a raw, re-usable capture file, such that you can view the text output with:sysdig -r output.scapYou could also use this as the basis to apply filters or any other Sysdig functionality you want to apply as you revisit the original events.
Watch a running process with PID=1363strace -p 1363 sysdig proc.pid=1363
Print a timestamp for each output line of the tracestrace -t who sysdig proc.name=who Sysdig prints timestamps by default.
Print relative time for system callsstrace -r who sysdig -tD proc.name=who Sysdig offers several more ways to represent timestamps via the -t option.
Generate batch statistics reports of system callsstrace -c who sysdig -w output.scap proc.name=who
# Now run the "who" separately
For one-shot batch text reports:sysdig -r output.scap -c topscalls -c topscalls_timeOr for an interactive report that allows for further drill-down:csysdig -r output.scap -v syscalls
Sysdig's default behavior is more optimized for the case of presenting event data as it happens rather than "batch" reporting. This is why the Sysdig equivalent is done in two steps here.
Generate live, per-second statistics reports of system calls for running process with PID=1363N/Acsysdig -v syscalls proc.pid=1363While strace can show individual events as they happen live, or provide a single batch report for the execution of a command, csysdig's views provide a unique ability to show live, periodic reports

htop

Since htop is a live, interactive, curses-style tool, we'll compare it to the live, interactive, curses-style csysdig.For starters, both tools use the same approach of navigating the live table via Up/Down/Left/Right arrows and also PgUp/PgDn. For operations that affect a single process (killing, renicing, etc.) it is assumed you've used these controls to first highlight a particular process.

OperationhtopcsysdigNote
Change sort order based on a column of the tablePress F6, <, or > and then select a column by name, orPress M, P, or Tto sort by Memory, Processor Usage, or TimePress I to invert the sort orderPress F9 or > and then select a column by name, orPress shift<1-9> to sort by any columnn, and press repeatedly to invert sort order, orMouse-click on a column header
Kill a processPressF9 or kPressk
Renice a processPress F7 or ] to reduce the nice value by 1Press F8 or [ to increase the nice value by 1Press ] to reduce the nice value by 1Press [ to increase the nice value by 1This illustrates how easy it is to customize Sysdig. I noticed when first writing this article that csysdig was missing a couple minor features like this, so I used the opportunity to learn how easy it is to write/modify Chisels, then put up my improvements as a Pull Request. You can do the same!
Display only processes started by a user named "phil"Press u, thenSelect the user name phil from the listLaunch as:csysdig user.name=philOr mouse-click Filter: from within csysdig at the top of the default Processes view, then append and user.name=phil to the current filter text
Change the output refresh interval to once every 5 secondsLaunch as:htop -d 50Launch as:csysdig -d 5000As you can see, htop works in units of tenths-of-a-second, while csysdig works in milliseconds.
Start a system call trace on a processPress s to start an stracePress F6 to start a sysdig
List open files for a processPress l to run a one-time lsofPress f to run a one-time lsof Or to see real-time, updating reports of files/directories used by a process, drill down to a specific process by pressing Enter, then press F2 and select a View such as Files, File Opens List, or Directories.See the Note above for "Renice a process" about how the one-time lsof was recently added as an enhancement.
Follow a process, such that it remains highlighted even as its order in the list changesPress FDefault behavior is to always follow the highlighted process

lsof

OperationlsofcsysdigNote
List all open files belonging to all active processeslsofsysdig -c lsof
List processes that have opened the specific file /var/log/sysloglsof /var/log/syslogsysdig -c lsof "fd.name=/var/log/syslog"
List processes that have opened files under the directory /var/loglsof +d /var/logsysdig -c lsof "fd.directory=/var/log"
List files opened by processes named "sshd"lsof -c sshdsysdig -c lsof "proc.name=sshd"
List files opened by a specific user named "phil"lsof -u philsysdig -c lsof "user.name=phil"
List files opened by everyone except for the user named "phil"lsof -u ^philsysdig -c lsof "user.name!=phil"
List all open files for a specific process with PID=1081lsof -p 1081sysdig -c lsof "proc.pid=1081"
List all files opened by user "phil" or a process named "sshd" (OR logic)lsof -u phil -c sshdsysdig -c lsof "'user.name=phil or proc.name=sshd'"Note the use of two layers of quotes with the Sysdig filter.
List all files opened by an "sshd" process for user "phil" (AND logic)lsof -u phil -c sshd -asysdig -c lsof "'user.name=phil and proc.name=sshd'"Note the use of two layers of quotes with the Sysdig filter.
Observe repeating reports of open files based on live activityEnable repeat mode with one of:lsof -rlsof +r Similar live data can be obtained with a live/interactive csysdig view, launched like so:csysdig -v filescsysdig -v file_opens
List all network connectionslsof -isysdig -c lsof "fd.type=ipv4"
List network connections in use by a specific process with PID=1014lsof -i -a -p 1014sysdig -c lsof "'fd.type=ipv4 and proc.pid=1014'"Note the use of two layers of quotes with the Sysdig filter.
List processes that are listening on port 22lsof -i :22sysdig -c lsof "'fd.port=22 and fd.is_server=true'"Note the use of two layers of quotes with the Sysdig filter.
List all TCP or UDP connectionslsof -i tcplsof -i udpsysdig -c lsof "fd.l4proto=tcp"sysdig -c lsof "fd.l4proto=udp"

tcpdump

tcpdump is focused entirely on network traffic, while network traffic is only a subset of what Sysdig covers. Many tcpdump use cases involve filtering, and tcpdump uses network-specific BPF filters, whereas Sysdig uses its own broader Sysdig filtering. The two approaches look similar in many ways, but you'll want to look at the docs for each side-by-side as you progress to more advanced filtering needs. Also, since in Linux everything is a file, you'll notice the Sysdig filtering examples below all leverage a "network-connections-via-file-descriptors" approach.

OperationtcpdumpcsysdigNote
Capture packets from a particular interface eth0 (192.168.10.119)tcpdump -i eth0sysdig fd.ip=192.168.10.119Sysdig does not currently have filtering based on named interfaces, but the equivalent via IP address is shown here.
Capture only 100 packetstcpdump -c 100sysdig -n 100 fd.type=ipv4
Display captured packets in ASCIItcpdump -Asysdig -A fd.type=ipv4
Display captured packets in HEX and ASCIItcpdump -XXsysdig -X fd.type=ipv4
Capture packet data, writing it into into a filetcpdump -w saved.pcapsysdig -w saved.scap fd.type=ipv4The Sysdig file format is capable of holding event data for much more than just network packets (e.g. system calls).
Read back saved packet data from a filetcpdump -r saved.pcapsysdig -r saved.scap
Capture only packets longer/smaller than 1024 bytestcpdump greater 1024tcpdump less 1024sysdig "fd.type=ipv4 and evt.buflen > 1024"sysdig "fd.type=ipv4 and evt.buflen < 1024"The greater/less options in tcpdump reference overall packet length whereas evt.buflen in Sysdig is relative to payload size.
Capture only UDP or TCP packetstcpdump udptcpdump tcpsysdig fd.l4proto=udpsysdig fd.l4proto=tcpNote that we don't need to explicitly include fd.type=ipv4 since we're using other network-only filters here.
Capture only packets going to/from a particular porttcpdump port 22sysdig fd.port=22Note that we don't need to explicitly include fd.type=ipv4 since we're using other network-only filters here.
Capture packets for a particular destination IP and porttcpdump dst 54.165.81.189 and port 6666sysdig fd.rip=54.165.81.189 and fd.port=6666Note that we don't need to explicitly include fd.type=ipv4 since we're using other network-only filters here.

iftop

Since iftop is a live, interactive, curses-style tool, we'll compare it to the live, interactive, curses-style csysdig. Also, like tcpdump, iftop uses BPF filters. See the previous intro to the section on tcpdump for more detail about filtering differences.

OperationiftopcsysdigNote
Display a table of current bandwidth usage between pairs of hostsiftopLaunch as:csysdig -v connectionsOr press F2 from within csysdig to change the View, then up-arrow to select ConnectionsBy default iftop watches just the first interface it finds, whereas by default csysdig watches traffic across the entire host.
Turn on display of network portsLaunch as:iftop -POr press p from within iftopDefault behavior is to always display ports
Observe traffic for just the eth0 interface (192.168.10.119)Launch as:iftop -i eth0Launch as:csysdig -v connections fd.ip=192.168.10.119Or mouse-click on Filter: from within csysdig, then append and fd.ip=192.168.10.119 to the existing filter textsysdig/csysdig do not currently have filtering based on named interfaces, but the equivalent via IP address is shown here.
Resolve DNS namesPress n from within iftop to toggle resolution for all hosts shownPress n from within csysdig to run nslookup on the currently-highlighted remote host
Change sort order based on a column of the tablePress to sort by destinationPress F9 or > and then select a column by name, orPress shift <1-9> to sort by any columnn, and press repeatedly to invert sort order, orMouse-click on a column header
Filter to show only traffic going to/from IP address 54.84.222.1Launch as:iftop -f "host 54.84.222.1"Launch as:csysdig -v connections fd.ip=54.84.222.1Or mouse-click on Filter: from within csysdig, then append and fd.ip=54.84.22.1 to the existing filter text
Pause the displayPress PPress p
Scroll the displayPress j to scroll upPress k to scroll downPress Up/Down/Left/Right arrows or PgUp/PgDn to scroll through the tablesysdig/csysdig go well beyond scrolling through a single-table, since you can drill down into the Connections View to see data in other groupings such as per-container or per-thread.

A linux troubleshooting cheatsheet: strace, htop, lsof, tcpdump, iftop & sysdig https://t.co/XeIeAwwj9i

— Sysdig (@sysdig)April 14, 2016

Sysdig CheatSheet Acknowledgements

The author would like to acknowledge thegeekstuff.com, as most of the example-filled articles used for the table above were found at their site.Would you like a downloadable PDF version of this cheatsheet?Grab it here.

About the author

No items found.
featured resources

Test drive the right way to defend the cloud
with a security expert