The strace Command
The strace command can be used to intercept and record the system calls made, and the signals received by a process. This allows examination of the boundary layer between the user and kernel space which can be very useful for identifying why a process is failing.
Using strace to analyze how a program interacts with the system is especially useful when the source code is not readily available. In
addition to its importance in troubleshooting, strace can provide deep insight into how the system operates. Any user may trace their own running processes; additionally, the root user may trace any running processes. For example, the following could be used to attach to and trace the running rsyslogd daemon:
# strace -p $(pgrep rsyslogd)
Process 819 attached
select(1, NULL, NULL, NULL, {83009, 275934}
...
strace Output
The output from strace will correspond to either a system call or signal. The output from a system call is comprised of three components:
1. The system call
2. Any arguments surrounded by parenthesis
3. The result of the call following an equal sign
An exit status of -1 usually indicates an error. For example:
# strace ls file1
execve("/bin/ls", ["ls", "file1"], [/* 21 vars */]) = 0
brk(0) = 0xadb000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f516bb79000
.....
close(1) = 0
munmap(0x7f516bb78000, 4096) = 0
close(2) = 0
exit_group(0) = ?
+++ exited with 0 +++
Curly braces are used to indicate dereferenced C structures. Square braces are used to indicate simple pointers or an array of values.
Examples of strace command
Redirecting trace to a file
Since strace often creates a large amount of output, it’s often convenient to redirect it to a file. For example, the following could be used to launch the bash shell, trace any forked child processes, and record all file access to the files.trace file:
# strace -f -o files.trace -e trace=file bash
Counting number of sys calls
Run the ls command counting the number of times each system call was made and print totals showing the number and time spent in each call (useful for basic profiling or bottleneck isolation):
# strace -c ls
Viewing files opened by a process/daemon
The following example shows the three config files that OpenSSH’s sshd reads as it starts. Note that strace sends its output to STDERR by default, so if you want to pipe it to other commands like grep for further altering you must redirect the output appropriately:
# strace -f -eopen /usr/sbin/sshd 2>&1 | grep ssh
Tracing only network related system calls
Trace just the network related system calls as Netcat attempts to connect to a local telnetd service:
# strace -e trace=network nc localhost 23
The ltrace Command
The ltrace command can be used to intercept and record the dynamic calls made to shared libraries. The amount of output generated by the ltrace command can be overwhelming for some commands (especially if the -S option is used to also show system calls). You can focus the output to just the interaction between the program and some list of libraries. For example, to execute the id -Z command and show the calls made to the libselinux.so module, execute:
$ ltrace -l /lib/libselinux.so.1 id -Z
is_selinux_enabled(0xc1c7a0, 0x9f291e8, 0xc1affc, 0, -1)a
=1
getcon(0x804c2c8, 0xfee80ff4, 0x804b179, 0x804c020, 0)a
=0
user_u:system_r:unconfined_t
Remember that you can see what libraries a program is linked against using the ldd command.