In this post, a number of basic profiling tools that are included in most Linux systems by default will be discussed. The tools introduced in this post, vmstat, sar, iostat, and mpstat, are relatively simple but provide basic data which can be very useful when analyzing a system’s performance. For most examples we will be using a CentOS 8 machine.
vmstat: Virtual memory statistics
One of the most useful tools when troubleshooting memory-related performance issues is vmstat. The vmstat command is part of the procps-ng package, which includes other useful performance analysis commands such as free and top.
The vmstat command, if given no arguments, will print out the averages of various system statistics since boot. The vmstat command accepts two arguments. The first is the delay, and the second is the count. The delay is a value in seconds between output. The count is the number of iterations of statistics to report. If no count is given, vmstat will continuously report statistics.
The memory statistics are reported in KiB by default. The —S option allows for changing this to report in KB, MB, or MiB with -S k, -S m, and -S M, respectively.
The first line of vmstat output is always an average since boot, so when gathering metrics, remove the first line and use the subsequent lines. The first two lines of output are header information, and the third is the uptime average. The following example shows output of vmstat starting from the fourth line of output:
# vmstat 10 | tail -n +4 1 0 0 479748 1044 570784 0 0 0 5 90 132 0 0 100 0 0 0 0 0 475952 1044 570912 0 0 0 0 81 130 0 0 100 0 0 0 0 0 479456 1044 570856 0 0 0 52 87 137 0 0 100 0 0 ...
Overview of vmstat Columns
vmstat examples
1. The vmstat command, if given no arguments, will print out the averages of various system statistics since boot:
# vmstat procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 0 424444 1044 615940 0 0 89 14 58 78 1 0 98 1 0
2. Here is an example where vmstat will output statistics every 10 seconds until interrupted:
# vmstat 10 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 0 426440 1044 615872 0 0 88 14 58 78 1 0 98 1 0 0 0 0 426072 1044 615872 0 0 0 1 92 137 0 0 100 0 0 0 0 0 426072 1044 615872 0 0 0 5 74 116 0 0 100 0 0 ...
3. If you wanted vmstat to exit after printing four reports, add a count argument:
# vmstat 10 4 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 2 0 0 425236 1044 616252 0 0 86 13 58 78 1 0 98 1 0 0 0 0 424936 1044 616252 0 0 0 1 85 129 0 0 100 0 0 0 0 0 424876 1044 616252 0 0 0 15 70 111 0 0 100 0 0 0 0 0 423360 1044 616252 0 0 0 1 89 130 0 0 100 0 0
sar: the system activity reporter
The sar command is a multipurpose analysis tool that is part of the sysstat package. It works in two modes. It can read data collected by a cron job every 10 minutes, or it can be used to collect instantaneous data about the state of the system.
Make sure sar is installed before you begin. If its not installed, install it using yum:
# yum install sysstat
Once installed, configure sar to collect data at 5 minutes interval:
# vim /etc/cron.d/sysstat # Run system activity accounting tool every 10 minutes */5 * * * * root /usr/lib64/sa/sa1 1 1
For best results when using sar, make sure to set a locale with a LANG environment variable that provides 24-hour time support. For instance, if sorting sar data by the first column (the time column), the en_US.UTF-8 locale will place 01:00:00 PM before 02:00:00 AM. This is likely to skew the data points and make a graph worthless. The c locale is one which meets this sorting requirement. When dealing with sar data, override the locale like this:
# LANG=C sar -q Linux 4.18.0-193.28.1.el8_2.x86_64 (2713138b141c.mylabserver.com) 02/09/21 _x86_64_ (2 CPU) 00:00:01 runq-sz plist-sz ldavg-l ldavg-S 1davg-15 blocked 00:10:01 0 317 0.20 0.17 0.18 0 00:20:01 0 317 0.00 0.07 0.13 0 ... Output Truncated ...
In fact, an alias may simply be created for sar. Add the following line to /etc/bashrc or ~/.bashrc:
alias sar='LANG=C sar'
sar can be used to read one of the log files (or any sar data file in general) by using the -f option.
# sar -q -f /var/log/sa/sa09 Linux 4.18.0-193.28.1.el8_2.x86_64 (2713138b141c.mylabserver.com) 02/09/2021 _x86_64_ (2 CPU) 00:00:01 runq-sz plist-sz ldavg-l ldavg-S 1davg-15 blocked 00:10:01 0 317 0.20 0.17 0.18 0 00:20:01 0 317 0.00 0.07 0.13 0 ... Output Truncated ...
Some configuration can be done in the /etc/sysconfig/sysstat file. For instance, change the HISTORY variable to set the number of days archives are kept for sar.
sar examples
1. Create a system-wide alias for sar that forces sar to report using 24-hour time. Source this into the current shell.
# echo "alias sar='LANG=c sar'" >> /etc/bashrc # source /etc/bashrc
2. Keep 60 days of archives instead of 28. Change the HISTORY variable in /etc/sysconfig/sysstat.
# vim /etc/sysconfig/sysstat HISTORY=60
3. Report the I/O and transfer rate statistics:
# sar -b Linux 5.4.10-x86_64-linode132 (mylabserver.com) 02/09/2021 _x86_64_ (1 CPU) 12:00:02 AM tps rtps wtps bread/s bwrtn/s 12:10:01 AM 38.16 31.03 7.14 1655.33 390.20 12:20:01 AM 12.11 5.81 6.30 280.99 239.08 12:30:01 AM 8.40 2.34 6.06 105.99 229.95 12:40:01 AM 7.38 1.62 5.76 23.48 211.41 12:50:01 AM 8.69 2.64 6.04 125.33 230.58 01:00:01 AM 8.09 2.26 5.83 77.72 227.70 01:10:02 AM 12.25 6.10 6.15 215.14 235.73 ... Average: 12.81 6.41 6.40 343.31 255.49
4. Report the utilization of CPU0:
# sar -P 0 Linux 5.4.10-x86_64-linode132 (web.thegeekdiary.com) 02/09/2021 _x86_64_ (1 CPU) 12:00:02 AM CPU %user %nice %system %iowait %steal %idle 12:10:01 AM 0 11.69 0.00 2.96 0.28 0.06 85.01 12:20:01 AM 0 9.87 0.00 2.53 0.12 0.04 87.44 12:30:01 AM 0 9.96 0.00 2.48 0.09 0.04 87.43 12:40:01 AM 0 9.59 0.00 2.43 0.06 0.04 87.88 12:50:01 AM 0 9.80 0.00 2.46 0.09 0.05 87.59 ... Average: 0 10.00 0.02 2.60 0.14 0.15 87.08
5. Report the network device statistics from the current log file.
# sar -n DEV Linux 5.4.10-x86_64-linode132 (mylabserver.com) 02/09/2021 _x86_64_ (1 CPU) 12:00:02 AM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 12:10:01 AM dummy0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:10:01 AM tunl0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:10:01 AM ip_vti0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:10:01 AM ip6gre0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:10:01 AM teql0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:10:01 AM sit0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 ... Average: dummy0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: tunl0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: ip_vti0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: ip6gre0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: teql0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: sit0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 ...
6. Configure an additional cron job to gather power usage data in addition to the disk data. Modify the following to the /etc/sysconfig/sysstat file:
SADC_OPTIONS="-S POWER"
iostat and mpstat
Like sar, iostat and mpstat are also part of the sysstat package. Both iostat and mpstat support the delay and count arguments just like the vmstat command. iostat reports CPU and I/O statistics for devices. partitions, and network file systems (NFS).
The mpstat command reports CPU-related statistics. Like sar, it may be necessary to configure the LANG for 24-hour time.
iostat and mpstat examples
1. Show extended output for /dev/sda.
# iostat -x sda
Linux 5.4.10-x86_64-linode132 (mylabserver.com) 02/09/2021 _x86_64_ (1 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
8.12 0.08 2.85 0.09 0.24 88.63
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 1.76 7.13 3.55 4.17 122.57 93.88 56.07 0.00 0.83 0.71 0.93 0.88 0.68 2. Show CPU statistics for CPU0:
# LANG=C mpstat -P 0 Linux 5.4.10-x86_64-linode132 (mywebserver.com) 02/09/21 _x86_64_ (1 CPU) 06:32:13 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 06:32:13 0 8.12 0.08 2.37 0.09 0.22 0.25 0.24 0.00 0.00 88.63