There are many tools/utilities that can be used to analyze the current system performance. But how do we gauge the historic system performance? For that you can use the System Activity Report (SAR) tool. Using the sar tool, you will be able to look back over a period of time and see how the server has been running.
In this post we will see how to install and use the sysstat tools; thus, allowing you to examine historical system statistics.
Installing sysstat
First thing first, install the “sysstat” which provides the sar utility.
# apt-get install sysstat ### Debian-based distributions # yum install sysstat ### RedHat Based distribution
The historic data for various linux distribution is stored in below directories:
1. Red Hat, Fedora, CentOS, and Scientific Linux should use the /var/log/sa directory
2. Debian, Mint, and Ubuntu users should use the /var/log/sysstat directory
By default, sar stats are collected every 10 minutes. The data is collected using a simple cron job configured within /etc/cron.d/sysstat. This job can be amended to collect the data as frequently as you require. Example /etc/cron.d/sysstat file is provided below.
# cat /etc/cron.d/sysstat # Run system activity accounting tool every 10 minutes */10 * * * * root /usr/lib64/sa/sa1 1 1 # 0 * * * * root /usr/lib64/sa/sa1 600 6 & # Generate a daily summary of process accounting at 23:53 53 23 * * * root /usr/lib64/sa/sa2 -A
Gathering historical sar data
1. Getting live sar data is easy with sar. By default sar operates in CPU mode. Simply using the command as named, we will receive CPU activity samples for every 10 minutes of the current day.
# sar Linux 2.6.32-504.el6.x86_64 (geeklab) 09/28/2018 _x86_64_ (16 CPU) 12:00:01 AM CPU %user %nice %system %iowait %steal %idle 12:10:01 AM all 7.28 0.00 2.94 0.02 0.00 89.77 12:20:01 AM all 7.16 0.00 2.47 0.02 0.00 90.35 12:30:01 AM all 6.96 0.00 2.43 0.01 0.00 90.59 12:40:01 AM all 6.92 0.00 2.42 0.01 0.00 90.64 ...
2. Lets suppose you want to analyse the system performance at some past time. For example view system performance between 10 A.M. to 12 A.M. when application team faced a slowness in their app.
# sar -r -s 10:00:00 -e 12:00:00
sar uses the -s arguement to specify the start time of a data extract and another (-e) to set the end time. These parameters must be written in HH:MM:SS format, or sar will ignore them with an error. The -r argument is used to display memory usage data.
3. sar historic data is stored in /var/log/sa directory in case of RedHat based distributions. Files are simply binary formats containing sar data for each retained date. Files are prefixed with sa. Thus, sa23 is the sar data for the 23rd of the month. For example, on a CenOS/RHEL system.
# ls /var/log/sa sa01 sa04 sa07 sa10 sa13 sa16 sa19 sa22 sa25 sa28 sar01 sar04 sar07 sar10 sar13 sar16 sar19 sar22 sar25 sar30 sa02 sa05 sa08 sa11 sa14 sa17 sa20 sa23 sa26 sa30 sar02 sar05 sar08 sar11 sar14 sar17 sar20 sar23 sar26 sar31 sa03 sa06 sa09 sa12 sa15 sa18 sa21 sa24 sa27 sa31 sar03 sar06 sar09 sar12 sar15 sar18 sar21 sar24 sar27
Execute the following command to view past I/O statistics for the 10th of the month:
# sar -f /var/log/sysstat/sa10 -b
Changing sar historical data retention
One can change the historical sar data retention using the file /etc/sysconfig/sysstat or /etc/sysstat/sysstat. Change the HISTORY parameter setting to the desired amount of days to retain data. For example, to keep 28 days of records, we could use this:
# cat /etc/sysconfig/sysstat # sysstat-9.0.4 configuration file. # How long to keep log files (in days). # If value is greater than 28, then log files are kept in # multiple directories, one for each month. HISTORY=28 # Compress (using gzip or bzip2) sa and sar files older than (in days): COMPRESSAFTER=31 # Parameters for the system activity data collector (see sadc manual page) # which are used for the generation of log files. SADC_OPTIONS="-S DISK" # Compression program to use. ZIP="bzip2"