Following is the list of OS commands apart from basic commands that are useful in diagnosing the problems at OS end causing the slow performance. Please note that all the commands mentioned above are tested in a Linux environment only.
If you are ready to track down your performance problems, here are some tools to get you started:
- sar: With the sysstat package installed, use the sar command to display activity information for a range of system features or to get a sense of resource usage over time. Once installed, sysstat cron jobs gather system data every 10 minutes that can be displayed later with the sar command. With sar, you can also display the same data types live.
- dstat: Created as a replacement for vmstat, iostat, and ifstat, dstat offers many options for combining and displaying different types of system data. It shows data in color and lets you present different pieces of information in the columns you choose. With dstat, you can see statistics about your system’s CPU, disk I/O, network activity, paging, interrupts, load averages, memory, processes, time counters, and file systems. Use dstat to compare live system activity for different types of resources. For example, during large file downloads, you can compare network and disk activity.
- ps: The ps command is the common way of seeing which processes are running. Combined with options, ps lets you choose which columns to display for each process and sort data by any of those columns. Use ps to see information about all or any processes currently running on your system.
- top: With top, you can watch a screen-oriented display of processes running on your system that is updated every few seconds. You can sort by CPU or memory usage and kill or renice processes. Use top to immediately find your most consumptive processes.
- iostat: Primarily, iostat lets you display I/O statistics for devices (such as disks and network interfaces). This is helpful for checking for I/O bottlenecks because data is displayed by the percentage of the device’s read/write capacity.
- mpstat: The mpstat command lists CPU usage. Use mpstat to see how much CPU is being consumed and to get a general idea of where that consumption is coming from.
- vmstat: List memory statistics with the vmstat command. If performance is poor, use vmstat to see if memory is exhausted and if the system is swapping excessively.
- iptraf: For a graphical tool to monitor network traffic, run the iptraf command. Use iptraf to see which specific IP addresses are generating the most network traffic to and from your system.
- tcpdump: This text-based tool lets you watch packets hitting selected network interfaces and optionally filter those packets to look for certain types of activity. Use tcpdump to track down performance problems related to specific services—you can choose to view only packets destined for a particular service.
- wireshark: This tool is a graphical version of tcpdump, with many analytic features. The tool includes support for decoding many commonly used protocols and displaying them in an interactive graphical user interface.
Now, let’s dig deeper into a few of the tools we just listed out.
1. “iotop” command output for Linux environment.
The iotop command is a top-like utility for disk I/O. It watches I/O usage information output by the Linux kernel (requires v2.6.20 or later) and displays a table of current I/O usage by processes or threads on the system. This requires the root user to run the command.
2. To collect the filesystem cache information using “free” command output to get the filesystem cache usage:
- “free” command to check for memory usage.
- “Buffers” represent how much portion of RAM is dedicated to cache disk block. “Cached” is similar like “Buffers”, only this time it caches pages from file reading.
- A buffer is something that has yet to be “written” to disk.
- A cache is something that has been “read” from the disk and stored for later use.
3. To check the list of OS patches applied in Linux using below command. This will help to identify the changes CT did in past:
# rpm -qa --last > last-rpms.txt
4. To collect the info about Memory/CPU/IO/NUMA etc for any changes from dmesg output. Can be collected every 30 seconds:
– To display hardware information related to Ethernet port eth0:
$ dmesg | grep -i eth0
– To display total memory available and shared memory details:
$ dmesg | grep -i Memory
– To display tty information:
$ dmesg | grep -i tty
– To display NUMA info:
$ dmesg | grep -i numa
– To display hard disks info:
$ dmesg | grep -i sda
– To display CPU related info:
$ dmesg | grep -i CPU
5. To collect global system messages log:
/var/log/messages /var/log/dmesg
The dmesg command shows the current content of the kernel syslog ring buffer messages while the /var/log/dmesg file contains what was in that ring buffer when the boot process last completed. /var/log/dmesg stays unchanged until next reboot. Old messages are replaced by new messages in the ring buffer.
dmesg is the subset of /var/log/messages and is maintained in ring buffer. /var/log/messages include all the system messages including from starting of the system along with the messages in dmesg. In a nutshell logs from dmesg are dumped in /var/log/messages.
Common Linux log files names and usage:
- /var/log/messages : General message and system related stuff
- /var/log/auth.log : Authenication logs
- /var/log/kern.log : Kernel logs
- /var/log/cron.log : Crond logs (cron job)
- /var/log/maillog : Mail server logs
- /var/log/qmail/ : Qmail log directory (more files inside this directory)
- /var/log/httpd/ : Apache access and error logs directory
- /var/log/lighttpd/ : Lighttpd access and error logs directory
- /var/log/boot.log : System boot log
- /var/log/mysqld.log : MySQL database server log file
- /var/log/secure or /var/log/auth.log : Authentication log
- /var/log/utmp or /var/log/wtmp : Login records file
- /var/log/yum.log : Yum command log file.
6. To check whether hugepages are used. To execute only at start and end of OSW collection:
# grep -i Hugepages /proc/meminfo # cat /proc/meminfo
7. To check the filesystem used at OS. To be executed only once:
# cat /etc/fstab
8. To collect the info about system activity for the last week:
/var/log/sa has files for sa and sar which has the CPU usage history information.
# ls -al /var/log/sa | grep "Mar 12" -rw-r--r--. 1 root root 721996 Mar 12 23:50 sa12 -rw-r--r--. 1 root root 655607 Mar 12 23:53 sar12
“sar -W -f /var/log/sa/sa12” – shows paging/swapping historical info
# less /var/log/sa/sa12
9. To get the CPU Arch information:
# lscpu # cat /proc/cpuinfo
10. To get the kernel shared memory/semaphores info:
# ipcs -lm # ipcs -ls # cat /proc/sys/kernel/shmmax # cat /proc/sys/kernel/shmmni getconf PAGE_SIZE # cat /proc/sys/kernel/shmall
11. To get info about File handles:
# cat /proc/sys/fs/file-max # cat /proc/sys/fs/file-nr
12. To get info about ulimit for Open file descriptors/ maxproc:
# su - oracle # ulimit -n # ulimit -u
13. To check the maximum IO size per IO request:
# cat /sys/block/[device]/queue/max_sectors_kb
To determine the maximum I/O size in bytes when Asynch IO is used:
# cat /proc/sys/fs/aio-max-size
or
# cat /proc/sys/fs/aio-max-nr
14. To show the disk device model, standards, configuration, capacity etc:
Example:
# hdparm -I /dev/sda # sudo hdparm -i /dev/sda
15. To check the block size for particular device:
Example:
# blockdev --getbsz /dev/xvda1 # tune2fs -l /dev/vgxx/lvolx