The system log daemon is responsible for logging the system messages generated by applications or kernel. The system log daemon also supports the remote logging. The messages are differentiated by facility and priority. In principle, the logs handled by syslog are available in the /var/log/ directory on Linux system :
# ls /var/log acpid cron.1 maillog.3 rpmpkgs.3 spooler.3 anaconda.log cron.2 maillog.4 rpmpkgs.4 spooler.4 anaconda.syslog cron.3 messages sa squid anaconda.xlog cron.4 messages.1 samba tallylog audit cups messages.2 scrollkeeper.log vbox boot.log dmesg messages.3 secure wtmp boot.log.1 faillog messages.4 secure.1 Xorg.0.log boot.log.2 gdm oracle-validated secure.2 Xorg.0.log.old boot.log.3 httpd pm secure.3 YaST2 boot.log.4 lastlog ppp secure.4 yum.log btmp mail prelink setroubleshoot conman maillog rpmpkgs spooler conman.old maillog.1 rpmpkgs.1 spooler.1 cron maillog.2 rpmpkgs.2 spooler.2
where some of the logs are dumped under a subdirectory like cups, samba, httpd. Among the logs under /var/log the /var/log/messages is the most common one as the kernel / core system logs are held there. The kernel modules generally dumps there too. So, for problem diagnosis / monitoring the /var/log/messages is the primary log file to examine.
The system log daemon/service and it’s configuration file differs depending on the version of Linux used i.e.:
RHEL 5: syslogd - /etc/syslog.conf RHEL 6: rsyslogd - /etc/rsyslog.conf
Rsyslog
Rsyslog is the new logging daemon starting RHEL6 to compete with the old syslog-ng daemon. Few of the benefits rsyslog daemon provides over syslog-ng are :
1. Reliable Networking
– Rsyslog uses TCP instead of UDP which is more reliable. TCP uses the acknowledgment and retransmission capabilities.
– with Rsyslog daemon you can specify multiple destination hosts/files for messages delivery if rsyslogd is unable to deliver a message to aprticular destination.
2. Precision
– it is possible to filter messages on any part of log message rather than the priority of the message and the original facility.
– support for precise timestamps to log messages that the syslog daemon.
3. Other features
– TLS encryption
– ability to log to SQL databases.
rsyslog.conf
The configuration file – /etc/rsyslog.conf for the rsyslogd daemon is used to handle all the messages. The configuration file basically provides rules statements which in turn provides 2 things :
– what messages to match.
– selector consists of a facility and priority separated by a dot (.)(e.g. mail.info)
2. actions
– what to do with matched messages
– usually a destination to log the message (file on local machine or a remote host)
Selectors and actions
Selectors are made up of 2 things facilities and priorities. They specify which messages to match. The action field specifies what action to apply to the matched message. For Example :
kern.debug /var/log/kernlog
– The messages with with a facility of kernel and priority debug are logged into the file /var/log/kernlog.
– Priority statements are hierarchical in selectors. Rsyslog matches all the messages with specified priority and higher. So all the messages from kernel with priority debug and higher are logged. Debug being the lowest priority all the messages with facility kern are matched.
– Another way to do this is to use the asterisk (*). For example :
kern.* /var/log/kernlog
– multiple selectors can be specified on a single line separated by semicolons. This is useful when same action needs to be applied to multiple messages.
– when a file is listed in action field, the matched messages are written into the file.
– There can be other devices such as FIFO, terminal etc to write the messages to.
– If a username is listed in action field, the matched messages are printed to the users all the terminals if they are logged in.
– (*) in the action field specifies
Facilities
The facility is used to specify which type of program or application is generating the message. Thus enabling the syslog daemon to handle different sources differently. The table below lists the standard facilities and their description :
| Facility | Description |
|---|---|
| auth/authpriv | security/authorization messages (private) |
| cron | clock daemon (crond and atd messages) |
| daemon | messages from system daemons without separate facility |
| kern | kernel messages |
| local0 – local7 | reserved for local use |
| lpr | line printer subsystem |
| messages from mail daemons | |
| news | USENET news subsystem |
| syslog | messages generated internally by system log daemon |
| user | generic user-level messages |
| uucp | UUCP subsystem |
Priority
The priority of a message signifies the importance of that message. Table below lists the standard priorities and their meanings :
| Priority | Description |
|---|---|
| emerg | system is unusable |
| alert | action must be taken immediately |
| crit | critical conditions |
| err | error conditions |
| warning | warning conditions |
| notice | normal but significant importance |
| info | informational messages |
| debug | debugging messages |
Log Rotation
Log files grow regularly overtime and thus they needs to be trimmed regularly. Linux provides a utility to provide this functionality without user intervention. The logrotate program can be used to automate the log file rotation. The basic logrotate configuration is done in the configuration file /etc/logrotate.conf. In the configuration file we can set options such as – how frequently logs should be rotated and how many old logs to be kept.
# cat /etc/logrotate.conf
weekly
rotate 4
create
include /etc/logrotate.d
/var/log/wtmp {
monthly
minsize 1M
create 0664 root utmp
rotate 1
} As per the above logrotate configuration file the logs are rotated every week (renaming the existing log to filename.number order):
minsize 1M – logrotate runs and trims the messages files if the file size is equal to or greater than 1 MB.
rotate 4 – keep the most recent 4 files while rotating.
create – create new file while rotating with specified permission and ownership.
include – include the files mentioned here for the daemon specific log rotation settings.
# ls -l /var/log/messages* -rw------- 1 root root 1973 Jun 10 15:07 /var/log/messages -rw------- 1 root root 10866 Jun 6 04:02 /var/log/messages.1 -rw------- 1 root root 19931 May 30 04:02 /var/log/messages.2 -rw------- 1 root root 238772 May 23 04:02 /var/log/messages.3 -rw------- 1 root root 171450 May 14 18:29 /var/log/messages.4
– The logrotate daemon mainly reads all the configuration from file /etc/logrotate.conf and then includes daemon specific configuration files from /etc/logrotate.d/ directory.
– The logrotate daemon along with rotation and removal of old logs, allows compression of log files.
– The daemon runs daily from /etc/cron.daily/logrotate.
Logwatch
– RHEL systems are also shipped with logwatch packages.
– Logwatch is used to analyze the logs to identify any interesting messages.
– Logwatch can configured to analyze logfiles from popular services and email administrator the results.
– It can be configured on hourly or nightly basis for any suspicious activity. By default in a RHEL system, it is run on nightly basis and report is mailed to root user.