Reading logs and troubleshooting your setup (Simple)

Logfiles can contain useful information for troubleshooting. For example, you might see a directive that is not correctly set up, a syntax error, or even a communications error. Then it is time to review the configuration file, correct the errors, and restart/start the service.

Getting started

From the moment you install Debian your system will start recording logs. They are sitting on /var/log, are rotated daily (if needed) and compressed, and you should back them up. There are system-level and application-level logs. You can even enable additional verbosity and/or debugging information by searching the configuration files for log directives. It is important that you familiarize yourself with the contents of /var/log and the files that each application and/or service generates.

How to do it…

Familiarize yourself with grep and less (sudo apt-get install less), which enable you to:

• Add your user to the adm group by running sudo usermod –a –G adm devops. The adm group will allow you to read several logfiles under the /var/log folder, thus reducing the need to use sudo for any subsequent commands.

• Search for a particular term in a file (grep –i 'disk is full' /var/log/syslog)

• Then use tail to monitor additions to the file, tail –f /var/log/bacula/bacula.log.

• The following are default logfiles that can be useful for a web server:

  • Any error 500 should be logged in /var/log/apache2/error.log

  • /var/log/mysql.err and /var/log/mysql.log contains MySQL logs and /var/log/postgresql/postgresql-9.1-main.log contains Postgres'

  • /var/log/syslog is very useful for networking logs as well as for OOMs (out of memory) errors

There's more…

Debian uses logrotate to control periodic rotation of logs. After some days of running your server, you will notice that /var/log starts filling up with files ending in .0 and .gz. These are rotated logfiles that are archived based on time or size rules. You can move them to your backup, and .gz files can also be searched by using zgrep instead of grep.

Applications such as logcheck (sudo apt-get install logcheck) can help monitor for log anomalies. You may also want to use remote syslog servers or logging applications such as Prelude that centralize logs for several servers and help you determine the timelines of events.

Regarding timelines, it is also important to have synchronized clocks in all your servers. This is something that is easily achievable by installing the ntp daemon, sudo apt-get install ntp, or by using a manual tool such as ntpdate-debian, available in the ntpdate package.