Reading log files and affecting the output

Viewing or reading a log file is very easy and depending on your personal preferences, the basic syntax to view any of these files can be expressed in any of the following formats:

# less /var/log/filename
# more /var/log/filename
# cat /var/log/filename
# cat /var/log/filename | less
# cat /var/log/filename | more

Note

Remember, depending on the system configuration, you may need root privileges to view a specific log file. The same can be said when you are attempting to make changes to any system files, and for this reason, we will continue as the root user. However, those who use sudo or su (switch user) should change the instructions accordingly.

Log files can vary between applications and services, but the general purpose of these files is to record the time and date of an event and the security level, and to provide a message or general description. Most messages will be general notices or warnings of one type or another, but on certain occasions, errors will also be trapped.

For example, you may see something like this:

Dec  4 12:49:05 localhost postfix/postfix-script[1909]: starting the Postfix mail system

Messages like this are quite ordinary and merely explain what is happening and when it happened. Yes, you can safely ignore them, but due to the number of messages you see, some may remark or feel that the system is acting a little oversensitive to the extent that a log file is being flooded with low-level information. This information may serve no real purpose to many, but in some circumstances, you may consider that the information supplied isn't sensitive enough, and more information is needed. In the end, only you can decide what best suits your needs. So, in order to take a case in point, let's increase log sensitivity for the purpose of troubleshooting the system.

To do this, we will begin by running the following command:

# cat /proc/sys/kernel/printk

The output of the preceding command enables you to view the current settings for the kernel, which, on a typical system, will look like this:

4       4       1       7

There is a relationship at work here, and it is important that you understand that printk maintains four numeric values that control a number of settings related to the logging of error messages, while every error message in turn maintains its very own log level in order to define the importance of that message.

The log level values can be summarized in the following way:

• 0: Kernel emergency
• 1: Kernel alert; action must be taken immediately
• 2: Condition of the kernel is considered critical
• 3: General kernel, error condition
• 4: General kernel, warning condition
• 5: Kernel notice of a normal but significant condition
• 6: Kernel informational message
• 7: Kernel debug-level messages

So, based on the above information, the log level values of 4, 4, 1, and 7 tell us that the following is now apparent:

• The first value (4) is called the console log level. This numeric value defines the lowest priority of any message printed to the console, thereby implying that the lower the priority, the higher the log level number.
• The second value (4) determines the default log level for all messages that do not maintain an exclusive log level.
• The third value (1) determines the lowest possible log level configuration for the overall console log level. The lower the priority, the higher the log level number.
• The fourth and final value (7) determines the default value for the overall console log level. Again, the lower the priority, the higher the log level number.

Consequently, you are now in a position to consider making changes to the log level through a configuration file found at /etc/sysctl.conf. This file enables you to make fine adjustments to default settings, and it can be accessed with your favorite text editor in the following manner:

# nano /etc/sysctl.conf

To make the required change use the following syntax:

kernel.printk = X X X X

Here, the actual value of X is a log level setting taken from the options described earlier.

For example, you can change the number of messages by adding the following line:

kernel.printk = 5 4 1 7

Of course, such a modification implies a change to the kernel, and for this reason a reboot would be warranted. So, having done this, you will find that the output of running cat /proc/sys/kernel/printk should now reflect the new values. However, and as a supplementary note of caution, having considered doing this (and yes, you can easily reverse any changes made), it is important to realize that there are many questions based on the validity of changing these settings. Look at it this way: it may not help you at all, so you should always read around the subject before making these changes in order to confirm that making this alteration will suit your general purposes.

To view the onboard manual, simply use the following command:

$ man sysctl

On the other hand, for the many other services and applications on your server, you will have additional avenues of investigation to consider and these are generally set by the service or application in question.

A common example of this is Apache. So, if you are debugging a web-based issue related to this service, you may be inclined to open the httpd configuration file like this:

# nano /etc/httpd/conf/httpd.conf

Look or search for the following line:

LogLevel warn

Then, replace the instruction with a more appropriate setting (before saving the file and restarting the service). In this case, you can use:

LogLevel debug

Fortunately, it is nice to know that most services and applications do support a form of debugging mode for an improved log output. This will make the log file much more descriptive and easier to work with when troubleshooting the server, but just before we leave this subject, here comes the small print…

When you are working with log files, you should be aware that the information contained within those log files will not always be enough to help you diagnose the issue at hand or discover the cause of a problem. Log files may not only lack the required information, but they can also contain unknown errors and misleading messages. After all, log files only contain a series of (mainly) predefined messages or break points in a package, and these messages have been designed by programmers to make a remark concerning a known event that could have, or has taken place.

Note

Remember…

When affecting the output of a log file, a verbose and detailed output may raise performance or security issues, while detailed logging can also place an undue burden on the CPU or disk I/O operations.

Based on these circumstances, there are no hard and fast rules because we also know that log files have limitations. So, in the end you will rely on a keen eye for detail and a great deal of patience, and for these reasons alone, you must always learn to "listen to the server" as a whole.

Let's put it this way: the answer is there, but it may not be in the log files. Perseverance and a calm (but firm) hand will win the day, and it is this point of contention that will be echoed throughout the pages of this book.