Tricks for parsing different files

In this section, we are going to learn about the tricks to use to parse different files. Before we start with the actual parsing, we must read the data first. You need to understand where you will be getting all the data from. But, you must also remember all the log files come in different sizes. To make your task simpler, here is a list to follow:

• Remember the log files can be either plain text or compressed.
• All the Log files have a .log extension for a plain text file and log.bz2 for a bzip2 file.
• You should process the set of files based on their name.
• All the parsing of log files must be combined into a single report.
• The tool you are using must operate on all files, from a specified directory or from different directories. Log files from all sub-directories should also be included.