Why do we need log management?

The log is the first place to check on how your system is behaving. We generally enable logging in our application to debug issues related to application, performance, and even security issues. Now the challenge is that each log has its own format. For example, the format of /var/log/messages that store system messages is completely different from /var/log/secure, which stores all security-related information. Traditionally, we use tools such as grep, sed, and awk to parse information inside these files, but that is not fool proof or scalable solution. In the cloud, where we could be dealing with a terabyte or even petabyte of data, these tools will not work because of its inherent limitation of dealing with a limited set of data. To overcome these challenges and attain effective log management, AWS provides its own set of tools to push data and analyze it, which is the topic of this chapter. We will look at some of the solutions provided by AWS, such...