Setting up audit logging in a secured Hadoop cluster

To enable security event monitoring and auditing in Hadoop, we need to enable the logging framework to write the detailed audit trails in the logfile. Enabling detailed audit logs needs careful planning. These logs could grow very fast if there are continuous exceptions and could fill up the disk space. There should be a system monitoring this log growth and taking corrective actions such as cleaning and compressing. This can be done by configuring the Log4j.properties file in the Hadoop configuration directory. By default, the Hadoop security and audit logfile appenders are set to Null appenders and hence, disabled. This needs to be modified to reflect the correct logfile location for audit and security logs. We also need to enable the capture of the authentication logs from the local KDC.

Configuring Hadoop audit logs

The following configuration shows the audit and security logging configurations that need to be done on all the nodes...