Security Incident and Event Monitoring in a Hadoop Cluster
A Security Incident and Event Monitoring (SIEM) system is responsible for collecting, monitoring, analyzing, and generating various security alerts for any suspicious activity in the cluster. SIEM systems usually collect the various system logs, network logs, and application logs to identify these security incidents and events. Hadoop itself can be used to perform the analysis and correlation of these security events in a batch mode.
The first step in any SIEM system is to collect the various system logs and identify corresponding events. The following are the events that need to be monitored in a Hadoop cluster to detect any security incidents:
User login and authorization events: User login events in a secured Hadoop cluster are generated when the end users or service principals authenticate themselves within the KDC or EIM system.
krb5kdc.logfor the KDC in the local Hadoop realm will contain the service login events. The central...
