Exploring auditing
In most environments where you run Kubernetes clusters, you will need to have an auditing system in place. While Kubernetes has some auditing features, they are often too limited for an enterprise to rely on for a complete audit trail, and logs are often only stored on each host filesystem.
In order to correlate events, you are required to pull all the logs you want to search through on your local system, and manually look through logs or pull them into a spreadsheet and attempt to create some macros to search and tie information together.
Fortunately, there are many third-party logging systems available for Kubernetes. Optional pay systems such as Splunk and Datadog are popular solutions and open source systems including the EFK stack are commonly used and included with many Kubernetes distributions. All of these systems include some form of a log forwarder that allows you to centralize your Kubernetes logs so you can create alerts, custom queries, and dashboards...
