Troubleshooting a service running in production

It is a recommended best practice to create minimal images for production that don’t contain anything that is not absolutely needed. This includes common tools that are usually used to debug and troubleshoot an application, such as netcat, iostat, ip, and others. Ideally, a production system only has container orchestration software such as Kubernetes installed on a cluster node with a minimal OS, such as CoreOS. The application container in turn ideally only contains the binaries absolutely necessary to run. This minimizes the attack surface and the risk of having to deal with vulnerabilities. Furthermore, a small image has the advantage of being downloaded quickly, using less space on disk and in memory, and showing faster startup times.

But this can be a problem if one of the application services running on our Kubernetes cluster shows unexpected behavior and maybe even crashes. Sometimes we are not able to find the root...