Avoiding pod-scheduling disruption

Running a heterogeneous Kubernetes cluster imposes some particularities that need to be considered to reduce application deployment disruption. By default, Amazon EKS uses kube-scheduler as the default scheduler for Kubernetes; when you schedule a pod, kube-scheduler uses a combination of filtering and scoring to select the optimal node to run the pod, which works nicely when you only run Linux-based applications.

kube-scheduler doesn’t take the OS into account and as a result, it is common to see Linux-based sidecar containers or plugins running as DaemonSets failing to be scheduled just because it doesn’t treat nodeSelector in their manifest. For instance, a DaemonSet ensures that all nodes run a copy of a pod, but if you have a heterogenous cluster, Windows pods cannot be deployed on Linux-based nodes and vice versa, failing to schedule the pod; thereby, your deployment will never have a 100% success rate.

We can work around...