Using rate limiters to prevent runaway workflows
DevOps engineers can be responsible for a service that is made up of dozens of microservices. These microservices can then number in the dozens to the tens of thousands of instances running in data centers around the globe. Once a service consists of more than a couple of instances, some form of rate control needs to exist to prevent bad rollouts or configuration changes from causing mass destruction.
Some type of a rate limiter for work with forced pause intervals is critical to prevent runaway infrastructure changes.
Rate limiting is easy to implement, but the scope of the rate limiter is going to depend on what your workflows are doing. For services, you may only want one type of change to happen at a time or only affect some number of instances at a time.
The first type of rate limiting would prevent multiple instances of a workflow type from running at a time; for example, you might only want one satellite disk erasure...
