Processing event streams

At first glance, stream processing might seem like it is some extreme paradigm shift. And truth be told, if you have to do it all from scratch, it is not for the fainthearted. For example, we need a process that consumes from a stream and invokes the business logic. We need one of these for each shard. We need logic to increase or decrease the number of these processes with the number of shards. We need to restart these processes when they fall over. Finally, we need to keep track of the offset so that we read from the right position on the next call.

This can add up to a decent cloud bill, not to mention all the development elbow grease required. Of course, we can implement this processing as a container sidecar. But serverless, function-as-a-service offerings, such as AWS Lambda, provide this capability for just the cost of invoking the functions to execute the business logic. It takes care of all the heavy lifting.

Still, stream processing is different...