Implementing idempotence and order tolerance

In Chapter 4, Trusting Facts and Eventual Consistency, we learned that exactly-once delivery of messages is unrealistic. For example, a client request may time out due to network unreliability and have no choice but to resubmit the request because it cannot be certain that the service successfully processed the request; a stream processor may fail in the middle of a batch and retry the entire batch even though part of the batch was successfully processed; or we may replay events from the event lake to repair a service that may have dropped a subset of those events. To account for the reality of at-least-once delivery, we must design our systems to be idempotent. In other words, no matter how many times we receive and process an event or request, it must only update the system once.

We also learned that delivering messages in order can be problematic. A stream will certainly deliver events in the order that it received them, but it may...