Defining splittable DoFn as a unification for bounded and unbounded sources
Beam offers a wide variety of source and sink transforms. We will not walk through them in this book because their details can be easily found online. In this book, we have used the KafkaIO transform heavily – other source and sink transforms are used analogously but specifically on the target storage system.
The question that arises is this: what should we do when there are either specific requirements for the way the data is read (or stored) or when we need to connect to a data source that Beam does not have a connector for? Let's first see how to implement a custom source.
A fundamental requirement for any source is that it has the ability to split itself. We need to split a bounded source in order to be able to parallelize its processing and we need to split an unbounded source to get a persistent moment in time we can return to in case of a failure. Such a moment in time is typically...
