Task 10 – Separating droppable data from the rest of the data processing, part 2

First, let's rephrase our problem definition from Task 9.

Defining the problem

Create a pipeline that will separate droppable data elements from the rest of the data elements. It will send droppable data to one output topic and the rest to another topic. Make the separation work even in cases when the very first element in a particular window is droppable.

Discussing the problem decomposition

The main problem of our previous approach was that we were not able to distinguish a data element as late in the case when it was the very first data element in that particular window. Therefore, we need to be able to generate window labels prior to receiving any data for that particular window. We can do that using a technique called looping timers – that is, we set a timer and then reset it for a fixed duration in an infinite loop. If possible, we would like to align this timer with...