Streaming data and its challenges
Streaming data poses infrastructural and processing challenges. Major tech companies are inventing new data structures and server mechanisms to handle the huge volume and velocity of the streaming data. Software infrastructures such as Kafka, Storm, Bolt, and other similar technologies are being invented to handle this from an infrastructure perspective. We will not go into the details here. Our concern is primarily with the processing challenges.
The processing challenges in stream data are shown in the following figure:
Bounded problems
The first challenge is deciding on a window, and what size window we need to accommodate to make sense of the incoming data. By window, we mean storing the last n data points. For streaming data, it is rare to actually process records one at a time. One of the most common ways to process streaming data is to process them in a window. This can either bundle data points into a group and process them as a unit or it can be a...
