Building an End-to-End Analytics System – Real-Time Analytics

Ask a business user how quickly they need their data available for analysis and the response will likely be “in real time.” Discuss data latency with an architect and the conversation will involve functionality of the technology stack and cost. The reality for most use cases is that batch data processing is good enough. There are, however, times when real-time data is necessary.

Even in the case of real-time data processing, there will always be some amount of latency. When launching rockets into space carrying astronauts to the International Space Station, every microsecond counts. When transmitting GPS coordinates for directions, a second or two of latency is likely acceptable. When tracking sales at the local electronics retailer, real time starts to look more like microbatches.

There are several aspects of real-time data that need to be dealt with. First, how quickly can the data be transmitted...