With the exponential growth in the amount of data being generated and advanced data-capturing capabilities, enterprises are facing the challenge of making sense out of this mountain of raw data. On the batch processing front, Hadoop has emerged as the go-to framework to deal with big data. Until recently, there has been a void when one looks for frameworks to build real-time stream processing applications. Such applications have become an integral part of a lot of businesses as they enable them to respond swiftly to events and adapt to changing situations. Examples of this are monitoring social media to analyze public response to any new product that you launch and predicting the outcome of an election based on the sentiments of election-related posts.
Organizations are collecting a large volume of data from external sources and want to evaluate/process the data in real time to get market trends, detect fraud, identify user behavior, and so on. The need for real-time processing is increasing day by day and we require a real-time system/platform that should support the following features:
- Scalable: The platform should be horizontally scalable without any down time.
- Fault tolerance: The platform should be able to process the data even after some of the nodes in a cluster go down.
- No data lost: The platform should provide the guaranteed processing of messages.
- High throughput: The system should be able to support millions of records per second and also support any size of messages.
- Easy to operate: The system should have easy installation and operation. Also, the expansion of clusters should be an easy process.
- Multiple languages: The platform should support multiple languages. The end user should be able to write code in different languages. For example, a user can write code in Python, Scala, Java, and so on. Also, we can execute different language code inside the one cluster.
- Cluster isolation: The system should support isolation so that dedicated processes can be assigned to dedicated machines for processing.
