Kafka architecture
This section aims to explain ins and outs of Apache Kafka. We will try to dive deep into its architecture and then, later on try expanding each part of it's architecture's components in a bit more detail.
So, let's stream forward.
Core architecture principles of Kafka
The main motivation behind Kafka when developed by LinkedIn’s engineering team was
To create a unified messaging platform to cater to real-time data from various applications in a big organization.
There are core architecture principles based on which Kafka was conceived and designed. The bulleted points sum up these principles:
- Maximize performance (compression and B-tree usage is an example)
- Wherever possible, core kernel capabilities to offload work to drive optimization and performance (zero-copy and direct use of Linux filesystem cache is an example)
- Distributed architecture
- Fault tolerance
- Durability of messages
- Wherever possible, eliminate redundant work
- Offload responsibility of tasks to consuming...
