The Flume Architecture
We discussed in previous section, the architecture principles based on which Flume was conceived, now let's deep dive into the architecture. Let's start off with a very basic diagram detailing the architecture of Flume (Figure 06) and then in the following sections keep diving deep.
Figure 06: Basic Flume Architecture
A simple Flume architecture has three important components, which work together to transfer a data from source to destination in real time fashion (stream or log data). They are:
- Source: The responsibility of listening to stream data or events and then putting it to the channel
- Channel: A pipe where events are stored until it has been taken by someone else
- Sink: The responsibility of taking away events from the channel for further processing (sending to another source) or persisting to a data store. If sink operation fails, it will keep trying until success.
The following table summarizes some of examples for each of the components in the Flume architecture...
