Flume sink
Similar to the source, the sink is managed by SinkRunne, which manages the thread and execution model. Unlike a source, however, a sink is polling-based and polls the channel for events. The sink is the component that outputs (according to type of output required) it from the agent to an external or other source. Sinks also participate in transaction management, and when the output from a sink is successful, an acknowledgement is passed back to the channel. The channel then takes the event away from the persistence mechanism. Transaction management will be covered in detail in a separate section.
There are a variety of existing sinks available, as follows:
- HDFS: Write to HDFS. This currently supports writing text and sequence files (in compressed format as well). The following is a sample HDFS sink configuration (taken from Flume user guide) for an agent named
a1. The full configuration can be found in the Flume user guide (https://flume.apache.org):
a1.channels = c1 a1.sinks = k1...
