Summary

In this chapter we covered the two channel types you are most likely to use in your data processing pipelines.

The memory channel offers speed at the cost of data loss in the event of failure.

Alternatively, the file channel provides a more reliable transport, in that it can tolerate agent failures and restarts, at a performance cost.

You will need to decide which channel is appropriate for your use cases. When trying to decide if a memory channel is appropriate, ask yourself what is the monetary cost if you lose some data. Weigh that against the additional costs of more hardware to cover the difference in performance when deciding if you need a durable channel after all. Another consideration is whether or not the data could be resent. Not all data you may ingest into Hadoop will come from streaming application logs. If you receive daily downloads of data, you can get away with using a memory channel because if you encounter a problem, you can always rerun the import.

Tip

Possible (or...