A pipeline is a way of structuring the application flow, and is obtained by splitting the main execution into stages that can talk with one another using certain means of communication. This could be either of the following:
- External, such as a network connection or a file
- Internal to the application, like Go's channels
The first stage is often referred to as the producer, while the last one is often called the consumer.
The set of concurrency tools that Go offers allows us to efficiently use multiple CPUs and optimize their usage by blocking input or output operations. Channels in particular are the perfect tools for internal pipeline communication. They can be represented by functions that receive an inbound channel and return an outbound one. The base structure would look something like this:
func stage(in <-chan interface{}) <...
