What if the amount of work at different steps in our pipeline is very different?
For example, it takes a lot more time to fetch the HTML than to parse it. Or what if we don't have a pipeline at all, just a lot of tasks we would like to distribute between coroutines.
That's where the fan-out design pattern kicks in. The number of coroutines may read from the same channel, distributing the work.
We can have one coroutine produce some results:
private fun producePages() = produce {
for (i in 1..10_000) {
for (c in 'a'..'z') {
send(i to "page$c")
}
}
}
And have a function that would create a coroutine that reads those results:
private fun consumePages(channel: ReceiveChannel<Pair<Int, String>>) = async {
for (p in channel) {
println(p)
}
}
This allows us to generate...
