MapReduce serverless architecture

MapReduce on a serverless platform is very different than in a system such as Hadoop. Most of the differences occur on the operational and system architecture side of things. Another huge difference is the limited processing power and memory we have with our FaaS. Because FaaS providers put in hard limits for both temporary storage space and memory, there are some problems that you cannot realistically solve with a serverless MapReduce implementation.

The good news is that the foundational ideas in the MapReduce design still hold true. If you look back up at the start of the initial list of benefits provided by MapReduce, we naturally get many of these for free, albeit with a few caveats. MapReduce truly shines, due in large part to the parallelization of computation. We have that with serverless functions. Similarly, much work goes into ensuring Hadoop nodes are healthy and able to perform work. Again, we get that for free with serverless functions.

A significant...