Introduction

Over the course of the last few chapters, we've been progressively moving outward. We started with the assumption that everything will run on one processor, probably in a single thread. Then we looked at how to structure our program without this assumption, performing different tasks on many threads. We then tried to speed up processing by getting multiple threads and cores working on the same task. Now we've pulled back about as far as we can, and we're going to take a look at how to break up work in order to execute it on multiple computers. For large amounts of data, this can be especially useful.

In fact, big data has become more and more common. The definition of big data is a moving target as disk sizes grow, but you're working with big data if you have trouble processing it in memory or even storing it on disk. There's a lot of information locked in that much data, but getting it out can be a real problem. The recipes in this chapter will help...