Understanding computer memory
To make sense of why processing large files requires a new approach, I will briefly discuss computer memory and databases here. Feel free to skip ahead if you are familiar with how memory and databases work.
In computer hardware, memory is the medium that stores data, programs, and files. The hardware for computer memory is split into primary storage, which generally takes the form of RAM (random access memory), and secondary storage, which generally takes the form of a hard drive. Primary storage is used for storing the machine code and the data of active programs, while secondary storage is used for storing all data and files not currently in use. This division in memory usage reflects a few differences in the hardware used for each.
The first difference is that RAM (used for primary storage) is cleared when the computer is shut down, while data stored on hard drives (used for secondary storage) persists. The second difference is that data stored on RAM is...
