Why?
There are a number of reasons why one might opt to process records individually or at least in smaller chunks. Three of the most compelling reasons are:
Limits of computing resources
Network latency
Pareto's principle
From the perspective of efficiency, any one of these reasons is good enough to warrant retrieval of smaller amounts of data. We look at each in greater detail in the following section.
Computing resources
Despite Moore's law holding true for many years now, it is coming under increasing pressure as a trend and cannot be presumed to hold true indefinitely. While one may have a budget with numbers so big that it looks like a phone directory, one's resources are only faster or more powerful on a comparative scale.
Note
Moore's law states that the number of transistors that one is able to fit on an integrated circuit will double every two years.
This law is not much of a natural law as a business observation was made by Gordon Moore, co-founder of Intel. He did not state in such absolute...
