Incremental loads and performance
Most data sources grow over time, and when dealing with large volumes of data, the time taken to reload documents increases along with the size of the data.
Background
The time taken to read all the data from a file or database increases as the size of the data grows, especially if this source is on the wrong side of a firewall or on an older, slower network infrastructure.
One way to reduce the time taken for the reload is to only load from source data that has been added or changed, reducing the amount of data retrieved from the source.
The less data that needs to pulled from a database, spreadsheet, or text file, the quicker the reload will be. Reading the bulk of the data from a QVD file is substantially faster.
How to do it
Incremental load is a technique whereby data is read from a source and stored on a local drive as a QVD file. Subsequent loads are first done from the QVD file; then, only the new (or updated) data is read from the source and finally...
