Chapter 8: Loading Large Datasets beyond the Available RAM in Power BI

In the previous chapter, you learned how to read from and write to a CSV file, both with Python and in R. When it comes to reading a file, whether you use Power BI's standard data import feature or the techniques shown in the previous chapter, the main limitation on the file size is due to the amount of RAM available on the machine where Power BI Desktop is installed.

In a data enrichment phase, it may be necessary to extract information needed for ongoing analysis from very large files (terabytes in size). In these cases, it is almost always necessary to implement big data solutions to be able to handle such masses of data. Very often, however, it is necessary to import files that are slightly larger than the available RAM in order to extract aggregate information and then persist it in a small table for reuse during processing. In such cases, it's not necessary to bother with demanding big data platforms...