Loading Large Datasets Beyond the Available RAM in Power BI
In the previous chapter, you learned how to read from and write to a CSV file using both Python and R. When it comes to reading a file, whether you use Power BI’s standard data import feature or the techniques shown in the previous chapter, the main limitation on the file size is the amount of RAM available on the machine where Power BI Desktop is installed.
In a data enrichment phase, it may be necessary to extract information needed for ongoing analysis from very large files (terabytes in size). In these cases, it is almost always necessary to implement big data solutions to be able to handle such large amounts of data. Very often, however, it is necessary to import files that are slightly larger than the available memory in order to extract aggregate information and then persist it in a small table for reuse during processing. In such cases, you don’t need to worry about sophisticated big data platforms...
