Most of the examples we have seen in this chapter involve extracting remote web-based data sources rather than processing local files. This is because there are more and more datasets provided by authorities via web services. I also find mixing remote and local resources more interesting than simply changing the formats of files or reprojecting them. As well as working with web-based data sources, we also worked with files. We downloaded ZIP and gzip archives and extracted them. We also read JSON files line by line and even made Node.js use ogr2ogr to import some shapefiles. We could do more file processing in ogr2ogr, GDAL, or psql but that would seem a bit dull.
Obviously our ETL examples were not very complex, and we did not design and execute any sophisticated data processing workflows. The important thing is that we did some task automation and have shown that adding value to our data does not...
