Streaming file content

In the previous sections, we learned how to read and write in large blocks using os.ReadFile() and os.WriteFile().

This works well when the files are small, which is usually the case when you're doing DevOps automation. However, sometimes, the files we want to read are very large – in most cases, you wouldn't want to read a 2 GiB file into memory. In those cases, we want to stream the contents of the file in manageable chunks that we can operate on while keeping memory usage low.

The most basic version of this was shown in the previous section. There, we used two streams to copy a file: io.ReadCloser from the HTTP client and io.WriteCloser for writing to local disk. We used the io.Copy() function to copy from the network file to the disk file.

Go's io interfaces also allow us to stream files to copy content, search for content, manipulate input to output, and more.

Stdin/Stdout/Stderr are just files

Throughout this book, you...