Understanding Buffers, Streams, and the Filesystem in Node

We've described several features of Node in the preceding topics. However, our introduction to Node wouldn't be complete without looking at some features, such as buffers, streams, and filesystems, which make Node stand out on the development scene. So, in this topic, we will look at how these features play their respective part in the Node environment.

Buffer

This is a chunk of memory where data is stored temporarily. This non-resizable memory is designed to handle raw binary data, and the integers in it are limited to values from 0 to 255 (2^8 - 1), with each integer representing one byte.

Let's look at a real-world scenario so that we have a better understanding of what buffers are and how they work by using the following screenshot from a YouTube video:

Figure 1.8: The workings of a buffer in YouTube

When a YouTube video starts to play, and if the internet is super fast, a gray area is observed in the playing stream. This area is the buffer zone, where data is being collected and stored temporarily (usually in the RAM) to allow for continuous playing, even when the internet is disconnected. The red zone is the playing zone (data processing zone) whose length is dependent on the video's playing speed and the amount of data that has been buffered. If the browser is refreshed, the temporary storage is re-initialized and the process is restarted.

Now that we have seen a real-time application using a buffer, we will now create, read, and write to buffers in the upcoming exercises.

Exercise 2: Creating, Reading, and Writing to a Buffer

In this exercise, our aim is to create, read from, and write to a buffer. Before attempting this exercise, make sure you have completed all of the previous activities in this chapter. To complete this exercise, perform the following steps:

Note

The code files for this exercise can be found here: http://bit.ly/2SiwAw6.

1. Create a new folder in an appropriate location in your system and rename it Buffer Operations.

2. Open the newly created Buffer Operations folder from your code editor and create a buffer.js file. Buffers can be created in the following ways:

   You can create an uninitialized buffer by passing the buffer size to Buffer.alloc(), or you can create an instance of a buffer class; for example, let's create an uninitiated buffer of 5 bytes using the following code:

   var buf1 = Buffer.alloc(5);
   var buf2 = new Buffer(5);
   console.log(buf1)
   console.log(buf2)

   The output is as follows:

   

   Figure 1.9: A buffer of 5 bytes

   You can create a buffer using a given array using from() or using an instance of a buffer; for example, let's initialize a buffer with the contents of the array [10, 20, 30, 40, 50]:

   varbuf3 = new Buffer([10, 20, 30, 40, 50]);
   varbuf4 = Buffer.from([ 10, 20, 30, 40, 50]);
   console.log(buf3)
   console.log(buf4) 

   Note that the integers that make up the array's contents represent bytes. The output can be seen as follows:

   

   Figure 1.10: A buffer created using array content

   Finally, you can create a buffer using a given string and, optionally, the encoding type using from() or using an instance of a buffer. The following code initializes the buffer to a binary encoding of the first argument, which is a string that's specified by the second argument, which is an encoding type:

   var buf5 = new Buffer("Hi Packt students!", "utf-8");
   var buf6 = Buffer.from("Hi Packt students!", "utf-8")
   console.log(buf5)
   console.log(buf6)

   The output can be seen as follows:

   

   Figure 1.11: Output of the buffer created using a string

   The buffer also supports encoding methods such as ascii, ucs2, base64, binary, and so on.

3. To write into a buffer, we can use the buff.write() method. The output returned after a buffer has been written to (created) is the number of octets written into it:

   buffer.write(string[, offset][, length][, encoding])

   Note that the first argument is the string to write to the buffer and the second argument is the encoding. If the encoding is not set, the default encoding, which is utf-8, will be used. Write into the buffer that we created in the preceding step using the following code:

   len = buf5.write("Packt student", "utf-8")
   console.log (len) //The length becomes 13 after writing into the buffer

   The output can be seen as follows:

   

   Figure 1.12: Output of the buffer using utf-8

4. To read from the buffer, the toString() method is commonly used, but keep in mind that many buffers contain text. This method is implemented as follows:

   buffer.toString([encoding][, start][, end]

   Here, we will read from the buffer that was written into in the preceding step and print the output on the command line using the following code:

   console.log(buf5.toString("utf-8", 0, 13))

   

   Figure 1.13: Reading from the buffer

There are a few more methods for buffers, which will be covered in the following sections.

Uninitialized Buffers

You can also create buffers using the allocUnsafe(length) method. The allocUnsafe(length) method creates an uninitialized buffer of the assigned length. When compared to the buffer.alloc() method, it is much faster, but old data in the returned buffer instance needs to be overwritten using either fill() or write().

Let's see how the allocUnsafe(length) method is being used in the following snippet:

var buf = Buffer.allocUnsafe(15);
var buf1 = Buffer.alloc(15);

console.log(buf);
console.log(buf1);

The preceding snippet yields the following output:

Figure 1.14: Output of the buffer created using allocateUnsafe(length) and alloc(length)

Some Other Buffer Methods

There are a few other methods that you need to be aware of. These are listed here:

• byteLength(string, encoding): This method is used to check the number of bytes required to encode a string with a given encoding.

• length: This method is used to check the length of your buffer, that is, how much memory is allocated.

• copy(target, targetStart=0, sourceStart=0, sourceEnd=buffer.length): This method is used to copy the contents of one buffer into another.

• buffer.slice(start, end=buffer.length): This is the same as Array.prototype.slice, except modifying the slice will also modify the original buffer.

Streams

Whenever you talk about reading data from a source and writing it to a destination, you're referring to streams (Unix pipes). A stream can be likened to an EventEmitter. There are four types of streams, and they are as follows:

• Readable: Allows you to read data from a source.

• Writable: Allows you to write data to a destination.

• Duplex: Allows you to read data from a source and write data to a destination.

• Transform: Allows you to modify or transform data while data is being read or written.

Reading Data from Streams

A stream is said to be readable if it permits data to be read from a source, irrespective of what the source is, be it another stream, a file in a filesystem, a buffer in memory, and so on. Various data events can be emitted at various points in a stream. Thus, streams can also be referred to as instances of EventEmitters. Listening to a data event and attaching a callback are the best ways to read data from a stream. A readable stream emits a data event, and your callback executes when data is available.

Let's observe how a stream is read, a data event is emitted, and a callback is executed using the filesystem module with a readable and writable file. See the following code:

const fs = require('fs');
const file = fs.createReadStream('readTextFile.txt');
file.on('data', function(data) {
console.log('Data '+ data);
});
file.on('end', function(){
console.log('Hey!, Am Done reading Data');
});

First, we created a file named readTextFile.txt. Then, a readable stream is created using the fs.createReadStream('filename') function. It is good to know that the stream is in a static state initially, and gets to a flowing state as soon as you listen to a data event and attach a callback. Streams emit an end event when there is no more data to read.

Writing to Streams

A stream is said to be writeable if it permits data to be written to a destination, irrespective of what the destination is. It could be another stream, a file in a filesystem, a buffer in memory, and so on. Similar to readable streams, various events that are emitted at various points in writeable streams can also be referred to as instances of EventEmitter. The write() function is available on the stream instance. It makes writing data to a stream possible.

Take a look at the following snippet:

const fs = require('fs');
const readableStream = fs.createReadStream('readTextFile.txt');
const writableStream = fs.createWriteStream('writeTextFile.txt');
readableStream.on('data', function (data) {
console.log('Hey!, I am about to write what has been read from the file readTextFile.txt');
if (writableStream.write(data) === true) {
console.log('Hey!, I am done writing. Open the file writeTextFile.txt to see what has been written');
    }
else
console.log('Writing is not successful');
});

First, we load the filesystem using the require directive. Then, we create two files: readTextFile.txt and writeTextFile.txt. We then write some text string into the readTextFile.txt file and leave writeTextFile.txt blank. Using the filesystem createReadStream() function with the file path or directory as an argument, we create a readable stream. Thereafter, a writeable stream is created using the filesystem createWriteStream() function with the file path or directory as an argument. readableStream.on('data', callback) listens to a data event and attaches a callback, whereas writableStream.write(data) writes data to a writable stream. Once this snippet is run, you will realize that the text string read from readTextFile.txt has been written into the writeTextFile.txt file.

Duplex

Recall from the short description in the previous section that a duplex is a type of stream that allows you to read data from a source and write data to a destination. Examples of duplex streams are as follows:

• crypto streams

• TCP socket streams

• zlib streams

Transform

This is another stream method that lets you modify or transform data while data is being read or written. In essence, transform streams are duplex streams with the aforementioned functionality. Examples of transform streams are as follows:

• crypto streams

• zlib streams

Some Other Stream Methods

end(): When this method is called, a finish event is emitted by the stream and a notification is sent to the stream that you have finished writing to.

setEncoding(): Upon being called, the stream becomes encoded. This method is used in a situation where a readable stream needs to be encoded.

pipe(): When this method is called on a readable stream, you can read data from the source and write to the destination without normal flow management.

pause(): When this method is called on a flowing stream, the stream pauses, data will be kept in the buffer, and data events will not be emitted anymore. When this is called on a non-flowing stream, data events will not be emitted, but the stream will start flowing.

resume(): When this method is called on a paused stream, the stream starts to flow again.

unpipe(): When this method is called on a piped readable stream, the destination streams are removed from pipes.

Filesystems

Node has a module name File System (fs) that works with the file systems on a computer. This module, fs, performs several methods that can be implemented both synchronously and asynchronously, but there is there is no guaranteed ordering when using them.

Reading, creating, updating, deleting, and renaming are some of the operations that can be performed on any text file that's saved in a directory.

Reading Files

After the required directive has been invoked for the fs module, assuming that we have a string of text contained in the text file, the file can be read by calling the readFile() function, as shown here:

var fs = require('fs');
fs.readFile('readFileName.txt', function(err, data) {
if (err) throw err;
console.log('Read!');
  });

Creating Files

The methods that are highlighted here are made available by the fs module for creating new files:

fs.appendFile(): This method appends a given content to a file. The file will be created in this case, even if it does not exist. This function can also be used to update a file. Its implementation is as follows:

var text = ' Hey John, Welcome to Packt class.'
var fs = require('fs');
fs.appendFile('newTextFile.txt', 'text'
', function (err) {
  if (err) throw err;
  console.log('Saved!');
  });

Once the preceding code is run, you will see the designated string written into the file.

fs.open(): When this method is called, a given file is opened for writing, and if the file doesn't exist, an empty file will be created. The implementation is as follows:

var fs = require('fs');
fs.open('TextFile.txt', 'w', function (err, file) {
  if (err) throw err;
// let's assume the file doesn't exist
  console.log('An empty file created');
});

This method takes a second argument, w, which is referred to as a flag for "writing."

fs.writeFile(): When this method is called on a given file and content exists in it, the existing content is replaced by new content. In a situation where the file doesn't exist, a new file containing the given content will be created. This function can also be used to update the file, and the implementation is as follows:

Var fs = require('fs');
fs.writeFile('textFile.txt', 'Hello content!', function (err) {
if(err)throwerr;
console.log('Saved!');
});

Deleting Files

fs.unlink(): When this method is called on a given file in the filesystem, the file is deleted. The implementation is as follows:

var fs = require('fs');
fs.unlink('textFile.txt', function (err) {
  if (err) throw err;
  console.log('File deleted!');
});

Renaming Files

fs.rename(): This method takes in three arguments: the old name of the file, the new name of the file, and a callback. When this method is called on a given file in the filesystem, the file is renamed. The implementation is as follows:

var fs = require('fs');
fs.rename('textFile.txt', 'renamedtextFile.txt', function (err) {
  if (err) throw err;
  console.log('File Renamed!');
});

Some Other Methods in the Filesystems Module

access(): This method is used to check whether a user has access to the file or directory.

close(): This method is used to close a file.

exists(): Though this method is deprecated, it's called on a file to check whether a file or folder exists.

fstat(): When this method is called on a file, it returns the status of that file.

link(): This method is used to create an additional name for a file. The name could either be the old or a new one.

mkdir(): This method is used make a new directory.

readdir(): This method is used to read the contents of a directory.

rmdir(): This method is used to remove a directory.

stat(): This method is used to return the status of a file.

truncate(): This method is used to truncate a file.

watch(): This method is used to watch for changes in a file name or directory name.

Note

For more information on the Node.js APIs, refer to this link: https://nodejs.org/api/.

Exercise 3: Reading and Writing Data Using Filesystem Operations

Our aim is to read and write data using the Node filesystem. To complete this exercise, the following steps have to be performed:

Note

The code files for this exercise can be found here: http://bit.ly/2U1FIH5.

1. Create a new folder at an appropriate location and rename it FileSystemProgram.

2. Open the newly created FileSystemProgram from your code editor and create write.js, read.js, write.txt, and read.txt, as shown here:

   

   Figure 1.15: Creating filesystem program files

3. Open the read.txt file and write some text; for example, "Welcome to Packt." To read what has been previously input in the read.txt file, open the read.js file and type in the following code:

   var fs = require('fs') 
   fs.readFile('read.txt', function(err, data) { 
   if (err) throw err; 
   console.log('Read!');});

   First, we will use var fs = require('fs') to declare fs (a filesystem variable) and assign a filesystem module to it. Then, we will use the readFile() function on the read.txt file and attach a callback. Next, we will check for errors using if (err) throw err. If no errors are present, then we will print "Read!" using console.log('Read!').

4. Press Ctrl + ` to open the integrated command-line terminal in Visual Studio and run the following command:

   node read.js

   

   Figure 1.16: Running the read program

5. Open the write.js file to write into the write.txt file and run the following command:

   var fs = require('fs');
   fs.writeFile('write.txt','Welcome to packt' function(err, data) {
   if (err) throw err; 
   console.log('Written!');  });

   We use var fs = require('fs') to declare fs (a filesystem variable) and assign a filesystem module to it. We then use fs.writeFile('write.txt', 'Welcome to Packt' function(err, data) to call the writeFile() function on the write.txt file, pass the text to be written as the second argument, and attach a callback as the third argument. Thereafter, we check for if errors using if (err) throw err. If there are no errors, we print Written! using console.log('Written!').

6. Press Ctrl + ` to open the integrated command-line terminal in Visual Studio and run the following command:

   node write.js

   You will obtain the following output:

   

   Figure 1.17: Running the write program

7. Open the write.txt file to view what has been written from the output, as shown here:

   

   Figure 1.18: Contents written to the file

The previous examples and activities have explored the knowledge and implementation of different built-in modules in Node. The next activity will be a hands-on implementation of streaming data from one file to another.

Activity 2: Streaming Data to a File

You have been tasked with developing an application that streams data (copies some text) from a local file on your system into another local file that's located in a different directory. The aim is to read and write data to a stream.

Before we begin this activity, it is essential that you have completed all the previous exercises and activities. Then, use the IDE to create a project called StreamingProgram.

To complete this exercise, the following steps have to be completed:

Note

The code files for this activity can be found here: http://bit.ly/2EkRW8j.

1. Create a folder/directory and name it StreamingProgram.

2. Create stream.js, readTextFile.txt, and writeTextFile.txt in the StreamingProgram directory.

3. Load and import the filesystem module into stream.js file.

4. Create a readable stream on the readTextFile.txt file.

5. Create a writeable stream on the writeTextFile.txt file.

6. Call the on() function on the readableStream() method to read data.

7. Call the write() method on the writeableStream() method to write data.

8. Run the stream.js program via the command line and open the writeTextFile.txt file to confirm that the text has been written to it.

Note

The solution for this activity can be found on page 251.