So far, we have learned the basics of how to work with the filesystem with the shell. We also looked at some shell mechanisms such as wildcards and completion that simplify life in the command line. In this chapter, we will examine the building blocks that are used to perform data-processing tasks on the shell.

The Unix approach is to favor small, single-purpose utilities with very well-defined interfaces. Redirection and pipes let us connect these small commands and files together so that we can compose them like the elements of an electronic circuit to perform complex tasks. This concept of joining together small units into a more complex mechanism is a very powerful technique.

Most data that we typically work with is textual in nature, so we will study the most useful text-oriented commands in this chapter, along with various practical examples of their usage.

Redirection is a method of connecting files to a command. This mechanism is used to capture the output of a command or to feed input to it.

Note

During this section, we will introduce a few commands briefly, in order to illustrate some concepts. The commands are only used as examples, and their usage does not have any connection to the main topics being covered here. The detailed descriptions of all the features and uses of those commands will be covered in the topic on text-processing commands.

Input and Output Streams

Every command that is run has a channel for data input, termed standard input (stdin), data output, termed standard output (stdout) and standard error (stderr). A command reads data from stdin and writes its results to stdout. If any error occurs, the error messages are written to stderr. These channels can also be thought of as streams through which data flows.

By convention, stdin, stdout, and stderr are assigned the numbers 0, 1, and 2...

A shell pipeline or simply a pipeline refers to a construct where data is pushed from one command to another in an assembly line fashion. It is expressed as a series of commands separated by a pipe symbol |. These pipes connect the stdout of each command to the stdin of the subsequent command. Internally, a pipe is a special memory FIFO (first in, first out) buffer provided by the OS.

The basic syntax of a pipeline is as follows:

command1 | command2

Any number of commands can be linked:

command1 | command2 | command3 | command4

Pipelines are analogous to assembly lines in a factory. Like an assembly line lets multiple workers simultaneously do one designated job repeatedly, ending up with a finished product, a pipeline lets a series of commands work on a stream of data, each doing one task, eventually leading to the desired output.

Pipelines ensure maximum throughput and optimal usage of computing power. The time taken for a pipeline task in most cases will...

In the previous sections, we learned about the two main composing mechanisms of command lines: redirection and piping. In this topic, we will expand our vocabulary of commands that, when combined, let us do a wide variety of data-processing tasks.

We focus on text-processing because it applies to a wide range of real-life data, and will be useful to professionals in any field. A huge amount of data on the internet is in textual form, and text happens to be the easiest way to share data in a portable way as simple columnar CSV (comma-separated values) or TSV (tab-separated values) files. Once you learn these commands, you do not have to rely on the knowledge of any specific software GUI tool, and you can run complex tasks on the shell itself. In many cases, running a quick shell pipeline is much faster than setting up the data in a more complex GUI tool. In a later chapter, you will learn how to save your commands as a sort of automatic recipe, with shell...

In this chapter, you have been introduced to several concepts such as input, output, redirection, and pipelines. You have also learned basic text-processing tools, along with both common and uncommon use cases of these tools, to demonstrate their flexibility. At a conceptual level, several techniques related to processing tabular data have been explored.

A large number of details have been covered. If you are being introduced to these for the first time, you should attempt to understand the concepts at an abstract level and not be overwhelmed by details (which you can always refer to when in doubt). To this end, some additional complexities have been avoided in order to focus on the essential concepts. The students can pick up more nuances as they continue to learn and practice in the future, beyond this brief book.

In the next chapter, you will learn about several more concepts related to the shell, including basic regular expressions, shell expansion, and command substitution...