Linking commands, pipes, and input/output

This section is probably one of the most important in the book because it describes a fundamental and powerful feature on Linux and Unix: the ability to use pipes and redirect input or output. By themselves, pipes are a fairly trivial feature - commands and scripts can redirect their output to files or commands. So what? This could be considered a massive understatement in the Bash scripting world, because pipes and redirection allow you to enhance commands with the functionality of other commands or features. 

Let's look into this with an example using commands called tail and grep. In this example, the user, Bob, wants to look at his logs in real time (live), but he only wants to find the entries related to the wireless interface. The name of Bob's wireless device can be found using the iwconfig command:

$ iwconfig
wlp3s0 IEEE 802.11abgn ESSID:"127.0.0.1-2.4ghz" 
          Mode:Managed Frequency:2.412 GHz Access Point: 18:D6:C7:FA:26:B1 
          Bit Rate=144.4 Mb/s Tx-Power=22 dBm 
          Retry short limit:7 RTS thr:off Fragment thr:off
          Power Management:on
          Link Quality=58/70 Signal level=-52 dBm 
          Rx invalid nwid:0 Rx invalid crypt:0 Rx invalid frag:0
          Tx excessive retries:0 Invalid misc:90 Missed beacon:0

The iwconfig command is deprecated now. The following commands also will give you wireless interface information:

$ iw dev                # This will give list of wireless interfaces
$ iw dev wlp3s0 link    # This will give detailed information about particular wireless interface

Now that Bob knows his wireless card's identifying name (wlp3s0), Bob can search his system's logs. It is usually found within /var/log/messages. Using the tail command and the -F flag, which allows continuously outputting the logs to the console, Bob can now see all the logs for his system. Unfortunately, he would like to filter the logs using grep, such that only logs with the keyword wlp3s0 are visible.

Bob is faced with a choice: does he search the file continuously, or can he combine tail and grep together to get the results he desires? The answer is yes—using pipes!

$ tail -F /var/log/messages | grep wlp3s0
Nov 10 11:57:13 moon kernel: wlp3s0: authenticate with 18:d6:c7:fa:26:b1
Nov 10 11:57:13 moon kernel: wlp3s0: send auth to 18:d6:c7:fa:26:b1 (try 1/3)
Nov 10 11:57:13 moon kernel: wlp3s0: send auth to 18:d6:c7:fa:26:b1 (try 2/3)
...

As new logs come in, Bob can now monitor them in real time and can stop the console output using Ctrl+C.

Using pipes, we can combine commands into powerful hybrid commands, extending the best features of each command into one single line. Remember pipes!

The usage and flexibility of pipes should be relatively straightforward, but what about directing the input and output of commands? This requires the introduction of three commands to get information from one place to another:

• stdin (standard in)
• stdout (standard out)
• stderr (standard error)

If we are thinking about a single program, stdin is anything that can be provided to it, usually either as a parameter or a user input, using read for example. Stdout and stderr are two streams where output can be sent. Usually, output for both is sent to the console for display, but what if you only want the errors within the stderr stream to go to a file?

$ ls /filethatdoesntexist.txt 2> err.txt
$ ls ~/ > stdout.txt
$ ls ~/ > everything.txt 2>&1 # Gets stderr and stdout
$ ls ~/ >> everything.txt 2>&1 # Gets stderr and stdout
$ cat err.txt
ls: cannot access '/filethatdoesntexist.txt': No such file or directory
$ cat stdout.txt
.... # A whole bunch of files in your home directory

When we cat err.txt, we can see the error output from the stderr stream. This is useful when you only want to record errors and not everything being output to the console. The key feature to observe from the snippet is the usage of >, 2>, and 2>&1. With the arrows we can redirect the output to any file or even to other programs!

Take note of the difference between a single > and double >>. A single > will truncate any file that will have output directed to it, while >> will append any file.

There is a common error when redirecting both stderr and stdout to the same file. Bash should pick up the output to a file first, and then the duplication of the output file descriptors. For more information on file descriptors, see: https://en.wikipedia.org/wiki/File_descriptor
# This is correct
ls ~/ > everything.txt 2>&1
# This is erronous
ls ~/ 2>&1> everything.txt

Now that we know the basics of one of the most powerful features available in Bash, let's try an example—redirection and pipes bonzanza.

Redirection and pipe bonzanza

Open a shell and create a new bash file in your favorite editor:

#!/bin/sh

# Let's run a command and send all of the output to /dev/null
echo "No output?"
ls ~/fakefile.txt > /dev/null 2>&1

# Retrieve output from a piped command 
echo "part 1"
HISTORY_TEXT=`cat ~/.bashrc | grep HIST`
echo "${HISTORY_TEXT}"

# Output the results to history.config
echo "part 2"
echo "${HISTORY_TEXT}" > "history.config"

# Re-direct history.config as input to the cat command
cat < history.config

# Append a string to history.config
echo "MY_VAR=1" >> history.config

echo "part 3 - using Tee"
# Neato.txt will contain the same information as the console
ls -la ~/fakefile.txt ~/ 2>&1 | tee neato.txt

First, ls is a way of producing an error and, instead of pushing erroneous output to the console, it is instead redirected to a special device in Linux called /dev/null. /dev/null is particularly useful as it is a dump for any input that will not be used again. Then, we combine the cat command with grep to find any lines of text with a pipe and use a fork to capture the output to a variable (HISTORY_TEXT).

Then, we echo the contents of HISTORY_TEXT to a file (history.config) using a stdout redirect. Using the history.configfile, we redirect cat to use the raw file—this will be displayed on the console.

Using a double >>, we append an arbitrary string to the history.config file.

Finally, we end the script with redirection for both stdout and stderr, a pipe,, and the tee command. The tee command is useful because it can be used to display content even if it has been redirected to a file (as we just demonstrated).