Chapter 2. Getting Hands-on with I/O, Redirection Pipes, and Filters

In day-to-day work, we come across different kinds of files such as text files, source code files from different programming languages (for example, file.sh, file.c, and file.cpp), and so on. While working, we often perform various operations on files or directories such as searching for a given string or pattern, replacing strings, printing few lines of a file, and so on. Performing these operations is not easy if we have to do it manually. Manual searching for a string or pattern in a directory having thousands of files can take months, and has high chances of making errors.

Shell provides many powerful commands to make our work easier, faster, and error-free. Shell commands have the ability to manipulate and filter text from different streams such as standard input, file, and so on. Some of these commands are grep, sed, head, tr, sort, and so on. Shell also comes with a feature of redirecting output from one...

Standard I/O and error streams

In shell programming, there are different ways to provide an input (for example, via a keyboard and terminal) and display an output (for example, terminal and file) and error (for example, terminal), if any, during the execution of a command or program.

The following examples show the input, output, and error while running the commands:

The input from a user by a keyboard and the input obtained by a program via a standard input stream, that is terminal, is taken as follows:
```
$ read -p "Enter your name:"
Enter your name:Foo
```
The output printed on the standard output stream, that is terminal, is as follows:
```
$ echo "Linux Shell Scripting"
Linux Shell Scripting
```
The error message printed on the standard error stream, that is terminal, is as follows:
```
$  cat hello.txt
cat: hello.txt: No such file or directory
```

When a program executes, by default, three files get opened with it which are stdin, stdout, and stderr. The following table provides a short description...

Redirecting the standard I/O and error streams

We have an option to redirect standard input, output, and errors, for example, to a file, another command, intended stream, and so on. Redirection is useful in different ways. For example, I have a bash script whose output and errors are displayed on a standard output—that is, terminal. We can avoid mixing an error and output by redirecting one of them or both to a file. Different operators are used for redirection. The following table shows some of operators used for redirection, along with its description:

Operator	Description
`>`	This redirects a standard output to a file
`>>`	This appends a standard output to a file
`<`	This redirects a standard input from a file
`>&`	This redirects a standard output and error to a file
`>>&`	This appends a standard output and error to a file
`\|`	This redirects an output to another command

Redirecting standard output

An output of a program or command can...

Pipe and pipelines – connecting commands

The outputs of the programs are generally saved in files for further use. Sometimes, temporary files are created in order to use an output of a program as an input to another program. We can avoid creating temporary files and feed the output of a program as an input to another program using bash pipe and pipelines.

Pipe

The pipe denoted by the operator | connects the standard output of a process in the left to the standard input in the right process by inter process communication mechanism. In other words, the | (pipe) connects commands by providing the output of a command as the input to another command.

Consider the following example:

$ cat /proc/cpuinfo | less

Here, the cat command, instead of displaying the content of the /proc/cpuinfo file on stdout, passes its output as an input to the less command. The less command takes the input from cat and displays on the stdout per page.

Another example using pipe is as follows:

$ ps -aux | wc -l   ...

Regular expressions

Regular expression (also known as regex or regexp) provides a way of specifying a pattern to be matched in a given big chunk of text data. It supports a set of characters to specify the pattern. It is widely used for a text search and string manipulation. A lot of shell commands provide an option to specify regex such as grep, sed, find, and so on.

The regular expression concept is also used in other programming languages such as C++, Python, Java, Perl, and so on. Libraries are available in different languages to support regular expression's features.

Regular expression metacharacters

The metacharacters used in regular expressions are explained in the following table:

Metacharacters	Description
* (Asterisk)	This matches zero or more occurrences of the previous character
+ (Plus)	This matches one or more occurrences of the previous character
?	This matches zero or one occurrence of the previous element
. (Dot)	This matches any one character
^	This matches...

Filtering an output using grep

One of the powerful and widely used command in shell is grep. It searches in an input file and matches lines in which the given pattern is found. By default, all the matched patterns are printed on stdout that is usually terminal. We can also redirect the matched output to other streams such as file. Instead of giving an input from a file, grep can also take the input from the redirected output of the command executed on the left-hand side of '|'.

Syntax

The syntax of using the grep command is as follows:

grep [OPTIONS] PATTERN [FILE...]

Here, FILE can be multiple files for a search. If no file is given as an input for a search, it will search the standard input.

PATTERN can be any valid regular expression. Put PATTERN within single quotes (') or double quotes (") as per need. For example, use single quotes (') to avoid any bash expansion and double quotes (") for expansion.

A lot of OPTIONS are available in grep. Some of the important...

Standard I/O and error streams

In shell programming, there are different ways to provide an input (for example, via a keyboard and terminal) and display an output (for example, terminal and file) and error (for example, terminal), if any, during the execution of a command or program.

The following examples show the input, output, and error while running the commands:

The input from a user by a keyboard and the input obtained by a program via a standard input stream, that is terminal, is taken as follows:
```
$ read -p "Enter your name:"
Enter your name:Foo
```
The output printed on the standard output stream, that is terminal, is as follows:
```
$ echo "Linux Shell Scripting"
Linux Shell Scripting
```
The error message printed on the standard error stream, that is terminal, is as follows:
```
$  cat hello.txt
cat: hello.txt: No such file or directory
```

When a program executes, by default, three files get opened with it which are stdin, stdout, and stderr. The following table provides a short description of each of these...

Redirecting the standard I/O and error streams

We have an option to redirect standard input, output, and errors, for example, to a file, another command, intended stream, and so on. Redirection is useful in different ways. For example, I have a bash script whose output and errors are displayed on a standard output—that is, terminal. We can avoid mixing an error and output by redirecting one of them or both to a file. Different operators are used for redirection. The following table shows some of operators used for redirection, along with its description:

Operator	Description
`>`	This redirects a standard output to a file
`>>`	This appends a standard output to a file
`<`	This redirects a standard input from a file
`>&`	This redirects a standard output and error to a file
`>>&`	This appends a standard output and error to a file
`\|`	This redirects an output to another command

Redirecting standard output

An output of a program or command can be redirected...

Pipe and pipelines – connecting commands

The outputs of the programs are generally saved in files for further use. Sometimes, temporary files are created in order to use an output of a program as an input to another program. We can avoid creating temporary files and feed the output of a program as an input to another program using bash pipe and pipelines.

Pipe

The pipe denoted by the operator | connects the standard output of a process in the left to the standard input in the right process by inter process communication mechanism. In other words, the | (pipe) connects commands by providing the output of a command as the input to another command.

Consider the following example:

$ cat /proc/cpuinfo | less

Here, the cat command, instead of displaying the content of the /proc/cpuinfo file on stdout, passes its output as an input to the less command. The less command takes the input from cat and displays on the stdout per page.

Another example using pipe is as follows:

$ ps -aux | wc -l    # Showing...

Regular expressions

Regular expression (also known as regex or regexp) provides a way of specifying a pattern to be matched in a given big chunk of text data. It supports a set of characters to specify the pattern. It is widely used for a text search and string manipulation. A lot of shell commands provide an option to specify regex such as grep, sed, find, and so on.

The regular expression concept is also used in other programming languages such as C++, Python, Java, Perl, and so on. Libraries are available in different languages to support regular expression's features.

Regular expression metacharacters

The metacharacters used in regular expressions are explained in the following table:

Metacharacters	Description
* (Asterisk)	This matches zero or more occurrences of the previous character
+ (Plus)	This matches one or more occurrences of the previous character
?	This matches zero or one occurrence of the previous element
. (Dot)	This matches any one character
^	This matches the...

Filtering an output using grep

One of the powerful and widely used command in shell is grep. It searches in an input file and matches lines in which the given pattern is found. By default, all the matched patterns are printed on stdout that is usually terminal. We can also redirect the matched output to other streams such as file. Instead of giving an input from a file, grep can also take the input from the redirected output of the command executed on the left-hand side of '|'.

Syntax

The syntax of using the grep command is as follows:

grep [OPTIONS] PATTERN [FILE...]

Here, FILE can be multiple files for a search. If no file is given as an input for a search, it will search the standard input.

PATTERN can be any valid regular expression. Put PATTERN within single quotes (') or double quotes (") as per need. For example, use single quotes (') to avoid any bash expansion and double quotes (") for expansion.

A lot of OPTIONS are available in grep. Some of the important and widely used options...

Editing output using sed

The sed command is a non-interactive stream editor that allows you to modify the content of the standard input or file. It performs an operation on each line in a pipeline. The syntax will be:

sed [OPTIONS]... {script} [input-file …]

By default, the output is displayed on stdout, but can be redirected to a file if specified.

The input-file are the files on which sed needs to be run. If no files are specified, it reads from stdin.

The script can be a command or a file with multiple commands to pass to sed, and OPTIONS to sed are described in the following table:

Option	Description
-n	This suppresses automatic printing of pattern space
-e script	This allows multiple scripts to be executed
-r	This uses the extended regex in the script
-l N	This specifies line wrap length
--posix	This disables all GNU extensions
-u	This loads the minimal amounts of data from input and flushes output buffers frequently

String substitution using s

The sed command is widely used...

Duplicating a stream using tee

In some cases, it's necessary to print an output on stdout and save an output in a file. In general, command output can either be printed or can be saved in a file. To solve it, the tee command is used. This command reads from the standard input and writes to both standard output and files. The syntax of tee will be as follows:

tee [OPTION] [FILE …]

The tee command copies the output to each FILE and also to stdout. The OPTIONS can be as follows:

Option	Description
`-a, --append`	This appends to the `FILE` instead of overwriting
`-i, --ignore-interrupts`	This ignores interrupt signals, if any

Writing an output to stdout and file: In general, to write an output to stdout and file, we will call the same command twice, with and without redirection. For example, the following command shows how to print an output on stdout and save it to a file:

$  ls /usr/bin/*.pl  # Prints output on stdout
/usr/bin/rsyslog-recover-qi.pl  /usr/bin/syncqt.pl
$  ls /usr/bin/*...

Linux Shell Scripting Essentials: Learn shell scripting to solve complex shell-related problems and to efficiently automate your day-to-day tasks

What do you get with Print?

Linux Shell Scripting Essentials

Chapter 2. Getting Hands-on with I/O, Redirection Pipes, and Filters

Standard I/O and error streams

Redirecting the standard I/O and error streams

Redirecting standard output

Pipe and pipelines – connecting commands

Pipe

Regular expressions

Regular expression metacharacters

Filtering an output using grep

Syntax

Standard I/O and error streams

Redirecting the standard I/O and error streams

Redirecting standard output

Pipe and pipelines – connecting commands

Pipe

Regular expressions

Regular expression metacharacters

Filtering an output using grep

Syntax

Editing output using sed

String substitution using s

Duplicating a stream using tee

Page 1 of 13

Description

Who is this book for?

What you will learn

Product Details

What do you get with Print?

Product Details

Frequently bought together

Table of Contents

Recommendations for you

Customer reviews

People who bought this also bought

About the author

FAQs

Linux Shell Scripting Essentials: Learn shell scripting to solve complex shell-related problems and to efficiently automate your day-to-day tasks

What do you get with Print?

Contact Details

Shipping Address

Billing Address

Description

Who is this book for?

What you will learn

Product Details

What do you get with Print?

Contact Details

Shipping Address

Billing Address

Product Details

Packt Subscriptions

Frequently bought together

Table of Contents

Recommendations for you

Customer reviews

People who bought this also bought

About the author

FAQs