Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Free Learning
Arrow right icon
Linux Shell Scripting Cookbook
Linux Shell Scripting Cookbook

Linux Shell Scripting Cookbook: Do amazing things with the shell and automate tedious tasks , Third Edition

Arrow left icon
Profile Icon Clif Flynt Profile Icon Sarath Lakshman Profile Icon Shantanu Tushar
Arrow right icon
$54.99
Full star icon Full star icon Full star icon Full star icon Empty star icon 4 (4 Ratings)
Paperback May 2017 552 pages 3rd Edition
eBook
$9.99 $43.99
Paperback
$54.99
Subscription
Free Trial
Renews at $19.99p/m
Arrow left icon
Profile Icon Clif Flynt Profile Icon Sarath Lakshman Profile Icon Shantanu Tushar
Arrow right icon
$54.99
Full star icon Full star icon Full star icon Full star icon Empty star icon 4 (4 Ratings)
Paperback May 2017 552 pages 3rd Edition
eBook
$9.99 $43.99
Paperback
$54.99
Subscription
Free Trial
Renews at $19.99p/m
eBook
$9.99 $43.99
Paperback
$54.99
Subscription
Free Trial
Renews at $19.99p/m

What do you get with Print?

Product feature icon Instant access to your digital eBook copy whilst your Print order is Shipped
Product feature icon Paperback book shipped to your preferred address
Product feature icon Download this book in EPUB and PDF formats
Product feature icon Access this title in our online reader with advanced features
Product feature icon DRM FREE - Read whenever, wherever and however you want
OR
Modal Close icon
Payment Processing...
tick Completed

Shipping Address

Billing Address

Shipping Methods
Table of content icon View table of contents Preview book icon Preview Book

Linux Shell Scripting Cookbook

Shell Something Out

In this chapter, we will cover the following recipes:

  • Displaying output in a terminal
  • Using variables and environment variables
  • Function to prepend to environment variables
  • Math with the shell
  • Playing with file descriptors and redirection
  • Arrays and associative arrays
  • Visiting aliases
  • Grabbing information about the terminal
  • Getting and setting dates and delays
  • Debugging the script
  • Functions and arguments
  • Sending output from one command to another
  • Reading n characters without pressing the return key
  • Running a command until it succeeds
  • Field separators and iterators
  • Comparisons and tests
  • Customizing bash with configuration files

Introduction

In the beginning, computers read a program from cards or tape and generated a single report. There was no operating system, no graphics monitors, not even an interactive prompt.

By the 1960s, computers supported interactive terminals (frequently a teletype or glorified typewriter) to invoke commands.

When Bell Labs created an interactive user interface for the brand new Unix operating system, it had a unique feature. It could read and evaluate the same commands from a text file (called a shell script), as it accepted being typed on a terminal.

This facility was a huge leap forward in productivity. Instead of typing several commands to perform a set of operations, programmers could save the commands in a file and run them later with just a few keystrokes. Not only does a shell script save time, it also documents what you did.

Initially, Unix supported one interactive shell, written by Stephen Bourne, and named it the Bourne Shell (sh).

In 1989, Brian Fox of the GNU Project took features from many user interfaces and created a new shell—the Bourne Again Shell (bash). The bash shell understands all of the Bourne shell constructs and adds features from csh, ksh, and others.

As Linux has become the most popular implementation of Unix like operating systems, the bash shell has become the de-facto standard shell on Unix and Linux.

This book focuses on Linux and bash. Even so, most of these scripts will run on both Linux and Unix, using bash, sh, ash, dash, ksh, or other sh style shells.

This chapter will give readers an insight into the shell environment and demonstrate some basic shell features.

Displaying output in a terminal

Users interact with the shell environment via a terminal session. If you are running a GUI-based system, this will be a terminal window. If you are running with no GUI, (a production server or ssh session), you will see the shell prompt as soon as you log in.

Displaying text in the terminal is a task most scripts and utilities need to perform regularly. The shell supports several methods and different formats for displaying text.

Getting ready

Commands are typed and executed in a terminal session. When a terminal is opened, a prompt is displayed. The prompt can be configured in many ways, but frequently resembles this:

username@hostname$

Alternatively, it can also be configured as root@hostname # or simply as $ or #.

The $ character represents regular users and # represents the administrative user root. Root is the most privileged user in a Linux system.

It is a bad idea to directly use the shell as the root user (administrator) to perform tasks. Typing errors have the potential to do more damage when your shell has more privileges. It is recommended that you log in as a regular user (your shell may denote this as $ in the prompt), and use tools such as sudo to run privileged commands. Running a command as sudo <command> <arguments> will run it as root.

A shell script typically begins with a shebang:

#!/bin/bash

Shebang is a line on which #! is prefixed to the interpreter path. /bin/bash is the interpreter command path for Bash. A line starting with a # symbol is treated by the bash interpreter as a comment. Only the first line of a script can have a shebang to define the interpreter to be used to evaluate the script.

A script can be executed in two ways:

  1. Pass the name of the script as a command-line argument:
        bash myScript.sh
  1. Set the execution permission on a script file to make it executable:
        chmod 755 myScript.sh
        ./myScript.sh.

If a script is run as a command-line argument for bash, the shebang is not required. The shebang facilitates running the script on its own. Executable scripts use the interpreter path that follows the shebang to interpret a script.

Scripts are made executable with the chmod command:

$ chmod a+x sample.sh

This command makes a script executable by all users. The script can be executed as follows:

$ ./sample.sh #./ represents the current directory

Alternatively, the script can be executed like this:

$ /home/path/sample.sh # Full path of the script is used

The kernel will read the first line and see that the shebang is #!/bin/bash. It will identify /bin/bash and execute the script as follows:

$ /bin/bash sample.sh

When an interactive shell starts, it executes a set of commands to initialize settings, such as the prompt text, colors, and so on. These commands are read from a shell script at ~/.bashrc (or ~/.bash_profile for login shells), located in the home directory of the user. The Bash shell maintains a history of commands run by the user in the ~/.bash_history file.

The ~ symbol denotes your home directory, which is usually /home/user, where user is your username or /root for the root user. A login shell is created when you log in to a machine. However, terminal sessions you create while logged in to a graphical environment (such as GNOME, KDE, and so on), are not login shells. Logging in with a display manager such as GDM or KDM may not read a .profile or .bash_profile (most don't), but logging in to a remote system with ssh will read the .profile. The shell delimits each command or command sequence with a semicolon or a new line. Consider this example: $ cmd1 ; cmd2
This is equivalent to these: 
$ cmd1
$ cmd2

A comment starts with # and proceeds up to the end of the line. The comment lines are most often used to describe the code, or to disable execution of a line of code during debugging:

# sample.sh - echoes "hello world"
echo "hello world"

Now let's move on to the basic recipes in this chapter.

How to do it...

The echo command is the simplest command for printing in the terminal.

By default, echo adds a newline at the end of every echo invocation:

$ echo "Welcome to Bash"
Welcome to Bash

Simply, using double-quoted text with the echo command prints the text in the terminal. Similarly, text without double quotes also gives the same output:

$ echo Welcome to Bash
Welcome to Bash

Another way to do the same task is with single quotes:

$ echo 'text in quotes'

These methods appear similar, but each has a specific purpose and side effects. Double quotes allow the shell to interpret special characters within the string. Single quotes disable this interpretation.

Consider the following command:

$ echo "cannot include exclamation - ! within double quotes"

This returns the following output:

bash: !: event not found error

If you need to print special characters such as !, you must either not use any quotes, use single quotes, or escape the special characters with a backslash (\):

$ echo Hello world !

Alternatively, use this:

$ echo 'Hello world !'

Alternatively, it can be used like this:

$ echo "Hello World\!" #Escape character \ prefixed.

When using echo without quotes, we cannot use a semicolon, as a semicolon is the delimiter between commands in the Bash shell:

echo hello; hello 

From the preceding line, Bash takes echo hello as one command and the second hello as the second command.

Variable substitution, which is discussed in the next recipe, will not work within single quotes.

Another command for printing in the terminal is printf. It uses the same arguments as the C library printf function. Consider this example:

$ printf "Hello world"

The printf command takes quoted text or arguments delimited by spaces. It supports formatted strings. The format string specifies string width, left or right alignment, and so on. By default, printf does not append a newline. We have to specify a newline when required, as shown in the following script:

#!/bin/bash
#Filename: printf.sh

printf  "%-5s %-10s %-4s\n" No Name  Mark
printf  "%-5s %-10s %-4.2f\n" 1 Sarath 80.3456
printf  "%-5s %-10s %-4.2f\n" 2 James 90.9989
printf  "%-5s %-10s %-4.2f\n" 3 Jeff 77.564

We will receive the following formatted output:

No    Name       Mark
1     Sarath     80.35
2     James      91.00
3     Jeff       77.56

How it works...

The %s, %c, %d, and %f characters are format substitution characters, which define how the following argument will be printed. The %-5s string defines a string substitution with left alignment (- represents left alignment) and a 5 character width. If - was not specified, the string would have been aligned to the right. The width specifies the number of characters reserved for the string. For Name, the width reserved is 10. Hence, any name will reside within the 10-character width reserved for it and the rest of the line will be filled with spaces up to 10 characters total.

For floating point numbers, we can pass additional parameters to round off the decimal places.

For the Mark section, we have formatted the string as %-4.2f, where .2 specifies rounding off to two decimal places. Note that for every line of the format string, a newline (\n) is issued.

There's more...

While using flags for echo and printf, place the flags before any strings in the command, otherwise Bash will consider the flags as another string.

Escaping newline in echo

By default, echo appends a newline to the end of its output text. Disable the newline with the -n flag. The echo command accepts escape sequences in double-quoted strings as an argument. When using escape sequences, use echo as echo -e "string containing escape sequences". Consider the following example:

echo -e "1\t2\t3"
1  2  3

Printing a colored output

A script can use escape sequences to produce colored text on the terminal.

Colors for text are represented by color codes, including, reset = 0, black = 30, red = 31, green = 32, yellow = 33, blue = 34, magenta = 35, cyan = 36, and white = 37.

To print colored text, enter the following command:

echo -e "\e[1;31m This is red text \e[0m"

Here, \e[1;31m is the escape string to set the color to red and \e[0m resets the color back. Replace 31 with the required color code.

For a colored background, reset = 0, black = 40, red = 41, green = 42, yellow = 43, blue = 44, magenta = 45, cyan = 46, and white=47, are the commonly used color codes.

To print a colored background, enter the following command:

echo -e "\e[1;42m Green Background \e[0m"

These examples cover a subset of escape sequences. The documentation can be viewed with man console_codes.

Using variables and environment variables

All programming languages use variables to retain data for later use or modification. Unlike compiled languages, most scripting languages do not require a type declaration before a variable is created. The type is determined by usage. The value of a variable is accessed by preceding the variable name with a dollar sign. The shell defines several variables it uses for configuration and information like available printers, search paths, and so on. These are called environment variables.

Getting ready

Variables are named as a sequence of letters, numbers, and underscores with no whitespace. Common conventions are to use UPPER_CASE for environment variables and camelCase or lower_case for variables used within a script.

All applications and scripts can access the environment variables. To view all the environment variables defined in your current shell, issue the env or printenv command:

$> env 
PWD=/home/clif/ShellCookBook 
HOME=/home/clif 
SHELL=/bin/bash 
# ... And many more lines

To view the environment of other processes, use the following command:

cat /proc/$PID/environ

Set PID with a process ID of the process (PID is an integer value).

Assume an application called gedit is running. We obtain the process ID of gedit with the pgrep command:

$ pgrep gedit
12501

We view the environment variables associated with the process by executing the following command:

$ cat /proc/12501/environ
GDM_KEYBOARD_LAYOUT=usGNOME_KEYRING_PID=1560USER=slynuxHOME=/home/slynux
Note that the previous output has many lines stripped for convenience. The actual output contains more variables.
The /proc/PID/environ special file contains a list of environment variables and their values. Each variable is represented as a name=value pair, separated by a null character (\0). This is not easily human readable.

To make a human-friendly report, pipe the output of the cat command to tr, to substitute the \0 character with \n:

$ cat /proc/12501/environ  | tr '\0' '\n'

How to do it...

Assign a value to a variable with the equal sign operator:

varName=value

The name of the variable is varName and value is the value to be assigned to it. If value does not contain any space character (such as space), it need not be enclosed in quotes, otherwise it must be enclosed in single or double quotes.

Note that var = value and var=value are different. It is a usual mistake to write var = value instead of var=value. An equal sign without spaces is an assignment operation, whereas using spaces creates an equality test.

Access the contents of a variable by prefixing the variable name with a dollar sign ($).

var="value" #Assign "value" to var
echo $var

You may also use it like this:

echo ${var}

This output will be displayed:

value

Variable values within double quotes can be used with printf, echo, and other shell commands:

#!/bin/bash
#Filename :variables.sh
fruit=apple
count=5
echo "We have $count ${fruit}(s)"

The output will be as follows:

We have 5 apple(s)

Because the shell uses a space to delimit words, we need to add curly braces to let the shell know that the variable name is fruit, not fruit(s).

Environment variables are inherited from the parent processes. For example, HTTP_PROXY is an environment variable that defines which proxy server to use for an Internet connection.

Usually, it is set as follows:

HTTP_PROXY=192.168.1.23:3128
export HTTP_PROXY

The export command declares one or more variables that will be inherited by child tasks. After variables are exported, any application executed from the current shell script, receives this variable. There are many standard environment variables created and used by the shell, and we can export our own variables.

For example, the PATH variable lists the folders, which the shell will search for an application. A typical PATH variable will contain the following:

$ echo $PATH 
/home/slynux/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games

Directory paths are delimited by the : character. Usually, $PATH is defined in /etc/environment, /etc/profile or ~/.bashrc.

To add a new path to the PATH environment, use the following command:

export PATH="$PATH:/home/user/bin"

Alternatively, use these commands:

$ PATH="$PATH:/home/user/bin" 
$ export PATH 
$ echo $PATH 
/home/slynux/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/home/user/bin

Here we have added /home/user/bin to PATH.

Some of the well-known environment variables are HOME, PWD, USER, UID, and SHELL.

When using single quotes, variables will not be expanded and will be displayed as it is. This means, $ echo '$var' will display $var.

Whereas, $ echo "$var" will display the value of the $var variable if it is defined, or nothing if it is not defined.

There's more...

The shell has many more built-in features. Here are a few more:

Finding the length of a string

Get the length of a variable's value with the following command:

length=${#var}

Consider this example:

$ var=12345678901234567890$
echo ${#var}
20

The length parameter is the number of characters in the string.

Identifying the current shell

To identify the shell which is currently being used, use the SHELL environment variable.

echo $SHELL

Alternatively, use this command:

echo $0

Consider this example:

$ echo $SHELL
/bin/bash

Also, by executing the echo $0 command, we will get the same output:

$ echo $0
/bin/bash

Checking for super user

The UID environment variable holds the User ID. Use this value to check whether the current script is being run as a root user or regular user. Consider this example:

If [ $UID -ne 0 ]; then 
  echo Non root user. Please run as root. 
else 
  echo Root user 
fi

Note that [ is actually a command and must be separated from the rest of the string with spaces. We can also write the preceding script as follows:

if test $UID -ne 0:1 
  then 
    echo Non root user. Please run as root 
  else 
    echo Root User 
fi

The UID value for the root user is 0.

Modifying the Bash prompt string (username@hostname:~$)

When we open a terminal or run a shell, we see a prompt such as user@hostname: /home/$. Different GNU/Linux distributions have different prompts and different colors. The PS1 environment variable defines the primary prompt. The default prompt is defined by a line in the ~/.bashrc file.

  • View the line used to set the PS1 variable:
        $ cat ~/.bashrc | grep PS1 
        PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '
  • To modify the prompt, enter the following command:
        slynux@localhost: ~$ PS1="PROMPT> " # Prompt string changed 
        PROMPT> Type commands here.
  • We can use colored text using the special escape sequences such as \e[1;31 (refer to the Displaying output in a terminal recipe of this chapter).

Certain special characters expand to system parameters. For example, \u expands to username, \h expands to hostname, and \w expands to the current working directory.

Function to prepend to environment variables

Environment variables are often used to store a list of paths of where to search for executables, libraries, and so on. Examples are $PATH and  $LD_LIBRARY_PATH, which will typically resemble this:

PATH=/usr/bin;/bin 
LD_LIBRARY_PATH=/usr/lib;/lib

This means that whenever the shell has to execute an application (binary or script), it will first look in /usr/bin and then search /bin.

When building and installing a program from source, we often need to add custom paths for the new executable and libraries. For example, we might install myapp in /opt/myapp, with binaries in a /opt/myapp/bin folder and libraries in /opt/myapp/lib.

How to do it...

This example shows how to add new paths to the beginning of an environment variable. The first example shows how to do this with what's been covered so far, the second demonstrates creating a function to simplify modifying the variable. Functions are covered later in this chapter.

export PATH=/opt/myapp/bin:$PATH 
export LD_LIBRARY_PATH=/opt/myapp/lib;$LD_LIBRARY_PATH

The PATH and LD_LIBRARY_PATH variables should now look something like this:

PATH=/opt/myapp/bin:/usr/bin:/bin 
LD_LIBRARY_PATH=/opt/myapp/lib:/usr/lib;/lib

We can make adding a new path easier by defining a prepend function in the .bashrc file.

prepend() { [ -d "$2" ] && eval $1=\"$2':'\$$1\" && export $1; }

This can be used in the following way:

prepend PATH /opt/myapp/bin 
prepend LD_LIBRARY_PATH /opt/myapp/lib

How it works...

The prepend() function first confirms that the directory specified by the second parameter to the function exists. If it does, the eval expression sets the variable, with the name in the first parameter equal to the second parameter string, followed by : (the path separator), and then the original value for the variable.

If the variable is empty when we try to prepend, there will be a trailing : at the end. To fix this, modify the function to this:

prepend() { [ -d "$2" ] && eval $1=\"$2\$\{$1:+':'\$$1\}\" && export $1 ; }
In this form of the function, we introduce a shell parameter expansion of the form:
${parameter:+expression}
This expands to expression if parameter is set and is not null.
With this change, we take care to try to append : and the old value if, and only if, the old value existed when trying to prepend.

Math with the shell

The Bash shell performs basic arithmetic operations using the let, (( )), and [] commands. The expr and bc utilities are used to perform advanced operations.

How to do it...

  1. A numeric value is assigned to a variable the same way strings are assigned. The value will be treated as a number by the methods that access it:
        #!/bin/bash
        no1=4;
        no2=5;
  1. The let command is used to perform basic operations directly. Within a let command, we use variable names without the $ prefix. Consider this example:
        let result=no1+no2
        echo $result

                  Other uses of let command are as follows:

  • Use this for increment: 
                $ let no1++
  • For decrement, use this:
                $ let no1--
  • Use these for shorthands:
                let no+=6
                let no-=6

                These are equal to let no=no+6 and let no=no-6, respectively.

  • Alternate methods are as follows:

              The [] operator is used in the same way as the let command:

                    result=$[ no1 + no2 ]

              Using the $ prefix inside the [] operator is legal; consider this example:

                    result=$[ $no1 + 5 ]

      The (( )) operator can also be used. The prefix variable names                        with a $ within the (( )) operator:

                    result=$(( no1 + 50 ))

             The expr expression can be used for basic operations:

                    result=`expr 3 + 4`
                    result=$(expr $no1 + 5)

      The preceding methods do not support floating point numbers,
      and operate on integers only.

  1. The bc application, the precision calculator, is an advanced utility for mathematical operations. It has a wide range of options. We can perform floating point arithmetic and use advanced functions:
        echo "4 * 0.56" | bc
        2.24
        no=54;
        result=`echo "$no * 1.5" | bc`
        echo $result
        81.0

The bc application accepts prefixes to control the operation. These are separated from each other with a semicolon.

  • Decimal places scale with bc: In the following example, the scale=2 parameter sets the number of decimal places to 2. Hence, the output of bc will contain a number with two decimal places:
                echo "scale=2;22/7" | bc
                3.14
  • Base conversion with bc: We can convert from one base number system to another one. This code converts numbers from decimal to binary and binary to decimal:
                #!/bin/bash
                Desc: Number conversion
                no=100
                echo "obase=2;$no" | bc 
                1100100
                no=1100100
                echo "obase=10;ibase=2;$no" | bc
                100
  • The following examples demonstrate calculating squares and square roots:
                echo "sqrt(100)" | bc #Square root
                echo "10^10" | bc #Square

Playing with file descriptors and redirection

File descriptors are integers associated with the input and output streams. The best-known file descriptors are stdin, stdout, and stderr. The contents of one stream can be redirected to another. This recipe shows examples on how to manipulate and redirect with file descriptors.

Getting ready

Shell scripts frequently use standard input (stdin), standard output (stdout), and standard error (stderr). A script can redirect output to a file with the greater-than symbol. Text generated by a command may be normal output or an error message. By default, both normal output (stdout) and error messages (stderr) are sent to the display. The two streams can be separated by specifying a specific descriptor for each stream.

File descriptors are integers associated with an opened file or data stream. File descriptors 0, 1, and 2 are reserved, as given here:

  • 0: stdin
  • 1: stdout
  • 2: stderr

How to do it...

  1. Use the greater-than symbol to append text to a file:
        $ echo "This is a sample text 1" > temp.txt

This stores the echoed text in temp.txt. If temp.txt already exists, the single greater-than sign will delete any previous contents.

  1. Use double-greater-than to append text to a file:
        $ echo "This is sample text 2" >> temp.txt
  1. Use cat to view the contents of the file:
        $ cat temp.txt
        This is sample text 1
        This is sample text 2

The next recipes demonstrate redirecting stderr. A message is printed to the stderr stream when a command generates an error message. Consider the following example:

$ ls +
ls: cannot access +: No such file or directory

Here + is an invalid argument and hence an error is returned.

Successful and unsuccessful commands
When a command exits because of an error, it returns a nonzero exit status. The command returns zero when it terminates after successful completion. The return status is available in the special variable $? (run echo $? immediately after the command execution statement to print the exit status).

The following command prints the stderr text to the screen rather than to a file (and because there is no stdout output, out.txt will be empty):

$ ls + > out.txt
ls: cannot access +: No such file or directory 

In the following command, we redirect stderr to out.txt with 2> (two greater-than):

$ ls + 2> out.txt # works

You can redirect stderr to one file and stdout to another file.

$ cmd 2>stderr.txt 1>stdout.txt

It is also possible to redirect stderr and stdout to a single file by converting stderr to stdout using this preferred method:

$ cmd 2>&1 allOutput.txt

This can be done even using an alternate approach:

$ cmd &> output.txt 

If you don't want to see or save any error messages, you can redirect the stderr output to /dev/null, which removes it completely. For example, consider that we have three files a1, a2, and a3. However, a1 does not have the read-write-execute permission for the user. To print the contents of all files starting with the letter a, we use the cat command. Set up the test files as follows:

$ echo A1 > a1
$ echo A2 > a2
$ echo A3 > a3
$ chmod 000 a1  #Deny all permissions

Displaying the contents of the files using wildcards (a*), will generate an error message for the a1 file because that file does not have the proper read permission:

$ cat a*
cat: a1: Permission denied
A2
A3

Here, cat: a1: Permission denied belongs to the stderr data. We can redirect the stderr data into a file, while sending stdout to the terminal.

$ cat a* 2> err.txt #stderr is redirected to err.txt
A2
A3

$ cat err.txt
cat: a1: Permission denied

Some commands generate output that we want to process and also save for future reference or other processing. The stdout stream is a single stream that we can redirect to a file or pipe to another program. You might think there is no way for us to have our cake and eat it too.

However, there is a way to redirect data to a file, while providing a copy of redirected data as stdin to the next command in a pipe. The tee command reads from stdin and redirects the input data to stdout and one or more files.

command | tee FILE1 FILE2 | otherCommand

In the following code, the stdin data is received by the tee command. It writes a copy of stdout to the out.txt file and sends another copy as stdin for the next command. The cat -n command puts a line number for each line received from stdin and writes it into stdout:

$ cat a* | tee out.txt | cat -n
cat: a1: Permission denied
     1 A2
     2 A3

Use cat to examine the contents of out.txt:

$ cat out.txt
A2
A3
Observe that cat: a1: Permission denied does not appear, because it was sent to stderr. The tee command reads only from stdin.

By default, the tee command overwrites the file. Including the -a option will force it to append the new data.

$ cat a* | tee -a out.txt | cat -n

Commands with arguments follow the format: command FILE1 FILE2 ... or simply command FILE.

To send two copies of the input to stdout, use - for the filename argument:

$ cmd1 | cmd2 | cmd -

Consider this example:

$ echo who is this | tee -
who is this
who is this

Alternately, we can use /dev/stdin as the output filename to use stdin.
Similarly, use /dev/stderr for standard error and /dev/stdout for standard output. These are special device files that correspond to stdin, stderr, and stdout.

How it works...

The redirection operators (> and >>) send output to a file instead of the terminal. The > and >> operators behave slightly differently. Both redirect output to a file, but the single greater-than symbol (>) empties the file and then writes to it, whereas the double greater-than symbol (>>) adds the output to the end of the existing file.

By default, the redirection operates on standard output. To explicitly take a specific file descriptor, you must prefix the descriptor number to the operator.

The > operator is equivalent to 1> and similarly it applies for >> (equivalent to 1>>).

When working with errors, the stderr output is dumped to the /dev/null file. The ./dev/null file is a special device file where any data received by the file is discarded. The null device is often known as a black hole, as all the data that goes into it is lost forever.

There's more...

Commands that read input from stdin can receive data in multiple ways. It is possible to specify file descriptors of our own, using cat and pipes. Consider this example:

$ cat file | cmd
$ cmd1 | cmd2

Redirection from a file to a command

We can read data from a file as stdin with the less-than symbol (<):

$ cmd < file

Redirecting from a text block enclosed within a script

Text can be redirected from a script into a file. To add a warning to the top of an automatically generated file, use the following code:

#!/bin/bash 
cat<<EOF>log.txt 
This is a generated file. Do not edit. Changes will be overwritten. 
EOF

The lines that appear between cat <<EOF >log.txt and the next EOF line will appear as the stdin data. The contents of log.txt are shown here:

$ cat log.txt 
This is a generated file. Do not edit. Changes will be overwritten. 

Custom file descriptors

A file descriptor is an abstract indicator for accessing a file. Each file access is associated with a special number called a file descriptor. 0, 1, and 2 are reserved descriptor numbers for stdin, stdout, and stderr.

The exec command can create new file descriptors. If you are familiar with file access in other programming languages, you may be familiar with the modes for opening files. These three modes are commonly used:

  • Read mode
  • Write with append mode
  • Write with truncate mode

The < operator reads from the file to stdin. The > operator writes to a file with truncation (data is written to the target file after truncating the contents). The >> operator writes to a file by appending (data is appended to the existing file contents and the contents of the target file will not be lost). File descriptors are created with one of the three modes.

Create a file descriptor for reading a file:

$ exec 3<input.txt # open for reading with descriptor number 3

We can use it in the following way:

$ echo this is a test line > input.txt
$ exec 3<input.txt

Now you can use file descriptor 3 with commands. For example, we will use cat<&3:

$ cat<&3
this is a test line

If a second read is required, we cannot reuse the file descriptor 3. We must create a new file descriptor (perhaps 4) with exec to read from another file or re-read from the first file.

Create a file descriptor for writing (truncate mode):

$ exec 4>output.txt # open for writing

Consider this example:

$ exec 4>output.txt
$ echo newline >&4
$ cat output.txt
newline

Now create a file descriptor for writing (append mode):

$ exec 5>>input.txt

Consider the following example:

$ exec 5>>input.txt
$ echo appended line >&5
$ cat input.txt
newline
appended line

Arrays and associative arrays

Arrays allow a script to store a collection of data as separate entities using indices. Bash supports both regular arrays that use integers as the array index, and associative arrays, which use a string as the array index. Regular arrays should be used when the data is organized numerically, for example, a set of successive iterations. Associative arrays can be used when the data is organized by a string, for example, host names. In this recipe, we will see how to use both of these.

Getting ready

To use associate arrays, you must have Bash Version 4 or higher.

How to do it...

Arrays can be defined using different techniques:

  1. Define an array using a list of values in a single line:
        array_var=(test1 test2 test3 test4)
        #Values will be stored in consecutive locations starting 
        from index 0.

Alternately, define an array as a set of index-value pairs:

        array_var[0]="test1"
        array_var[1]="test2"
        array_var[2]="test3"
        array_var[3]="test4"
        array_var[4]="test5"
        array_var[5]="test6"
  1. Print the contents of an array at a given index using the following commands:
        echo ${array_var[0]}
        test1
        index=5
        echo ${array_var[$index]}
        test6
  1. Print all of the values in an array as a list, using the following commands:
        $ echo ${array_var[*]}
        test1 test2 test3 test4 test5 test6

  Alternately, you can use the following command:

        $ echo ${array_var[@]}
        test1 test2 test3 test4 test5 test6
  1. Print the length of an array (the number of elements in an array):
        $ echo ${#array_var[*]}6

There's more...

Associative arrays have been introduced to Bash from Version 4.0. When the indices are a string (site names, user names, nonsequential numbers, and so on), an associative array is easier to work with than a numerically indexed array.

Defining associative arrays

An associative array can use any text data as an array index. A declaration statement is required to define a variable name as an associative array:

$ declare -A ass_array

After the declaration, elements are added to the associative array using either of these two methods:

  • Inline index-value list method:
        $ ass_array=([index1]=val1 [index2]=val2)
  • Separate index-value assignments:
        $ ass_array[index1]=val1
        $ ass_array'index2]=val2

For example, consider the assignment of prices for fruits, using an associative array:

$ declare -A fruits_value
$ fruits_value=([apple]='100 dollars' [orange]='150 dollars')

Display the contents of an array:

$ echo "Apple costs ${fruits_value[apple]}"
Apple costs 100 dollars

Listing of array indexes

Arrays have indexes for indexing each of the elements. Ordinary and associative arrays differ in terms of index type.

Obtain the list of indexes in an array.

$ echo ${!array_var[*]}

Alternatively, we can also use the following command:

$ echo ${!array_var[@]}

In the previous fruits_value array example, consider the following command:

$ echo ${!fruits_value[*]}
orange apple

This will work for ordinary arrays too.

Visiting aliases

An alias is a shortcut to replace typing a long-command sequence. In this recipe, we will see how to create aliases using the alias command.

How to do it...

These are the operations you can perform on aliases:

  1. Create an alias:
        $ alias new_command='command sequence'

This example creates a shortcut for the apt-get install command:

        $ alias install='sudo apt-get install'

Once the alias is defined, we can type install instead of sudo apt-get install.

  1. The alias command is temporary: aliases exist until we close the current terminal. To make an alias available to all shells, add this statement to the ~/.bashrc file. Commands in ~/.bashrc are always executed when a new interactive shell process is spawned:
        $ echo 'alias cmd="command seq"' >> ~/.bashrc
  1. To remove an alias, remove its entry from ~/.bashrc (if any) or use the unalias command. Alternatively, alias example= should unset the alias named example.
  2. This example creates an alias for rm that will delete the original and keep a copy in a backup directory:
        alias rm='cp $@ ~/backup && rm $@'
When you create an alias, if the item being aliased already exists, it will be replaced by this newly aliased command for that user.

There's more...

When running as a privileged user, aliases can be a security breach. To avoid compromising your system, you should escape commands.

Escaping aliases

Given how easy it is to create an alias to masquerade as a native command, you should not run aliased commands as a privileged user. We can ignore any aliases currently defined, by escaping the command we want to run. Consider this example:

$ \command

The \ character escapes the command, running it without any aliased changes. When running privileged commands on an untrusted environment, it is always a good security practice to ignore aliases by prefixing the command with \. The attacker might have aliased the privileged command with his/her own custom command, to steal critical information that is provided by the user to the command.

Listing aliases

The alias command lists the currently defined aliases:

$ aliasalias lc='ls -color=auto'
alias ll='ls -l'
alias vi='vim'
Left arrow icon Right arrow icon

Key benefits

  • Become an expert in creating powerful shell scripts and explore the full possibilities of the shell
  • Automate any administrative task you could imagine, with shell scripts
  • Packed with easy-to-follow recipes on new features on Linux, particularly, Debian-based, to help you accomplish even the most complex tasks with ease

Description

The shell is the most powerful tool your computer provides. Despite having it at their fingertips, many users are unaware of how much the shell can accomplish. Using the shell, you can generate databases and web pages from sets of files, automate monotonous admin tasks such as system backups, monitor your system's health and activity, identify network bottlenecks and system resource hogs, and more. This book will show you how to do all this and much more. This book, now in its third edition, describes the exciting new features in the newest Linux distributions to help you accomplish more than you imagine. It shows how to use simple commands to automate complex tasks, automate web interactions, download videos, set up containers and cloud servers, and even get free SSL certificates. Starting with the basics of the shell, you will learn simple commands and how to apply them to real-world issues. From there, you'll learn text processing, web interactions, network and system monitoring, and system tuning. Software engineers will learn how to examine system applications, how to use modern software management tools such as git and fossil for their own work, and how to submit patches to open-source projects. Finally, you'll learn how to set up Linux Containers and Virtual machines and even run your own Cloud server with a free SSL Certificate from letsencrypt.org.

Who is this book for?

If you are a beginner or an intermediate Linux user who wants to master the skill of quickly writing scripts and automate tasks without reading the entire man pages, then this book is for you. You can start writing scripts and one-liners by simply looking at the relevant recipe and its descriptions without any working knowledge of shell scripting or Linux. Intermediate / advanced users, system administrators / developers, and programmers can use this book as a reference when they face problems while coding.

What you will learn

  • • Interact with websites via scripts
  • • Write shell scripts to mine and process data from the Web
  • • Automate system backups and other repetitive tasks with crontab
  • • Create, compress, and encrypt archives of your critical data.
  • • Configure and monitor Ethernet and wireless networks
  • • Monitor and log network and system activity
  • • Tune your system for optimal performance
  • • Improve your system s security
  • • Identify resource hogs and network bottlenecks
  • • Extract audio from video files
  • • Create web photo albums
  • • Use git or fossil to manage revision control and interact with FOSS projects
  • • Create and maintain Linux containers and Virtual Machines
  • • Run a private Cloud server
Estimated delivery fee Deliver to Ukraine

Economy delivery 10 - 13 business days

$6.95

Premium delivery 6 - 9 business days

$21.95
(Includes tracking information)

Product Details

Country selected
Publication date, Length, Edition, Language, ISBN-13
Publication date : May 29, 2017
Length: 552 pages
Edition : 3rd
Language : English
ISBN-13 : 9781785881985
Tools :

What do you get with Print?

Product feature icon Instant access to your digital eBook copy whilst your Print order is Shipped
Product feature icon Paperback book shipped to your preferred address
Product feature icon Download this book in EPUB and PDF formats
Product feature icon Access this title in our online reader with advanced features
Product feature icon DRM FREE - Read whenever, wherever and however you want
OR
Modal Close icon
Payment Processing...
tick Completed

Shipping Address

Billing Address

Shipping Methods
Estimated delivery fee Deliver to Ukraine

Economy delivery 10 - 13 business days

$6.95

Premium delivery 6 - 9 business days

$21.95
(Includes tracking information)

Product Details

Publication date : May 29, 2017
Length: 552 pages
Edition : 3rd
Language : English
ISBN-13 : 9781785881985
Tools :

Packt Subscriptions

See our plans and pricing
Modal Close icon
$19.99 billed monthly
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Simple pricing, no contract
$199.99 billed annually
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Choose a DRM-free eBook or Video every month to keep
Feature tick icon PLUS own as many other DRM-free eBooks or Videos as you like for just $5 each
Feature tick icon Exclusive print discounts
$279.99 billed in 18 months
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Choose a DRM-free eBook or Video every month to keep
Feature tick icon PLUS own as many other DRM-free eBooks or Videos as you like for just $5 each
Feature tick icon Exclusive print discounts

Frequently bought together


Stars icon
Total $ 198.97
Linux: Powerful Server Administration
$99.99
Linux Shell Scripting Cookbook
$54.99
Working with Linux ??? Quick Hacks for the Command Line
$43.99
Total $ 198.97 Stars icon
Banner background image

Table of Contents

13 Chapters
Shell Something Out Chevron down icon Chevron up icon
Have a Good Command Chevron down icon Chevron up icon
File In, File Out Chevron down icon Chevron up icon
Texting and Driving Chevron down icon Chevron up icon
Tangled Web? Not At All! Chevron down icon Chevron up icon
Repository Management Chevron down icon Chevron up icon
The Backup Plan Chevron down icon Chevron up icon
The Old-Boy Network Chevron down icon Chevron up icon
Put On the Monitors Cap Chevron down icon Chevron up icon
Administration Calls Chevron down icon Chevron up icon
Tracing the Clues Chevron down icon Chevron up icon
Tuning a Linux System Chevron down icon Chevron up icon
Containers, Virtual Machines, and the Cloud Chevron down icon Chevron up icon

Customer reviews

Rating distribution
Full star icon Full star icon Full star icon Full star icon Empty star icon 4
(4 Ratings)
5 star 50%
4 star 0%
3 star 50%
2 star 0%
1 star 0%
RAGHAV Mar 21, 2018
Full star icon Full star icon Full star icon Full star icon Full star icon 5
book recieved in great condition.
Amazon Verified review Amazon
Raj Jun 03, 2019
Full star icon Full star icon Full star icon Full star icon Full star icon 5
👍👍👍👍👍👍✌✌✌✌✌🕵️‍♀️🕵️‍♀️💗
Amazon Verified review Amazon
qishan2002 Jan 08, 2018
Full star icon Full star icon Full star icon Empty star icon Empty star icon 3
Very lengthy illustration of the command options some of which are quite useful and some are quite obscure and we can do without. The book is overall weak in with respect to scripting -- have commands work together.
Amazon Verified review Amazon
Amazon Customer Jan 28, 2018
Full star icon Full star icon Full star icon Empty star icon Empty star icon 3
I would expect a book about Linux Shell Scripting to be be about actually scripting. This book shows you how to write glorified commands but is anemic with regards to actual scripting. If your goal is to write better commands and even how to structure them, this is your book but with hardcore scripting techniques, I can't reasonably encourage you to purchase this one.
Amazon Verified review Amazon
Get free access to Packt library with over 7500+ books and video courses for 7 days!
Start Free Trial

FAQs

What is the delivery time and cost of print book? Chevron down icon Chevron up icon

Shipping Details

USA:

'

Economy: Delivery to most addresses in the US within 10-15 business days

Premium: Trackable Delivery to most addresses in the US within 3-8 business days

UK:

Economy: Delivery to most addresses in the U.K. within 7-9 business days.
Shipments are not trackable

Premium: Trackable delivery to most addresses in the U.K. within 3-4 business days!
Add one extra business day for deliveries to Northern Ireland and Scottish Highlands and islands

EU:

Premium: Trackable delivery to most EU destinations within 4-9 business days.

Australia:

Economy: Can deliver to P. O. Boxes and private residences.
Trackable service with delivery to addresses in Australia only.
Delivery time ranges from 7-9 business days for VIC and 8-10 business days for Interstate metro
Delivery time is up to 15 business days for remote areas of WA, NT & QLD.

Premium: Delivery to addresses in Australia only
Trackable delivery to most P. O. Boxes and private residences in Australia within 4-5 days based on the distance to a destination following dispatch.

India:

Premium: Delivery to most Indian addresses within 5-6 business days

Rest of the World:

Premium: Countries in the American continent: Trackable delivery to most countries within 4-7 business days

Asia:

Premium: Delivery to most Asian addresses within 5-9 business days

Disclaimer:
All orders received before 5 PM U.K time would start printing from the next business day. So the estimated delivery times start from the next day as well. Orders received after 5 PM U.K time (in our internal systems) on a business day or anytime on the weekend will begin printing the second to next business day. For example, an order placed at 11 AM today will begin printing tomorrow, whereas an order placed at 9 PM tonight will begin printing the day after tomorrow.


Unfortunately, due to several restrictions, we are unable to ship to the following countries:

  1. Afghanistan
  2. American Samoa
  3. Belarus
  4. Brunei Darussalam
  5. Central African Republic
  6. The Democratic Republic of Congo
  7. Eritrea
  8. Guinea-bissau
  9. Iran
  10. Lebanon
  11. Libiya Arab Jamahriya
  12. Somalia
  13. Sudan
  14. Russian Federation
  15. Syrian Arab Republic
  16. Ukraine
  17. Venezuela
What is custom duty/charge? Chevron down icon Chevron up icon

Customs duty are charges levied on goods when they cross international borders. It is a tax that is imposed on imported goods. These duties are charged by special authorities and bodies created by local governments and are meant to protect local industries, economies, and businesses.

Do I have to pay customs charges for the print book order? Chevron down icon Chevron up icon

The orders shipped to the countries that are listed under EU27 will not bear custom charges. They are paid by Packt as part of the order.

List of EU27 countries: www.gov.uk/eu-eea:

A custom duty or localized taxes may be applicable on the shipment and would be charged by the recipient country outside of the EU27 which should be paid by the customer and these duties are not included in the shipping charges been charged on the order.

How do I know my custom duty charges? Chevron down icon Chevron up icon

The amount of duty payable varies greatly depending on the imported goods, the country of origin and several other factors like the total invoice amount or dimensions like weight, and other such criteria applicable in your country.

For example:

  • If you live in Mexico, and the declared value of your ordered items is over $ 50, for you to receive a package, you will have to pay additional import tax of 19% which will be $ 9.50 to the courier service.
  • Whereas if you live in Turkey, and the declared value of your ordered items is over € 22, for you to receive a package, you will have to pay additional import tax of 18% which will be € 3.96 to the courier service.
How can I cancel my order? Chevron down icon Chevron up icon

Cancellation Policy for Published Printed Books:

You can cancel any order within 1 hour of placing the order. Simply contact customercare@packt.com with your order details or payment transaction id. If your order has already started the shipment process, we will do our best to stop it. However, if it is already on the way to you then when you receive it, you can contact us at customercare@packt.com using the returns and refund process.

Please understand that Packt Publishing cannot provide refunds or cancel any order except for the cases described in our Return Policy (i.e. Packt Publishing agrees to replace your printed book because it arrives damaged or material defect in book), Packt Publishing will not accept returns.

What is your returns and refunds policy? Chevron down icon Chevron up icon

Return Policy:

We want you to be happy with your purchase from Packtpub.com. We will not hassle you with returning print books to us. If the print book you receive from us is incorrect, damaged, doesn't work or is unacceptably late, please contact Customer Relations Team on customercare@packt.com with the order number and issue details as explained below:

  1. If you ordered (eBook, Video or Print Book) incorrectly or accidentally, please contact Customer Relations Team on customercare@packt.com within one hour of placing the order and we will replace/refund you the item cost.
  2. Sadly, if your eBook or Video file is faulty or a fault occurs during the eBook or Video being made available to you, i.e. during download then you should contact Customer Relations Team within 14 days of purchase on customercare@packt.com who will be able to resolve this issue for you.
  3. You will have a choice of replacement or refund of the problem items.(damaged, defective or incorrect)
  4. Once Customer Care Team confirms that you will be refunded, you should receive the refund within 10 to 12 working days.
  5. If you are only requesting a refund of one book from a multiple order, then we will refund you the appropriate single item.
  6. Where the items were shipped under a free shipping offer, there will be no shipping costs to refund.

On the off chance your printed book arrives damaged, with book material defect, contact our Customer Relation Team on customercare@packt.com within 14 days of receipt of the book with appropriate evidence of damage and we will work with you to secure a replacement copy, if necessary. Please note that each printed book you order from us is individually made by Packt's professional book-printing partner which is on a print-on-demand basis.

What tax is charged? Chevron down icon Chevron up icon

Currently, no tax is charged on the purchase of any print book (subject to change based on the laws and regulations). A localized VAT fee is charged only to our European and UK customers on eBooks, Video and subscriptions that they buy. GST is charged to Indian customers for eBooks and video purchases.

What payment methods can I use? Chevron down icon Chevron up icon

You can pay with the following card types:

  1. Visa Debit
  2. Visa Credit
  3. MasterCard
  4. PayPal
What is the delivery time and cost of print books? Chevron down icon Chevron up icon

Shipping Details

USA:

'

Economy: Delivery to most addresses in the US within 10-15 business days

Premium: Trackable Delivery to most addresses in the US within 3-8 business days

UK:

Economy: Delivery to most addresses in the U.K. within 7-9 business days.
Shipments are not trackable

Premium: Trackable delivery to most addresses in the U.K. within 3-4 business days!
Add one extra business day for deliveries to Northern Ireland and Scottish Highlands and islands

EU:

Premium: Trackable delivery to most EU destinations within 4-9 business days.

Australia:

Economy: Can deliver to P. O. Boxes and private residences.
Trackable service with delivery to addresses in Australia only.
Delivery time ranges from 7-9 business days for VIC and 8-10 business days for Interstate metro
Delivery time is up to 15 business days for remote areas of WA, NT & QLD.

Premium: Delivery to addresses in Australia only
Trackable delivery to most P. O. Boxes and private residences in Australia within 4-5 days based on the distance to a destination following dispatch.

India:

Premium: Delivery to most Indian addresses within 5-6 business days

Rest of the World:

Premium: Countries in the American continent: Trackable delivery to most countries within 4-7 business days

Asia:

Premium: Delivery to most Asian addresses within 5-9 business days

Disclaimer:
All orders received before 5 PM U.K time would start printing from the next business day. So the estimated delivery times start from the next day as well. Orders received after 5 PM U.K time (in our internal systems) on a business day or anytime on the weekend will begin printing the second to next business day. For example, an order placed at 11 AM today will begin printing tomorrow, whereas an order placed at 9 PM tonight will begin printing the day after tomorrow.


Unfortunately, due to several restrictions, we are unable to ship to the following countries:

  1. Afghanistan
  2. American Samoa
  3. Belarus
  4. Brunei Darussalam
  5. Central African Republic
  6. The Democratic Republic of Congo
  7. Eritrea
  8. Guinea-bissau
  9. Iran
  10. Lebanon
  11. Libiya Arab Jamahriya
  12. Somalia
  13. Sudan
  14. Russian Federation
  15. Syrian Arab Republic
  16. Ukraine
  17. Venezuela