Linux Shell Scripting Cookbook, Second Edition: Don't neglect the shell – this book will empower you to use simple commands to perform complex tasks. Whether you're a casual or advanced Linux user, the cookbook approach makes it all so brilliantly accessible and, above all, useful. , Second Edition

echo is the basic command for printing in the terminal.

echo puts a newline at the end of every echo invocation by default:

$ echo "Welcome to Bash"
Welcome to Bash

Simply, using double-quoted text with the echo command prints the text in the terminal. Similarly, text without double quotes also gives the same output:

$ echo Welcome to Bash
Welcome to Bash

Another way to do the same task is by using single quotes:

$ echo 'text in quotes'

These methods may look similar, but some of them have a specific purpose and side effects too. Consider the following command:

$ echo "cannot include exclamation - ! within double quotes"

This will return the following output:

bash: !: event not found error

Hence, if you want to print special characters such as !, either do not use them within double quotes or escape them with a special escape character (\) prefixed with it, like so:

$ echo Hello world !

Or:

$ echo 'Hello world !'

Or:

$ echo "Hello world \!" #Escape character \ prefixed.

The side effects of each of the methods are as follows:

Another command for printing in the terminal is printf. It uses the same arguments as the printf command in the C programming language. For example:

$ printf "Hello world"

printf takes quoted text or arguments delimited by spaces. We can use formatted strings with printf. We can specify string width, left or right alignment, and so on. By default, printf does not have newline as in the echo command. We have to specify a newline when required, as shown in the following script:

#!/bin/bash 
#Filename: printf.sh

printf  "%-5s %-10s %-4s\n" No Name  Mark 
printf  "%-5s %-10s %-4.2f\n" 1 Sarath 80.3456 
printf  "%-5s %-10s %-4.2f\n" 2 James 90.9989 
printf  "%-5s %-10s %-4.2f\n" 3 Jeff 77.564

We will receive the formatted output:

No    Name       Mark
1     Sarath     80.35
2     James      91.00
3     Jeff       77.56

%s, %c, %d, and %f are format substitution characters for which an argument can be placed after the quoted format string.

%-5s can be described as a string substitution with left alignment (- represents left alignment) with width equal to 5. If - was not specified, the string would have been aligned to the right. The width specifies the number of characters reserved for that variable. For Name, the width reserved is 10. Hence, any name will reside within the 10-character width reserved for it and the rest of the characters will be filled with space up to 10 characters in total.

For floating point numbers, we can pass additional parameters to round off the decimal places.

For marks, we have formatted the string as %-4.2f, where .2 specifies rounding off to two decimal places. Note that for every line of the format string a newline (\n) is issued.

While using flags for echo and printf, always make sure that the flags appear before any strings in the command, otherwise Bash will consider the flags as another string.

echo -e "1\t2\t3"
1  2  3

echo -e "\e[1;31m This is red text \e[0m"

echo -e "\e[1;42m Green Background \e[0m"

Variables are named with the usual naming constructs. When an application is executing, it will be passed a set of variables called environment variables. To view all the environment variables related to a terminal, issue the env command. For every process, environment variables in its runtime can be viewed by:

cat /proc/$PID/environ

Set PID with a process ID of the process (PID always takes an integer value).

For example, assume that an application called gedit is running. We can obtain the process ID of gedit with the pgrep command as follows:

$ pgrep gedit
12501

You can obtain the environment variables associated with the process by executing the following command:

$ cat /proc/12501/environ
GDM_KEYBOARD_LAYOUT=usGNOME_KEYRING_PID=1560USER=slynuxHOME=/home/slynux

The aforementioned command returns a list of environment variables and their values. Each variable is represented as a name=value pair and are separated by a null character (\0). If you can substitute the \0 character with \n, you can reformat the output to show each variable=value pair in each line. Substitution can be made using the tr command as follows:

$ cat /proc/12501/environ  | tr '\0' '\n'

Now, let us see how to assign and manipulate variables and environment variables.

A variable can be assigned as follows:

var=value

var is the name of a variable and value is the value to be assigned. If value does not contain any space character (such as space), it need not be enclosed in quotes, Otherwise it is to be enclosed in single or double quotes.

Note that var = value and var=value are different. It is a usual mistake to write var =value instead of var=value. The later one is the assignment operation, whereas the earlier one is an equality operation.

Printing contents of a variable is done using by prefixing $ with the variable name as follows:

var="value" #Assignment of value to variable var.

echo $var

Or:

echo ${var}

We will receive an output as follows:

value

We can use variable values inside printf or echo in double quotes:

#!/bin/bash
#Filename :variables.sh
fruit=apple
count=5
echo "We have $count ${fruit}(s)"

The output will be as follows:

We have 5 apple(s)

Environment variables are variables that are not defined in the current process, but are received from the parent processes. For example, HTTP_PROXY is an environment variable. This variable defines which proxy server should be used for an Internet connection.

Usually, it is set as:

HTTP_PROXY=192.168.1.23:3128
export HTTP_PROXY

The export command is used to set the env variable. Now any application, executed from the current shell script, will receive this variable. We can export custom variables for our own purposes in an application or shell script that is executed. There are many standard environment variables that are available for the shell by default.

For example, PATH. A typical PATH variable will contain:

$ echo $PATH
/home/slynux/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games

When given a command for execution, the shell automatically searches for the executable in the list of directories in the PATH environment variable (directory paths are delimited by the ":" character). Usually, $PATH is defined in /etc/environment or /etc/profile or ~/.bashrc. When we need to add a new path to the PATH environment, we use:

export PATH="$PATH:/home/user/bin"

Or, alternately, we can use:

$ PATH="$PATH:/home/user/bin"
$ export PATH

$ echo $PATH
/home/slynux/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/home/user/bin

Here we have added /home/user/bin to PATH.

Some of the well-known environment variables are HOME, PWD, USER, UID, SHELL, and so on.

$ echo '$var' will print $var

Let us see more tips associated with standard and environment variables.

length=${#var}

$ var=12345678901234567890$
echo ${#var}
20

echo $SHELL

echo $0

$ echo $SHELL
/bin/bash

$ echo $0
/bin/bash

If [ $UID -ne 0 ]; then
    echo Non root user. Please run as root.
else
    echo Root user
fi

A way to do this is to say it as follows:

export PATH=/opt/myapp/bin:$PATH
export LD_LIBRARY_PATH=/opt/myapp/lib;$LD_LIBRARY_PATH

PATH and LD_LIBRARY_PATH should now look something like this:

PATH=/opt/myapp/bin:/usr/bin:/bin
LD_LIBRARY_PATH=/opt/myapp/lib:/usr/lib;/lib

However, we can make this easier by adding this function in .bashrc-:

prepend() { [ -d "$2" ] && eval $1=\"$2':'\$$1\" && export $1; }

This can be used in the following way:

prepend PATH /opt/myapp/bin
prepend LD_LIBRARY_PATH /opt/myapp/lib

We define a function called prepend(), which first checks if the directory specified by the second parameter to the function exists. If it does, the eval expression sets the variable with the name in the first parameter equal to the second parameter string followed by : (the path separator) and then the original value for the variable.

However, there is one caveat, if the variable is empty when we try to prepend, there will be a trailing : at the end. To fix this, we can modify the function to look like this:

prepend() { [ -d "$2" ] && eval $1=\"$2\$\{$1:+':'\$$1\}\" && export $1 ; }

${parameter:+expression}

With this change, we take care to try to append : and the old value if, and only if, the old value existed when trying to prepend.

The Bash shell environment can perform basic arithmetic operations using the commands let, (( )), and []. The two utilities expr and bc are also very helpful in performing advanced operations.

While writing scripts we use standard input (stdin), standard output (stdout), and standard error (stderr) frequently. Redirection of an output to a file by filtering the contents is one of the essential things we need to perform. While a command outputs some text, it can be either an error or an output (nonerror) message. We cannot distinguish whether it is output text or an error text by just looking at it. However, we can handle them with file descriptors. We can extract text that is attached to a specific descriptor.

File descriptors are integers associated with an opened file or data stream. File descriptors 0, 1, and 2 are reserved as follows:

For output redirection, > and >> operators are different. Both of them redirect text to a file, but the first one empties the file and then writes to it, whereas the later one adds the output to the end of the existing file.

When we use a redirection operator, the output won't print in the terminal but it is directed to a file. When redirection operators are used, by default, they operate on standard output. To explicitly take a specific file descriptor, you must prefix the descriptor number to the operator.

> is equivalent to 1> and similarly it applies for >> (equivalent to 1>>).

When working with errors, the stderr output is dumped to the /dev/null file. ./dev/null is a special device file where any data received by the file is discarded. The null device is often known as a black hole as all the data that goes into it is lost forever.

A command that reads stdin for input can receive data in multiple ways. Also, it is possible to specify file descriptors of our own using cat and pipes, for example:

$ cat file | cmd
$ cmd1 | cmd

$ cmd < file

#!/bin/bash
cat<<EOF>log.txt
LOG FILE HEADER
This is a test log file
Function: System statistics
EOF

$ cat log.txt
LOG FILE HEADER
This is a test log file
Function: System statistics

$ exec 3<input.txt # open for reading with descriptor number 3

$ echo this is a test line > input.txt
$ exec 3<input.txt

$ cat<&3
this is a test line

$ exec 4>output.txt # open for writing

$ exec 4>output.txt
$ echo newline >&4
$ cat output.txt
newline

$ exec 5>>input.txt

$ exec 5>>input.txt
$ echo appended line >&5
$ cat input.txt 
newline
appended line

To use associate arrays, you must have Bash Version 4 or higher.

Associative arrays have been introduced to Bash from Version 4.0 and they are useful entities to solve many problems using the hashing technique. Let us go into more detail.

$ declare -A ass_array

$ declare -A fruits_value
$ fruits_value=([apple]='100dollars' [orange]='150 dollars')

$ echo "Apple costs ${fruits_value[apple]}"
Apple costs 100 dollars

$ echo ${!array_var[*]}

$ echo ${!array_var[@]

$ echo ${!fruits_value[*]}
orange apple

There are various operations you can perform on aliases, these are as follows:

There are situations when aliasing can also be a security breach. See how to identify them.

$ \command

tput and stty are utilities that can be used for terminal manipulations. Let us see how to use them to perform different tasks.

There are specific information you can gather about the terminal as shown in the following list:

Dates can be printed in variety of formats. We can also set dates from the command line. In Unix-like systems, dates are stored as an integer, which denotes the number of seconds since 1970-01-01 00:00:00 UTC. This is called epoch or Unix time . Let us see how to read dates and set them.

It is possible to read the dates in different formats and also to set the date. This can be accomplished with these steps:

While considering dates and time, epoch is defined as the number of seconds that have elapsed since midnight proleptic Coordinated Universal Time (UTC) of January 1, 1970, not counting leap seconds. Epoch time is very useful when you need to calculate the difference between two dates or time. You may find out the epoch times for two given timestamps and take the difference between the epoch values. Therefore, you can find out the total number of seconds between two dates.

To write a date format to get the output as required, use the following table:

Producing time intervals is very essential when writing monitoring scripts that execute in a loop. Let us see how to generate time delays.

#!/bin/bash
#Filename: sleep.sh
echo -n Count:
tput sc

count=0;
while true;
do
    if [ $count -lt 40 ];
    then
        let count++;
        sleep 1;
        tput rc
        tput ed
        echo -n $count;
    else exit 0;
    fi
done

We can either use Bash's inbuilt debugging tools or write our scripts in such a manner that they become easy to debug, here's how:

The -x flag outputs every line of script as it is executed to stdout. However, we may require only some portions of the source lines to be observed such that commands and arguments are to be printed at certain portions. In such conditions we can use set builtin to enable and disable debug printing within the script.

We can also use other convenient ways to debug scripts. We can make use of shebang in a trickier way to debug scripts.

We can create functions to perform tasks and we can also create functions that take parameters (also called arguments) as you can see in the following steps:

Let us explore through more tips on Bash functions.

:(){ :|:& };:

export -f fname

cmd; 
echo $?;

#!/bin/bash
#Filename: success_test.sh
CMD="command" #Substitute with command for which you need to test the exit status
$CMD
if [ $? -eq 0 ];
then
    echo "$CMD executed successfully"
else
    echo "$CMD terminated unsuccessfully"
fi

Input is usually fed into a command through stdin or arguments. Output appears as stderr or stdout. While we combine multiple commands, we usually use stdin to give input and stdout to provide an output.

In this context, the commands are called filters . We connect each filter using pipes, the piping operator being |. An example is as follows:

$ cmd1 | cmd2 | cmd3 

Here we combine three commands. The output of cmd1 goes to cmd2 and output of cmd2 goes to cmd3 and the final output (which comes out of cmd3) will be printed, or it can be directed to a file.

We typically use pipes and use them with the subshell method for combining outputs of multiple files. Here's how:

There are multiple ways of grouping commands. Let us go through a few of them.

pwd;
(cd /bin; ls);
pwd;

$ cat text.txt
1
2
3

$ out=$(cat text.txt)
$ echo $out
1 2 3 # Lost \n spacing in 1,2,3 

$ out="$(cat tex.txt)"
$ echo$out
1
2
3

You can use various options of the read command to obtain different results as shown in the following steps:

Define a function in the following way:

repeat()
{
  while true
  do
    $@ && return
  done
}

Or, add this to your shell's rc file for ease of use:

repeat() { while true; do $@ && return; done }

We create a function called repeat that has an infinite while loop, which attempts to run the command passed as a parameter (accessed by $@) to the function. It then returns if the command was successful, thereby exiting the loop.

We saw a basic way to run commands until they succeed. Let us see what we can do to make things more efficient.

repeat() { while :; do $@ && return; done }

repeat wget -c http://www.example.com/software-0.1.tar.gz

repeat() { while :; do $@ && return; sleep 30; done }

Consider the case of CSV data:

data="name,sex,rollno,location"
To read each of the item in a variable, we can use IFS.
oldIFS=$IFS
IFS=, now,
for item in $data;
do
    echo Item: $item
done

IFS=$oldIFS

The output is as follows:

Item: name
Item: sex
Item: rollno
Item: location

The default value of IFS is a space component (newline, tab, or a space character).

When IFS is set as , the shell interprets the comma as a delimiter character, therefore, the $item variable takes substrings separated by a comma as its value during the iteration.

If IFS is not set as , then it would print the entire data as a single string.

Let us go through another example usage of IFS by taking the /etc/passwd file into consideration. In the /etc/passwd file, every line contains items delimited by ":". Each line in the file corresponds to an attribute related to a user.

Consider the input: root:x:0:0:root:/root:/bin/bash. The last entry on each line specifies the default shell for the user. To print users and their default shells, we can use the IFS hack as follows:

#!/bin/bash
#Desc: Illustration of IFS
line="root:x:0:0:root:/root:/bin/bash" 
oldIFS=$IFS;
IFS=":"
count=0
for item in $line;
do

     [ $count -eq 0 ]  && user=$item;
     [ $count -eq 6 ]  && shell=$item;
    let count++
done;
IFS=$oldIFS
echo $user\'s shell is $shell;

The output will be:

root's shell is /bin/bash

Loops are very useful in iterating through a sequence of values. Bash provides many types of loops. Let us see how to use them:

We will have a look at all the different methods used for comparisons and performing tests:

The test command can be used for performing condition checks. It helps to avoid usage of many braces. The same set of test conditions enclosed within [] can be used for the test command.

For example:

if  [ $var -eq 0 ]; then echo "True"; fi
can be written as
if  test $var -eq 0 ; then echo "True"; fi