Linux Shell Scripting Cookbook, Second Edition: Don't neglect the shell – this book will empower you to use simple commands to perform complex tasks. Whether you're a casual or advanced Linux user, the cookbook approach makes it all so brilliantly accessible and, above all, useful. , Second Edition

Unix-like systems have the privilege of having the best command-line tools. They help us achieve many tasks making our work easier. While each command has a specific focus, with practice you'll be able to solve complex problems by combining two or more commands. Some frequently used commands are grep, awk, sed, and find.

Mastering the Unix/Linux command line is an art; you will get better at using it as you practice and gain experience. This chapter will introduce you to some of the most interesting and useful commands.

cat is one of the first commands that a command-line warrior must learn. It is usually used to read, display, or concatenate the contents of a file, but cat is capable of more than just that. We even scratch our heads when we need to combine standard input data, as well as data from a file using a single-line command. The regular way of combining the stdin data, as well as file data, is to redirect stdin to a file and then append two files. But we can use the cat command to do it easily in a single invocation. In this recipe we will see basic and advanced usages of cat.

$ cat file1 file2 file3 ...

When you need to show somebody how to do something in the terminal, or you need to prepare a tutorial on how to do something through the command line, you would normally type the commands manually and show them. Or, you could record a screencast and playback the video to them. There are other options for doing this. Using the commands script and scriptreplay, we can record the order and timing of the commands and save the data to text files. Using these files, others can replay and see the output of the commands on the terminal until the playback is complete.

find is one of the great utilities in the Unix/Linux command-line toolbox. It is a very useful command for shell scripts; however, many people do not use it to its fullest effectiveness. This recipe deals with most of the common ways to utilize find to locate files.

$ find base_path

$ find . -print
# Print lists of files and folders

We use pipes to redirect stdout (standard output) of a command to stdin (standard input) of another command. For example:

cat foo.txt | grep "test"

Some of the commands accept data as command-line arguments rather than a data stream through stdin (standard input). In that case, we cannot use pipes to supply data through command-line arguments.

We should try alternate methods. xargs is a command that is very helpful in handling standard input data to the command-line argument conversions. It can manipulate stdin and convert to command-line arguments for the specified command. Also, xargs can convert any one-line or multiple-line text inputs into other formats, such as multiple lines (specified number of columns) or a single line and vice versa.

All Bash users love one-liner commands, which are command sequences that are joined by using the pipe operator, but do not use the semicolon terminator (;) between the commands used. Crafting one-line commands makes tasks more efficient...

tr is a small and beautiful command in the Unix command-warrior toolkit. It is one of the important commands frequently used to craft beautiful one-liner commands. It can be used to perform substitution of characters, deletion of the characters, and squeezing of repeated characters from the standard input. It is often called translate , since it can translate a set of characters to another set. In this recipe we will see how to use tr to perform basic translation between sets.

tr [options] set1 set2

Checksum programs are used to generate checksum key strings from the files and verify the integrity of the files later by using that checksum string. A file might be distributed over the network or any storage media to different destinations. Due to many reasons, there are chances of the file being corrupted due to a few bits missing during the data transfer by different reasons. These errors happen most often while downloading the files from the Internet, transferring through a network, CD-ROM damage, and so on.

Hence, we need to know whether the received file is the correct one or not by applying some kind of test. The special key string that is used for this file integrity test is known as a checksum.

We calculate the checksum for the original file as well as the received file. By comparing both of the checksums, we can verify whether the received file is the correct one or not. If the checksums (calculated from the original file at the source location and the...

Encryption techniques are used mainly to protect data from unauthorized access. There are many algorithms available and we have discussed the most commonly used ones. There are a few tools available in a Linux environment for performing encryption and decryption. Sometimes we use encryption algorithm hashes for verifying data integrity. This section will introduce a few commonly used cryptographic tools and a general set of algorithms that these tools can handle.

Sorting is a common task that we can encounter with text files. The sort command helps us to perform sort operations over text files and stdin. Most often, it can also be coupled with many other commands to produce the required output. uniq is another command that is often used along with a sort command. It helps to extract unique (or duplicate) lines from a text or stdin. This recipe illustrates most of the use cases with sort and uniq commands.

While writing shell scripts, we often need to store temporary data. The most suitable location to store temporary data is /tmp (which will be cleaned out by the system on reboot). We can use two methods to generate standard filenames for temporary data.

Splitting of files into many smaller pieces becomes essential in certain situations. Earlier, when memory was limited with devices such as floppy disks, it was crucial to split files into smaller file sizes to split files across many disks. However, nowadays we split files for other purposes, such as readability, for generating logs, sending files over e-mail, and so on. In this recipe we will see various ways of splitting files in different chunks.

$ split -b 10k data.file
$ ls
data.file  xaa  xab  xac  xad  xae  xaf  xag  xah  xai  xaj

Several shell scripts perform manipulations based on filenames. We may need to perform actions such as renaming the files by preserving the extension, converting files from one format to another (change the extension by preserving the name), extracting a portion of the filename, and so on. The shell comes with inbuilt features for slicing filenames based on different conditions. Let us see how to do it.

file_jpg="sample.jpg"
name=${file_jpg%.*}
echo File name is: $name

File name is: sample

extension=${file_jpg#*.}
echo Extension is: jpg

Extension is: jpg

Renaming a number of files is one of the tasks we frequently come across. A simple example is when you download photos from your digital camera to your computer you may delete unnecessary files and it causes discontinuous numbering of image files. Sometimes, you may need to rename them with a custom prefix and continuous numbering for filenames. We sometimes use third-party tools for performing rename operations. We can use Bash commands to perform a rename operation in a couple of seconds.

Moving all the files having a particular substring in their filenames (for example, the same prefix for filenames) or with a specific file type to a given directory is another use case we frequently perform. Let's see how to write scripts to perform these kinds of operations.

Most of the Linux distributions come with a dictionary file along with them. However, I find very few people to be aware of the dictionary file and hence, few make use of them. There is a command-line utility called aspell that functions as a spell checker. Let's go through a few scripts that make use of the dictionary file and the spell checker.

$ ls /usr/share/dict/ 
american-english  british-english

#!/bin/bash
#Filename: checkword.sh
word=$1
grep "^$1$" /usr/share/dict/british-english -q 
if [ $? -eq 0 ]; then
  echo $word is a dictionary word;
else
  echo $word is not a dictionary word;
fi

$ ./checkword.sh ful ...

Automating interactive input for command-line utilities are extremely useful for writing automation tools or testing tools. There will be many situations when we deal with commands that read input interactively. An example of executing a command and supplying the interactive input is as follows:

$ command
Enter a number: 1
Enter name : hello
You have entered 1,hello

1[Return]hello[Return]

"1\nhello\n"

Computing power has increased a lot over the last couple of years. However, this is not just because of having processors with higher clock cycles; the thing that makes modern processors faster is multiple cores. What this means to the user is in a single hardware processor there are multiple logical processors.

However, the multiple cores are useless unless the software makes use of them. For example, if you have a program that does huge calculations, it will only run on one of the cores, the others will sit idle. The software has to be aware and take advantage of the multiple cores if we want it to be faster.

In this recipe we will see how we can make our commands run faster.