Unix-like systems have the best command-line tools. Each command performs a simple function to make our work easier. These simple functions can be combined with other commands to solve complex problems. Combining simple commands is an art; you will get better at it as you practice and gain experience. This chapter introduces some of the most interesting and useful commands, including grep, awk, sed, and find.

The cat command displays or concatenates the contents of a file, but cat is capable of more. For example, cat can combine standard input data with data from a file. One way of combining the stdin data with file data is to redirect stdin to a file and then append two files. The cat command can do this in a single invocation. The next recipes show basic and advanced usages of cat.

The cat command is a simple and frequently used command and it stands for conCATenate.

The general syntax of cat for reading contents is as follows:

$ cat file1 file2 file3 ...

This command concatenates data from the files specified as command-line arguments and sends that data to stdout.

• To print contents of a single file, execute the following command...

Recording a screen session as a video is useful, but a video is an overkill for debugging terminal sessions or providing a shell tutorial.

The shell provides another option. The script command records your keystrokes and the timing of keystrokes as you type, and saves your input and the resulting output in a pair of files. The scriptreplay command will replay the session.

The script and scriptreplay commands are available in most GNU/Linux distributions. You can create tutorials of command-line hacks and tricks by recording the terminal sessions. You can also share the recorded files for others to playback and see how to perform a particular task with the command line. You can even invoke other interpreters...

The find command is one of the great utilities in the Unix/Linux command-line toolbox. It is useful both at the command line and in shell scripts. Like cat and ls, find has many features, and most people do not use it to its fullest. This recipe deals with some common ways to utilize find to locate files.

The find command uses the following strategy:  find descends through a hierarchy of files, matches files that meet the specified criteria, and performs some actions. The default action is to print the names of files and folders, which can be specified with the -print option.

Unix commands accept data either from the standard input (stdin) or as command line arguments. Previous examples have shown how to pass data from one application's standard output to another's standard input with a pipe.

We can invoke applications that accept command-line arguments in other ways. The simplest is to use the back-tic symbol to run a command and use its output as a command line:

$ gcc `find '*.c'`

This solution works fine in many situations, but if there are a lot of files to be processed, you'll see the dreaded Argument list too long error message. The xargs program solves this problem.

The xargs command reads a list of arguments from stdin and executes a command using these arguments in the command line. The xargs command can also convert any one-line or multiple-line text inputs into other formats, such as multiple lines (specified number of columns)...

The tr command is a versatile tool in the Unix command–warrior's kit. It is used to craft elegant one-liner commands. It performs substitution of characters, deletes selected characters, and can squeeze repeated characters from the standard input. Tr is short for translate, since it translates a set of characters to another set. In this recipe, we will see how to use tr to perform basic translation between sets.

The tr command accepts input through stdin (standard input) and cannot accept input through command-line arguments. It has this invocation format:

tr [options] set1 set2

Input characters from stdin are mapped from the first character in set1 to the first character in set2, and so on and the output is...

Checksum programs are used to generate a relatively small unique key from files. We can recalculate the key to confirm that a file has not changed. Files may be modified deliberately (adding a new user changes the password file), accidentally (a data read error from a CD-ROM drive), or maliciously (a virus is inserted). Checksums let us verify that a file contains the data we expect it to.

Checksums are used by backup applications to check whether a file has been modified and needs to be backed up.

Most software distributions also have a checksum file available. Even robust protocols such as TCP can allow a file to be modified in transit. Hence, we need to know whether the received file is the original one or not by applying some kind of test.

By comparing the checksum of the file we downloaded with the checksum calculated by the distributer, we can verify that the received file is...

Encryption techniques are used to protect data from unauthorized access. Unlike the checksum algorithms we just discussed, encryption programs can reconstruct the original data with no loss. There are many algorithms available and we will discuss those most commonly used in the Linux/Unix world.

Let's see how to use tools such as crypt, gpg, and base64:

• The crypt command is not commonly installed on Linux systems. It's a simple and relatively insecure cryptographic utility that accepts input from stdin, requests a passphrase, and sends encrypted output to stdout:

        $ crypt <input_file >output_file
        Enter passphrase:

We can provide a passphrase on the command line:

        $ crypt...

Sorting text files is a common task. The sort command sorts text files and stdin. It can be coupled with other commands to produce the required output. uniq is often used with sort to extract unique (or duplicate) lines. The following recipes illustrate some sort and uniq use cases.

The sort and uniq commands accept input as filenames or from stdin (standard input) and output the result by writing to stdout.

1. We can sort a set of files (for example, file1.txt and file2.txt), like this:

        $ sort file1.txt file2.txt > sorted.txt

 Alternatively...

Shell scripts often need to store temporary data. The most suitable location to do this is /tmp (which will be cleaned out by the system on reboot). There are two methods to generate standard filenames for temporary data.

The mktemp command will create a unique temporary file or folder name:

1. Create a temporary file:

        $ filename=`mktemp`
        $ echo $filename
        /tmp/tmp.8xvhkjF5fH

This creates a temporary file, stores the name in filename, and then displays the name.

2. Create a temporary directory:

        $ dirname=`mktemp -d`
        $ echo $dirname
        tmp.NI8xzW7VRX

This creates a temporary directory, stores the name in filename, and displays the name.

Splitting a large file into smaller pieces is sometimes necessary. Long ago, we had to split files to transport large datasets on floppy disks. Today, we split files for readability, for generating logs, or for working around size-restrictions on e-mail attachments. These recipes will demonstrate ways of splitting files in different chunks.

The split command was created to split files. It accepts a filename as an argument and creates a set of smaller files in which the first part of the original file is in the alphabetically first new file, the next set in the alphabetically next file, and so on.

For example, a 100 KB file can be divided into smaller files of 10k each by specifying the split size. The split command supports...

Many shell scripts perform actions that involve modifying filenames. They may need to rename the files and preserve the extension, or convert files from one format to another and change the extension, while preserving the name, extracting a portion of the filename, and so on.

The shell has built-in features for manipulating filenames.

The % operator will extract the name from name.extension. This example extracts sample from sample.jpg:

file_jpg="sample.jpg" 
name=${file_jpg%.*} 
echo File name is: $name 

The output is this:

File name is: sample

The # operator will extract the extension:

Extract .jpg from the filename stored in the file_jpg variable:

extension=${file_jpg#*.} 
echo Extension is...

We frequently need to move and perhaps rename a set of files. System housekeeping often requires moving files with a common prefix or file type to a new folder. Images downloaded from a camera may need to be renamed and sorted. Music, video, and e-mail files all need to be reorganized eventually.

There are custom applications for many of these operations, but we can write our own custom scripts to do it our way.

Let's see how to write scripts to perform these kinds of operation.

The rename command changes filenames using Perl regular expressions. By combining the find, rename, and mv commands, we can perform a lot of things.

Most Linux distributions include a dictionary file. However, very few people are aware of this, thus spelling errors abound. The aspell command-line utility is a spell checker. Let's go through a few scripts that make use of the dictionary file and the spell checker.

The /usr/share/dict/ directory contains one or perhaps more dictionary files, which are text files with a list of words. We can use this list to check whether a word is a dictionary word or not:

$ ls /usr/share/dict/ 
american-english  british-english

To check whether the given word is a dictionary word, use the following script:

#!/bin/bash 
#Filename: checkword.sh 
word=$1 
grep "^$1$" /usr/share/dict/british-english...

We looked at commands that accept arguments on the command line. Linux also supports many interactive applications ranging from passwd to ssh.

We can create our own interactive shell scripts. It's easier for casual users to interact with a set of prompts rather than remember command line flags and the proper order. For instance, a script to back up a user's work, but not to back up and lock files, might look like this:

$ backupWork.sh

• What folder should be backed up? notes
• What type of files should be backed up? .docx

Automating interactive applications can save you time when you need to rerun the same application and frustration while you're developing one.

The first step to automating a task...

Computing power constantly increases not only because processors have higher clock cycles but also because they have multiple cores. This means that in a single hardware processor there are multiple logical processors. It's like having several computers, instead of just one.

However, multiple cores are useless unless the software makes use of them. For example, a program that does huge calculations may only run on one core while the others will sit idle. The software has to be aware and take advantage of the multiple cores if we want it to be faster.

In this recipe, we will see how we can make our commands run faster.

Let's take an example of the md5sum command we discussed in the previous...

One of the commonest problems we deal with is finding misplaced files and sorting out mangled file hierarchies. This section will discuss tricks for examining a portion of the filesystem and presenting the contents.

The find command and loops we discussed give us tools to examine and report details in a directory and its contents.

The next recipes show two ways to examine a directory. First we'll display the hierarchy as a tree, then we'll see how to generate a summary of files and folders under a directory.