In this section, you will learn about regular expressions and why we use them. Then we will discuss the usage of regular expressions in AWK.

A regular expression, or regexpr, is a set of characters used to describe a pattern. A regular expression is generally used to match lines in a file that contain a particular pattern. Many Unix utilities operate on plain text files line by line, such as grep, sed, and awk. Regular expressions search for a pattern on a single line in a file.

Regular expressions are made up of two types of characters: normal text characters, called literals, and special characters, such as the asterisk (*, +, ?, .), called metacharacters. There are times when you want to match a metacharacter as a literal character. In such cases, we prefix that metacharacter with a backslash (\), which is called an escape sequence.

The basic regular expression construct can be summarized as follows:

Here is the list of metacharacters, also known as special characters, that are used in building regular expressions:

\ ^ $ . [ ] | ( ) * + ?

The following table lists the remaining elements that are used in building a basic regular expression, apart from the metacharacters mentioned before:

When we use special characters in regular expressions, they are called metacharacters because they have a special meaning. They enhance the flexibility, power, and usage of regular expressions. Now, we will discuss the list of metacharacters in regular expressions. All characters, that are not either metacharacters or escape sequences, match themselves in regular expressions. Let's understand each of them in detail.

A regular expression consisting of a literal or character (letter or digit) is a basic regular expression that matches itself. Sometimes, if we need to use the literal meaning of a metacharacter in a regular expression, we precede it with...

In regular expressions, the '|' alternation operator has the lowest precedence, then concatenation, followed by the +, *, and ? repetition operations as well as brackets, '{' and '}'. As in arithmetic expressions, operators of higher precedence are done before lower ones. Also, parentheses can change how operators are grouped. Due to these conventions, parentheses are often omitted, as follows:

$ awk '/Jane|Emily/' emp.dat is same as
$ awk '/(Jane)|(Emily)/' emp.dat

The output on execution of the preceding code is as follows:

Jane    Kaur    9837432312  jane@gmail.com      F   hr      1800
Emily   Kaur    8826175812  emily@gmail.com     F   Ops     2100

In POSIX, AWK, and GAWK, the '*', '+', and '?' operators stand for themselves when there is nothing in the regular...

GAWK has some more regular expression operators, in addition to those we've already discussed. The operators described in this section are specific to GAWK and are not available in other AWK implementations. These are basically shorthand character classes that are frequently used, and most of them deal with word matching (here, word refers to a sequence of one or more letters, digits, or underscores). We access them by using the escape character, '\', followed by a letter (here, the escape character adds special meaning to them instead of taking it away).

Let's create a file to practice GAWK-only operators:

vi  gopr.txt
start
bigstarting
starting
start end
white space example
e d
end
e_d
e1d
e;d
eNd
white space example with tab
e d

In bracket expressions, we can use both lowercase and uppercase characters to match the characters for case-insensitive results, but the same is not true for other regular expressions. However, this becomes difficult if you have to match to many characters. Here comes the two built-in string functions of AWK, tolower() and toupper(). Using these two functions, we can perform case-insensitive match operations.

For example, to match the first name 'John' of an employee in the emp.dat file, we can perform a case-insensitive match using the tolower() function, as follows:

$ awk 'tolower($1) ~ /john/' emp.dat

The output on execution of this code is as follows:

John    Kapur   9911556789  john@gmail.com      M   hr      2200

Another way to perform a case-insensitive match operation is by setting the IGNORECASE bash variable value to TRUE.

When using regular expressions and strings, certain characters cannot be included literally in string constants ("hello") or regular expression constants (/hello/). To specify the characters for which there may not be other notations, they are represented as escape sequences. Escape sequences begin with a backslash ('\'). For example, it can represent that an \n newline character is used, which otherwise cannot be represented in strings or regular expressions. Similarly, \b stands for backspace, \t stands for tab, and \/ represents a forward slash. They are quite useful in generating formatted output.

For example, to insert a new line after each line of the emp.dat file, we can use the \n escape sequence, as follows:

$ awk '{ print $0"\n" }' emp.dat

The output on execution of the given code is as follows:

Jack    Singh ...

In this chapter, we learned about regular expressions. We learned the regular expression construct using literals and metacharacters. Then, we learned the usage of various different regular expression metacharacters, which are also known as regular expression operators. Finally, we covered some GAWK-specific regular expression metacharacter usage and understood escape sequences and their role in regular expressions.

In the next chapter, we will learn about handling AWK variables and constants while writing AWK programs.