The regular expression syntax

Any experienced developer has undoubtedly used some kind of regular expression. For instance, in the operating system console, it's not uncommon to find the usage of the asterisk (*) or the question mark (?) to find files.

The question mark will match a single character with any value on a filename. For example, a pattern such as file?.xml will match file1.xml, file2.xml, and file3.xml, but it won't match file99.xml as the pattern expresses that anything that starts with file, followed by just one character of any value, and ends with .xml, will be matched.

A similar meaning is defined for asterisk (*). When asterisk is used, any number of characters with any value is accepted. In the case of file*.xml, anything that starts with file, followed by any number of characters of any value, and finishes with .xml, will be matched.

In this expression, we can find two kind of components: literals (file and .xml) and metacharacters (? or *). The regular expressions we will learn in this book are much more powerful than the simple patterns we can typically find on the operating system command line, but both can share one single definition:

A regular expression is a pattern of text that consists of ordinary characters (for example, letters a through z or numbers 0 through 9) and special characters known as metacharacters. This pattern describes the strings that would match when applied to a text.

Let's see our very first regular expression that will match any word starting with a:

Regex using literals and metacharacters

Note

Representation of regular expressions in this book

In the following figures of this book, regular expressions are going to be represented bounded by the / symbol. This is the QED demarcation that is followed in most of the text books. The code examples, however, won't use this notation.

On the other hand, even with monospaced font faces, the white spaces of a regular expression are difficult to count. In order to simplify the reading, every single whitespace in the figures will appear as .

The previous regular expression is again using literals and metacharacters. The literals here are and a, and the metacharacters are \ and w that match any alphanumeric character including underscore, and *, that will allow any number of repetitions of the previous character, and therefore, any number of repetitions of any word character, including underscore.

We will cover the metacharacters later in this chapter, but let's start by understanding the literals.

Literals

Literals are the simplest form of pattern matching in regular expressions. They will simply succeed whenever that literal is found.

If we apply the regular expression /fox/ to search the phrase The quick brown fox jumps over the lazy dog, we will find one match:

Searching using a regular expression

However, we can also obtain several results instead of just one, if we apply the regular expression /be/ to the following phrase To be, or not to be:

Multiple results searching with regex

We have just learned in the previous section that metacharacters can coexist with literals in the same expression. Because of this coexistence, we can find that some expressions do not mean what we intended. For example, if we apply the expression /(this is inside)/ to search the text this is outside (this is inside), we will find that the parentheses are not included in the result. This happens because parentheses are metacharacters and they have a special meaning.

Incorrectly unescaped metacharacters

We can use metacharacters as if they were literals. There are three mechanisms to do so:

• Escape the metacharacters by preceding them with a backslash.

• In python, use the re.escape method to escape non-alphanumeric characters that may appear in the expression. We will cover this in Chapter 2, Regular Expressions with Python.

• Quoting with \Q and \E: There is a third mechanism to quote in regular expressions, the quoting with \Q and \E. In the flavors that support them, it's as simple as enclosing the parts that have to be quoted with \Q (which starts a quote) and \E (which ends it).

However, this is not supported in Python at the moment.

Using the backslash method, we can convert the previous expression to /\(this is inside\)/ and apply it again to the same text to have the parentheses included in the result:

Escaped metacharacters in regex

In regular expressions, there are twelve metacharacters that should be escaped if they are to be used with their literal meaning:

• Backslash \

• Caret ^

• Dollar sign $

• Dot .

• Pipe symbol |

• Question mark ?

• Asterisk *

• Plus sign +

• Opening parenthesis (

• Closing parenthesis )

• Opening square bracket [

• The opening curly brace {

In some cases, the regular expression engines will do their best to understand if they should have a literal meaning even if they are not escaped; for example, the opening curly brace { will only be treated as a metacharacter if it's followed by a number to indicate a repetition, as we will learn later in this chapter.

Character classes

We are going to use a metacharacter for the first time to learn how to leverage the character classes. The character classes (also known as character sets) allow us to define a character that will match if any of the defined characters on the set is present.

To define a character class, we should use the opening square bracket metacharacter [, then any accepted characters, and finally close with a closing square bracket ]. For instance, let's define a regular expression that can match the word "license" in British and American English written form:

Searching using a character class

It is possible to also use the range of a character. This is done by leveraging the hyphen symbol (-) between two related characters; for example, to match any lowercase letter we can use [a-z]. Likewise, to match any single digit we can define the character set [0-9].

The character classes' ranges can be combined to be able to match a character against many ranges by just putting one range after the other—no special separation is required. For instance, if we want to match any lowercase or uppercase alphanumeric character, we can use [0-9a-zA-Z] (see next table for a more detailed explanation). This can be alternatively written using the union mechanism: [0-9[a-z[A-Z]]].

Element	Description
[	Matches the following set of characters
0-9	Matches anything between 0 and 9 (0, 1, 2, 3, 4, 5, 6, 7, 8, 9).
	Or
a-z	Matches anything between a and z (a, b, c, d, ..., z)
	Or
A-Z	Matches anything between A and Z (A, B, C, D, ..., Z)
]	End of character set

There is another possibility—the negation of ranges. We can invert the meaning of a character set by placing a caret (^) symbol right after the opening square bracket metacharacter ([). If we have a character class such as [0-9] meaning any digit, the negated character class [^0-9] will match anything that is not a digit. However, it is important to notice that there has to be a character that is not a digit; for example, /hello[^0-9]/ won't match the string hello because after the there has to be a non-digit character. There is a mechanism to do this—called negative lookahead—and it will be covered in Chapter 4, Look Around.

Predefined character classes

After using character classes for some time, it becomes clear that some of them are very useful and probably worthy of a shortcut.

Luckily enough, there are a number of predefined character classes that can be re-used and will be already known by other developers, making the expressions using them more readable.

These characters are not only useful as well-known shortcuts for typical character sets, but also have different meanings in different contexts. The character class \w, which matches any alphanumeric character, will match a different set of characters depending on the configured locale and the support of Unicode.

The following table shows the character classes supported at this moment in Python:

Element	Description (for regex with default flags)
.	This element matches any character except newline \n
\d	This matches any decimal digit; this is equivalent to the class [0-9]
\D	This matches any non-digit character; this is equivalent to the class [^0-9]
\s	This matches any whitespace character; this is equivalent to the class [\t\n\r\f\v]
\S	This matches any non-whitespace character; this is equivalent to the class [^ \t\n\r\f\v]
\w	This matches any alphanumeric character; this is equivalent to the class [a-zA-Z0-9_]
\W	This matches any non-alphanumeric character; this is equivalent to the class [^a-zA-Z0-9_]

Note

POSIX character classes in Python

The POSIX standard provides a number of character classes' denominations, for example, [:alnum:] for alphanumeric characters, [:alpha:] for alphabetic characters, or [:space:] for all whitespace characters, including line breaks.

All the POSIX character classes follow the same [:name:] notation, rendering them easily identifiable. However, they are not supported in Python at the moment.

If you come across one of them, you can implement the same functionality by leveraging the character classes' functionalities we just studied in this section. As an example, for an ASCII equivalent of [:alnum:] with an English locale, we can write [a-zA-Z0-9].

The first one from the previous table—the dot—requires special attention. The dot is probably one of the oldest and also one of the most used metacharacters. The dot can match any character except a newline character.

The reason to not match the newline is probably UNIX. In UNIX, the command-line tools usually worked line by line, and the regular expressions available at the moment were applied separately to those lines. Therefore, there were no newline characters to match.

Let's put the dot in practice by creating a regular expression that matches three characters of any value except newline:

/…/

Element	Description
.	Matches any character
.	Matches any character followed by the previous one
.	Matches any character followed by the previous one

The dot is a very powerful metacharacter that can create problems if it is not used moderately. In most of the cases where the dot is used, it could be considered overkill (or just a symptom of laziness when writing regular expressions).

To better define what is expected to be matched and to express more concisely to any ulterior reader what a regular expression is intended to do, the usage of character classes is much recommended. For instance, when working with Windows and UNIX file paths, to match any character except the slash or the backslash, you can use a negated character set:

[^\/\]

Element	Description
[	Matches a set of characters
^	Not matching this symbol's following characters
\/	Matches a / character
\	Matches a \ character
]	End of the set

This character set is explicitly telling you that we intend to match anything but a Windows or UNIX file path separator.

Alternation

We have just learned how to match a single character from a set of characters. Now, we are going to learn a broader approach: how to match against a set of regular expressions. This is accomplished using the pipe symbol |.

Let's start by saying that we want to match either if we find the word "yes" or the word "no". Using alternation, it will be as simple as:

/yes|no/

Element	Description
	Matches either of the following character sets
yes	The characters y, e, and s.
|	Or
no	The characters n and o.

On the other hand, if we want to accept more than two values, we can continue adding values to the alternation like this:

/yes|no|maybe/

Element	Description
	Matches either of the following character sets
yes	The literal "yes"
|	Or
no	The literal "no"
|	Or
maybe	The literal "maybe"

When using in bigger regular expressions, we will probably need to wrap our alternation inside parentheses to express that only that part is alternated and not the whole expression. For instance, if we make the mistake of not using the parentheses, as in the following expression:

/Licence: yes|no/

Element	Description
	Matches either of the following character sets
Licence: yes	The characters L, i, c, e, n, c, e, :, , y, e, and s
|	Or
no	The characters n and o.

We may think we are accepting either Licence: yes or Licence: no, but we are actually accepting either Licence: yes or no as the alternation has been applied to the whole regular expression instead of just the yes|no part. A correct approach for this will be:

Regular expression using alternation

Quantifiers

So far, we have learned how to define a single character in a variety of fashions. At this point, we will leverage the quantifiers—the mechanisms to define how a character, metacharacter, or character set can be repeated.

For instance, if we define that a \d can be repeated many times, we can easily create a form validator for the number of items field of a shopping cart (remember that \d matches any decimal digit). But let's start from the beginning, the three basic quantifiers: the question mark ?, the plus sign +, and the asterisk *.

Symbol	Name	Quantification of previous character
?	Question mark	Optional (0 or 1 repetitions)
*	Asterisk	Zero or more times
+	Plus sign	One or more times
{n,m}	Curly braces	Between n and m times

In the preceding table, we can find the three basic quantifiers, each with a specific utility. The question mark can be used to match the word car and its plural form cars:

/cars?/

Element	Description
car	Matches the characters c, a, r and s
s?	Optionally matches the character s

Note

In the previous example, the question mark is only applied to the character s and not to the whole word. The quantifiers are always applied only to the previous token.

Another interesting example of the usage of the question mark quantifier will be to match a telephone number that can be in the format 555-555-555, 555 555 555, or 555555555.

We now know how to leverage character sets to accept different characters, but is it possible to apply a quantifier to a character set? Yes, quantifiers can be applied to characters, character sets, and even to groups (a feature we will cover in Chapter 3, Grouping). We can construct a regular expression like this to validate the telephone numbers:

/\d+[-\s]?\d+[-\s]?\d+/

In the next table, we can find a detailed explanation of the preceding regular expression:

Element	Type	Description
\d	Predefined character set	Any decimal character
+	Quantifier	- that is repeated one or more times
[-\s]	Character set	A hyphen or whitespace character
?	Quantifier	- that may or may not appear
\d	Predefined character set	Any decimal character
+	Quantifier	- that is repeated one or more times
[-\s]	Character set	A hyphen or whitespace character
\d	Predefined character set	Any decimal character
+	Quantifier	- that is repeated one or more times

At the beginning of this section, one more kind of quantifier using the curly braces had been mentioned. Using this syntax, we can define that the previous character must appear exactly three times by appending it with {3}, that is, the expression \w{8} specifies exactly eight alphanumeric digits.

We can also define a certain range of repetitions by providing a minimum and maximum number of repetitions, that is, between three and eight times can be defined with the syntax {4,7}. Either the minimum or the maximum value can be omitted defaulting to 0 and infinite respectively. To designate a repetition of up to three times, we can use {,3}, we can also establish a repetition at least three times with {3,}.

Tip

Readability Tip

Instead of using {,1}, you can use the question mark. The same applies to {0,} for the asterisk * and {1,} for the plus sign +.

Other developers will expect you to do so. If you don't follow this practice, anyone reading your expressions will lose some time trying to figure out what kind of fancy stuff you tried to accomplish.

These four different combinations are shown in the next table:

Syntax	Description
{n}	The previous character is repeated exactly n times.
{n,}	The previous character is repeated at least n times.
{,n}	The previous character is repeated at most n times.
{n,m}	The previous character is repeated between n and m times (both inclusive).

Earlier in this chapter, we created a regular expression to validate telephone numbers that can be in the format 555-555-555, 555 555 555, or 555555555. We defined a regular expression to validate it using the metacharacter plus sign: /\d+[-\s]?\d+[-\s]?\d+/. It will require the digits (\d) to be repeated one or more times.

Let's fine-tune the regular expression by defining that the leftmost digit group can contain up to three characters, while the rest of the digit groups should contain exactly three digits:

Using quantifiers

Greedy and reluctant quantifiers

We still haven't defined what would match if we apply a quantifier such as this /".+"/ to a text such as the following: English "Hello", Spanish "Hola". We may expect that it matches "Hello" and "Hola" but it will actually match "Hello", Spanish "Hola".

This behavior is called greedy and is one of the two possible behaviors of the quantifiers in Python: greedy and non-greedy (also known as reluctant).

• The greedy behavior of the quantifiers is applied by default in the quantifiers. A greedy quantifier will try to match as much as possible to have the biggest match result possible.

• The non-greedy behavior can be requested by adding an extra question mark to the quantifier; for example, ??, *? or +?. A quantifier marked as reluctant will behave like the exact opposite of the greedy ones. They will try to have the smallest match possible.

Note

Possessive quantifiers

There is a third behavior of the quantifiers, the possessive behavior. This behavior is only supported by the Java and .NET flavors of the regular expressions at the moment.

They are represented with an extra plus symbol to the quantifier; for example, ?+, *+, or ++. Possessive quantifiers won't have further coverage in this book.

We can understand better how this quantifier works by looking at the next figure. We will apply almost the same regular expression (with the exception of leaving the quantifier as greedy or marking it as reluctant) to the same text, having two very different results:

Greedy and reluctant quantifiers

Boundary Matchers

Until this point, we have just tried to find out regular expressions within a text. Sometimes, when it is required to match a whole line, we may also need to match at the beginning of a line or even at the end. This can be done thanks to the boundary matchers.

The boundary matchers are a number of identifiers that will correspond to a particular position inside of the input. The following table shows the boundary matchers available in Python:

Matcher	Description
^	Matches at the beginning of a line
$	Matches at the end of a line
\b	Matches a word boundary
\B	Matches the opposite of \b. Anything that is not a word boundary
\A	Matches the beginning of the input
\Z	Matches the end of the input

These boundary matchers will behave differently in different contexts. For instance, the word boundaries (\b) will depend directly on the configured locale as different languages may have different word boundaries, and the beginning and end of line boundaries will behave differently based on certain flags that we will study in the next chapter.

Let's start working with boundary matchers by writing a regular expression that will match lines that start with "Name:". If you take a look at the previous table, you may notice the existence of the metacharacter ^ that expresses the beginning of a line. Using it, we can write the following expression:

/^Name:/

Element	Description
^	Matches the beginning of the line
N	Matches the followed by character N
a	Matches the followed by character a
m	Matches the followed by character m
e	Matches the followed by character e
:	Matches the followed by symbol colon

If we want to take one step further and continue using the caret and the dollar sign in combination to match the end of the line, we should take into consideration that from now on we are going to be matching against the whole line, and not just trying to find a pattern within a line.

Following the previous example, let's say that we want to make sure that after the name, there are only alphabetic characters or spaces until the end of the line. We will do this by matching the whole line until the end by setting a character set with the accepted characters and allowing their repetition any number of times until the end of the line.

/^Name:[\sa-zA-Z]+$/

Element	Description
^	Matches the beginning of the line.
N	Matches the followed by character N.
a	Matches the followed by character a.
m	Matches the followed by character m.
e	Matches the followed by character e.
:	Matches the followed by colon symbol.
[\sa-zA-Z]	Then matches the followed by whitespace, or any alphabetic lowercase or uppercase character.
+	The character can be repeated one or more times.
$	Until the end of the line.

Another outstanding boundary matcher is the word boundary \b. It will match any character that is not a word character (in the configured locale), and therefore, any potential word boundary. This is very useful when we want to work with isolated words and we don't want to create character sets with every single character that may divide our words (spaces, commas, colons, hyphens, and so on). We can, for instance, make sure that the word hello appears in a text by using the following regular expression:

/\bhello\b/

Element	Description
\b	Matches a word boundary.
h	Matches the followed by character h.
e	Matches the followed by character e.
l	Matches the followed by character l.
l	Matches the followed by character l.
o	Matches the followed by character o.
\b	Then matches another followed by word boundary.

As an exercise, we could think why the preceding expression is better than /hello/. The reason is that this expression will match an isolated word instead of a word containing "hello", that is, /hello/ will easily match hello, helloed, or Othello; while /\bhello\b/ will only match hello.