Modern Python Cookbook

Statements and Syntax

Python syntax is designed to be simple. There are a few rules; we'll look at some of the interesting statements in the language as a way to understand those rules. Concrete examples can help clarify the language's syntax.

We'll cover some basics of creating script files first. Then we'll move on to looking at some of the more commonly-used statements. Python only has about 20 or so different kinds of imperative statements in the language. We've already looked at two kinds of statements in Chapter 1, Numbers, Strings, and Tuples, the assignment statement and the expression statement.

When we write something like this:

>>> print("hello world")
hello world

We're actually executing a statement that contains only the evaluation of a function, print(). This kind of statement—where we evaluate a function or a method of an object—is common.

The other kind of statement we've already seen is the assignment statement. Python has many variations on this theme. Most of the time, we're assigning a single value to a single variable. Sometimes, however, we might be assigning two variables at the same time, like this:

quotient, remainder = divmod(355, 113)

These recipes will look at some of the more common of the complex statements, including if, while, for, try, and with. We'll touch on a few of the simpler statements as we go, like break and raise.

In this chapter, we'll look at the following recipes:

Writing Python script and module files - syntax basics
Writing long lines of code
Including descriptions and documentation
Better RST markup in docstrings
Designing complex if...elif chains
Saving intermediate results with the := "walrus"
Avoiding a potential problem with break statements
Leveraging exception matching rules
Avoiding a potential problem with an except: clause
Concealing an exception root cause
Managing a context using the with statement

We'll start by looking at the big picture – scripts and modules – and then we'll move down into details of individual statements. New with Python 3.8 is the assignment operator, sometimes called the "walrus" operator. We'll move into exception handling and context management as more advanced recipes in this section.

Writing Python script and module files – syntax basics

We'll need to write Python script files in order to do anything that's fully automated. We can experiment with the language at the interactive >>> prompt. We can also use JupyterLab interactively. For automated work, however, we'll need to create and run script files.

How can we make sure our code matches what's in common use? We need to look at some common aspects of style: how we organize our programming to make it readable.

We'll also look at a number of more technical considerations. For example, we need to be sure to save our files in UTF-8 encoding. While ASCII encoding is still supported by Python, it's a poor choice for modern programming. We'll also need to be sure to use spaces instead of tabs. If we use Unix newlines as much as possible, we'll also find it slightly simpler to create software that runs on a variety of operating systems.

Most text editing tools will work properly with Unix (newline) line endings as well as Windows or DOS (return-newline) line endings. Any tool that can't work with both kinds of line endings should be avoided.

Getting ready

To edit Python scripts, we'll need a good programming text editor. Python comes with a handy editor, IDLE. It works well for simple projects. It lets us jump back and forth between a file and an interactive >>> prompt, but it's not a good programming editor for larger projects.

There are dozens of programming editors. It's nearly impossible to suggest just one. So we'll suggest a few.

The JetBrains PyCharm editor has numerous features. The community edition version is free. See https://www.jetbrains.com/pycharm/download/.

ActiveState has Komodo IDE, which is also very sophisticated. The Komodo Edit version is free and does some of the same things as the full Komodo IDE. See http://komodoide.com/komodo-edit/.

Notepad++ is good for Windows developers. See https://notepad-plus-plus.org.

BBEdit is very nice for macOS X developers. See http://www.barebones.com/products/bbedit/.

For Linux developers, there are several built-in editors, including VIM, gedit, and Kate. These are all good. Since Linux tends to be biased toward developers, the editors available are all suitable for writing Python.

What's important is that we'll often have two windows open while we're working:

The script or file that we're working on in our editor of choice.
Python's >>> prompt (perhaps from a shell or perhaps from IDLE) where we can try things out to see what works and what doesn't. We may be creating our script in Notepad++ but using IDLE to experiment with data structures and algorithms.

We actually have two recipes here. First, we need to set some defaults for our editor. Then, once the editor is set up properly, we can create a generic template for our script files.

How to do it...

First, we'll look at the general setup that we need to do in our editor of choice. We'll use Komodo examples, but the basic principles apply to all editors. Once we've set the edit preferences, we can create our script files:

Open your editor of choice. Look at the preferences page for the editor.
Find the settings for preferred file encoding. With Komodo Edit Preferences, it's on the Internationalization tab. Set this to UTF-8.
Find the settings for indentation. If there's a way to use spaces instead of tabs, check this option. With Komodo Edit, we actually do this backward—we uncheck "prefer spaces over tabs." Also, set the spaces per indent to four. That's typical for Python code. It allows us to have several levels of indentation and still keep the code fairly narrow.

The rule is this: we want spaces; we do not want tabs.

Once we're sure that our files will be saved in UTF-8 encoding, and we're also sure we're using spaces instead of tabs, we can create an example script file:

The first line of most Python script files should look like this:
```
#!/usr/bin/env python3
```
This sets an association between the file you're writing and Python.

For Windows, the filename-to-program association is done through a setting in one of the Windows control panels. Within the Default Programs control panel, there's a panel to Set Associations. This control panel shows that .py files are bound to the Python program. This is normally set by the installer, and we rarely need to change it or set it manually.

Windows developers can include the preamble line anyway. It will make macOS X and Linux folks happy when they download the project from GitHub.
After the preamble, there should be a triple-quoted block of text. This is the documentation string (called a docstring) for the file we're going to create. It's not technically mandatory, but it's essential for explaining what a file contains:
```
"""
A summary of this script.
""" 
```
Because Python triple-quoted strings can be indefinitely long, feel free to write as much as necessary. This should be the primary vehicle for describing the script or library module. This can even include examples of how it works.
Now comes the interesting part of the script: the part that really does something. We can write all the statements we need to get the job done. For now, we'll use this as a placeholder:
```
print('hello world')
```
This isn't much, but at least the script does something. In other recipes, we'll look at more complex processing. It's common to create function and class definitions, as well as to write statements to use the functions and classes to do things.

For our first, simple script, all of the statements must begin at the left margin and must be complete on a single line. There are many Python statements that have blocks of statements nested inside them. These internal blocks of statements must be indented to clarify their scope. Generally—because we set indentation to four spaces—we can hit the Tab key to indent.

Our file should look like this:

#!/usr/bin/env python3
"""
My First Script: Calculate an important value.
"""
print(355/113)

How it works...

Unlike other languages, there's very little boilerplate in Python. There's only one line of overhead and even the #!/usr/bin/env python3 line is generally optional.

Why do we set the encoding to UTF-8? While the entire language is designed to work using just the original 128 ASCII characters, we often find that ASCII is limiting. It's easier to set our editor to use UTF-8 encoding. With this setting, we can simply use any character that makes sense. We can use characters like as Python variables if we save our programs in UTF-8 encoding.

This is legal Python if we save our file in UTF-8:

 = 355/113
print()

It's important to be consistent when choosing between spaces and tabs in Python. They are both more or less invisible, and mixing them can easily lead to confusion. Spaces are suggested.

When we set up our editor to use a four-space indent, we can then use the button labeled Tab on our keyboard to insert four spaces. Our code will align properly, and the indentation will show how our statements nest inside each other.

The initial #! line is a comment. Because the two characters are sometimes called sharp and bang, the combination is called "shebang." Everything between a # and the end of the line is ignored. The Linux loader (a program named execve) looks at the first few bytes of a file to see what the file contains. The first few bytes are sometimes called magic because the loader's behavior seems magical. When present, this two-character sequence of #! is followed by the path to the program responsible for processing the rest of the data in the file. We prefer to use /usr/bin/env to start the Python program for us. We can leverage this to make Python-specific environment settings via the env program.

There's more...

The Python Standard Library documents are derived, in part, from the documentation strings present in the module files. It's common practice to write sophisticated docstrings in modules. There are tools like pydoc and Sphinx that can reformat the module docstrings into elegant documentation. We'll look at this in other recipes.

Additionally, unit test cases can be included in the docstrings. Tools like doctest can extract examples from the document string and execute the code to see if the answers in the documentation match the answers found by running the code. Most of this book is validated with doctest.

Triple-quoted documentation strings are preferred over # comments. While all text between # and the end of the line is ignored, this is limited to a single line, and it is used sparingly. A docstring can be of indefinite size; they are used widely.

Prior to Python 3.6, we might sometimes see this kind of thing in a script file:

color = 355/113  # type: float

The # type: float comment can be used by a type inferencing system to establish that the various data types can occur when the program is actually executed. For more information on this, see Python Enhancement Proposal (PEP) 484: https://www.python.org/dev/peps/pep-0484/.

The preferred style is this:

color: float = 355/113

The type hint is provided immediately after the variable name. This is based on PEP 526, https://www.python.org/dev/peps/pep-0526. In this case, the type hint is obvious and possibly redundant. The result of exact integer division is a floating-point value, and type inferencing tools like mypy are capable of figuring out the specific type for obvious cases like this.

There's another bit of overhead that's sometimes included in a file. The VIM and gedit editors let us keep edit preferences in the file. This is called a modeline. We may see these; they can be ignored. Here's a typical modeline that's useful for Python:

# vim: tabstop=8 expandtab shiftwidth=4 softtabstop=4

This sets the Unicode u+0009 TAB characters to be transformed to eight spaces; when we hit the Tab key, we'll shift four spaces. This setting is carried in the file; we don't have to do any VIM setup to apply these settings to our Python script files.

Writing long lines of code

There are many times when we need to write lines of code that are so long that they're very hard to read. Many people like to limit the length of a line of code to 80 characters or fewer. It's a well-known principle of graphic design that a narrower line is easier to read. See http://webtypography.net/2.1.2 for a deeper discussion of line width and readability.

While shorter lines are easier on the eyes, our code can refuse to cooperate with this principle. Long statements are a common problem. How can we break long Python statements into more manageable pieces?

Getting ready

Often, we'll have a statement that's awkwardly long and hard to work with. Let's say we've got something like this:

>>> import math
>>> example_value = (63/25) * (17+15*math.sqrt(5)) / (7+15*math.sqrt(5))
>>> mantissa_fraction, exponent = math.frexp(example_value)
>>> mantissa_whole = int(mantissa_fraction*2**53)
>>> message_text = f'the internal representation is {mantissa_whole:d}/2**53*2**{exponent:d}'
>>> print(message_text)
the internal representation is 7074237752514592/2**53*2**2

This code includes a long formula, and a long format string into which we're injecting values. This looks bad when typeset in a book; the f-string line may be broken incorrectly. It looks bad on our screen when trying to edit this script.

We can't haphazardly break Python statements into chunks. The syntax rules are clear that a statement must be complete on a single logical line.

The term "logical line" provides a hint as to how we can proceed. Python makes a distinction between logical lines and physical lines; we'll leverage these syntax rules to break up long statements.

How to do it...

Python gives us several ways to wrap long statements so they're more readable:

We can use \ at the end of a line to continue onto the next line.
We can leverage Python's rule that a statement can span multiple logical lines because the (), [], and {} characters must balance. In addition to using () or \, we can also exploit the way Python automatically concatenates adjacent string literals to make a single, longer literal; ("a" "b") is the same as "ab".
In some cases, we can decompose a statement by assigning intermediate results to separate variables.

We'll look at each one of these in separate parts of this recipe.

Using a backslash to break a long statement into logical lines

Here's the context for this technique:

>>> import math
>>> example_value = (63/25) * (17+15*math.sqrt(5)) / (7+15*math.sqrt(5))
>>> mantissa_fraction, exponent = math.frexp(example_value)
>>> mantissa_whole = int(mantissa_fraction*2**53)

Python allows us to use \ to break the logical line into two physical lines:

Write the whole statement on one long line, even if it's confusing:

>>> message_text = f'the internal representation is {mantissa_whole:d}/2**53*2**{exponent:d}'

If there's a meaningful break, insert the \ to separate the statement:

>>> message_text = f'the internal representation is \
...     {mantissa_whole:d}/2**53*2**{exponent:d}'

For this to work, the \ must be the last character on the line. We can't even have a single space after the \. An extra space is fairly hard to see; for this reason, we don't encourage using back-slash continuation like this. PEP 8 provides guidelines on formatting and discourages this.

In spite of this being a little hard to see, the \ can always be used. Think of it as the last resort in making a line of code more readable.

Using the () characters to break a long statement into sensible pieces

Write the whole statement on one line, even if it's confusing:

>>> import math
>>> example_value1 = (63/25) * (17+15*math.sqrt(5)) / (7+15*math.sqrt(5))

Add the extra () characters, which don't change the value, but allow breaking the expression into multiple lines:

>>> example_value2 = (63/25) * ( (17+15*math.sqrt(5)) / (7+15*math.sqrt(5)) )
>>> example_value2 == example_value1
True

Break the line inside the () characters:

>>> example_value3 = (63/25) * (
...      (17+15*math.sqrt(5))
...    / (7+15*math.sqrt(5))
... )
>>> example_value3 == example_value1
True

The matching () character's technique is quite powerful and will work in a wide variety of cases. This is widely used and highly recommended.

We can almost always find a way to add extra () characters to a statement. In rare cases when we can't add () characters, or adding () characters doesn't improve readability, we can fall back on using \ to break the statement into sections.

Using string literal concatenation

We can combine the () characters with another rule that joins adjacent string literals. This is particularly effective for long, complex format strings:

Wrap a long string value in the () characters.

Break the string into substrings:

>>> message_text = (
...     f'the internal representation '
...     f'is {mantissa_whole:d}/2**53*2**{exponent:d}'
... )
>>> message_text
'the internal representation is 7074237752514592/2**53*2**2'

We can always break a long string into adjacent pieces. Generally, this is most effective when the pieces are surrounded by () characters. We can then use as many physical line breaks as we need. This is limited to those situations where we have particularly long string literals.

Assigning intermediate results to separate variables

Here's the context for this technique:

>>> import math
>>> example_value = (63/25) * (17+15*math.sqrt(5)) / (7+15*math.sqrt(5))

We can break this into three intermediate values:

Identify sub-expressions in the overall expression. Assign these to variables:
```
>>> a = (63/25)
>>> b = (17+15*math.sqrt(5))
>>> c = (7+15*math.sqrt(5))
```
This is generally quite simple. It may require a little care to do the algebra to locate sensible sub-expressions.
Replace the sub-expressions with the variables that were created:
```
>>> example_value = a * b / c
```

We can always take a sub-expression and assign it to a variable, and use the variable everywhere the sub-expression was used. The 15*sqrt(5) product is repeated; this, too, is a good candidate for refactoring the expression.

We didn't give these variables descriptive names. In some cases, the sub-expressions have some semantics that we can capture with meaningful names. In this case, however, we chose short, arbitrary identifiers instead.

How it works...

The Python Language Manual makes a distinction between logical lines and physical lines. A logical line contains a complete statement. It can span multiple physical lines through techniques called line joining. The manual calls the techniques explicit line joining and implicit line joining.

The use of \ for explicit line joining is sometimes helpful. Because it's easy to overlook, it's not generally encouraged. PEP 8 suggests this should be the method of last resort.

The use of () for implicit line joining can be used in many cases. It often fits semantically with the structure of the expressions, so it is encouraged. We may have the () characters as a required syntax. For example, we already have () characters as part of the syntax for the print() function. We might do this to break up a long statement:

>>> print(
...    'several values including',
...    'mantissa =', mantissa,
...    'exponent =', exponent
... )

There's more...

Expressions are used widely in a number of Python statements. Any expression can have () characters added. This gives us a lot of flexibility.

There are, however, a few places where we may have a long statement that does not specifically involve an expression. The most notable example of this is the import statement—it can become long, but doesn't use any expressions that can be parenthesized. In spite of not having a proper expression, it does, however, still permit the use of (). The following example shows we can surround a very long list of imported names:

>>> from math import ( 
...    sin, cos, tan,
...    sqrt, log, frexp)

In this case, the () characters are emphatically not part of an expression. The () characters are available syntax, included to make the statement consistent with other statements.

Including descriptions and documentation

When we have a useful script, we often need to leave notes for ourselves—and others—on what it does, how it solves some particular problem, and when it should be used.

Because clarity is important, there are some formatting recipes that can help make the documentation very clear. This recipe also contains a suggested outline so that the documentation will be reasonably complete.

Getting ready

If we've used the Writing Python script and module files – syntax basics recipe to build a script file, we'll have to put a small documentation string in our script file. We'll expand on this documentation string in this recipe.

There are other places where documentation strings should be used. We'll look at these additional locations in Chapter 3, Function Definitions, and Chapter 7, Basics of Classes and Objects.

We have two general kinds of modules for which we'll be writing summary docstrings:

Library modules: These files will contain mostly function definitions as well as class definitions. In this case, the docstring summary can focus on what the module is more than what it does. The docstring can provide examples of using the functions and classes that are defined in the module. In Chapter 3, Function Definitions, and Chapter 7, Basics of Classes and Objects, we'll look more closely at this idea of a package of functions or classes.
Scripts: These are files that we generally expect will do some real work. In this case, we want to focus on doing rather than being. The docstring should describe what it does and how to use it. The options, environment variables, and configuration files are important parts of this docstring.

We will sometimes create files that contain a little of both. This requires some careful editing to strike a proper balance between doing and being. In most cases, we'll provide both kinds of documentation.

How to do it...

The first step in writing documentation is the same for both library modules and scripts:

Write a brief summary of what the script or module is or does. The summary doesn't dig too deeply into how it works. Like a lede in a newspaper article, it introduces the who, what, when, where, how, and why of the module. Details will follow in the body of the docstring.

The way the information is displayed by tools like Sphinx and pydoc suggests a specific style for the summaries we write. In the output from these tools, the context is pretty clear, therefore it's common to omit a subject in the summary sentence. The sentence often begins with the verb.

For example, a summary like this: This script downloads and decodes the current Special Marine Warning (SMW) for the area AKQ has a needless This script. We can drop that and begin with the verb phrase Downloads and decodes....

We might start our module docstring like this:

    """
    Downloads and decodes the current Special Marine Warning (SMW)
    for the area 'AKQ'.
    """

We'll separate the other steps based on the general focus of the module.

Writing docstrings for scripts

When we document a script, we need to focus on the needs of a person who will use the script.

Start as shown earlier, creating a summary sentence.
Sketch an outline for the rest of the docstring. We'll be using ReStructuredText (RST) markup. Write the topic on one line, then put a line of = under the topic to make it a proper section title. Remember to leave a blank line between each topic.
Topics may include:
- SYNOPSIS: A summary of how to run this script. If the script uses the argparse module to process command-line arguments, the help text produced by argparse is the ideal summary text.
- DESCRIPTION: A more complete explanation of what this script does.
- OPTIONS: If argparse is used, this is a place to put the details of each argument. Often, we'll repeat the argparse help parameter.
- ENVIRONMENT: If os.environ is used, this is the place to describe the environment variables and what they mean.
- FILES: Names of files that are created or read by a script are very important pieces of information.
- EXAMPLES: Some examples of using the script are always helpful.
- SEE ALSO: Any related scripts or background information.
  Other topics that might be interesting include EXIT STATUS, AUTHOR, BUGS, REPORTING BUGS, HISTORY, or COPYRIGHT. In some cases, advice on reporting bugs, for instance, doesn't really belong in a module's docstring, but belongs elsewhere in the project's GitHub or SourceForge pages.
Fill in the details under each topic. It's important to be accurate. Since we're embedding this documentation within the same file as the code, it needs to be correct, complete, and consistent.
For code samples, there's a cool bit of RST markup we can use. Recall that all elements are separated by blank lines. In one paragraph, use :: by itself. In the next paragraph, provide the code example indented by four spaces.

Here's an example of a docstring for a script:

    """
    Downloads and decodes the current Special Marine Warning (SMW)
    for the area 'AKQ'
    SYNOPSIS
    ========
    ::
        python3 akq_weather.py
 
    DESCRIPTION
    ===========
    Downloads the Special Marine Warnings
    Files
    =====
    Writes a file, ''AKW.html''.
 
    EXAMPLES
    ========
    Here's an example::
        slott$ python3 akq_weather.py
        <h3>There are no products active at this time.</h3>
    """

In the Synopsis section, we used :: as a separate paragraph. In the Examples section, we used :: at the end of a paragraph. Both versions are hints to the RST processing tools that the indented section that follows should be typeset as code.

Writing docstrings for library modules

When we document a library module, we need to focus on the needs of a programmer who will import the module to use it in their code:

Sketch an outline for the rest of the docstring. We'll be using RST markup. Write the topic on one line. Include a line of = under each topic to make the topic into a proper heading. Remember to leave a blank line between each paragraph.
Start as shown previously, creating a summary sentence:
- DESCRIPTION: A summary of what the module contains and why the module is useful
- MODULE CONTENTS: The classes and functions defined in this module
- EXAMPLES: Examples of using the module
Fill in the details for each topic. The module contents may be a long list of class or function definitions. This should be a summary. Within each class or function, we'll have a separate docstring with the details for that item.
For code examples, see the previous examples. Use :: as a paragraph or the ending of a paragraph. Indent the code example by four spaces.

How it works...

Over the decades, the man page outline has evolved to contain a complete description of Linux commands. This general approach to writing documentation has proven useful and resilient. We can capitalize on this large body of experience, and structure our documentation to follow the man page model.

These two recipes for describing software are based on summaries of many individual pages of documentation. The goal is to leverage the well-known set of topics. This makes our module documentation mirror the common practice.

We want to prepare module docstrings that can be used by the Sphinx Python Documentation Generator (see http://www.sphinx-doc.org/en/stable/). This is the tool used to produce Python's documentation files. The autodoc extension in Sphinx will read the docstring headers on our modules, classes, and functions to produce the final documentation that looks like other modules in the Python ecosystem.

There's more...

RST markup has a simple, central syntax rule: paragraphs are separated by blank lines.

This rule makes it easy to write documents that can be examined by the various RST processing tools and reformatted to look extremely nice.

When we want to include a block of code, we'll have some special paragraphs:

Separate the code from the text with blank lines.
Indent the code by four spaces.
Provide a prefix of ::. We can either do this as its own separate paragraph or as a special double-colon at the end of the lead-in paragraph:
```
        Here's an example::
            more_code()  
```
The :: is used in the lead-in paragraph.

There are places for novelty and art in software development. Documentation is not really the place to push the envelope.

A unique voice or quirky presentation isn't fun for users who simply want to use the software. An amusing style isn't helpful when debugging. Documentation should be commonplace and conventional.

It can be challenging to write good software documentation. There's a broad chasm between too little information and documentation that simply recapitulates the code. Somewhere, there's a good balance. What's important is to focus on the needs of a person who doesn't know too much about the software or how it works. Provide this semi-knowledgeable user with the information they need to describe what the software does and how to use it.

In many cases, we need to separate two parts of the use cases:

The intended use of the software
How to customize or extend the software

These may be two distinct audiences. There may be users who are distinct from developers. Each has a unique perspective, and different parts of the documentation need to respect these two perspectives.

Writing better RST markup in docstrings

When we have a useful script, we often need to leave notes on what it does, how it works, and when it should be used. Many tools for producing documentation, including docutils, work with RST markup. What RST features can we use to make documentation more readable?

Getting ready

In the Including descriptions and documentation recipe, we looked at putting a basic set of documentation into a module. This is the starting point for writing our documentation. There are a large number of RST formatting rules. We'll look at a few that are important for creating readable documentation.

How to do it...

Be sure to write an outline of the key points. This may lead to creating RST section titles to organize the material. A section title is a two-line paragraph with the title followed by an underline using =, -, ^, ~, or one of the other docutils characters for underlining.

A heading will look like this:

        Topic 
        =====

The heading text is on one line and the underlining characters are on the next line. This must be surrounded by blank lines. There can be more underline characters than title characters, but not fewer.

The RST tools will infer our pattern of using underlining characters. As long as the underline characters are used consistently, the algorithm for matching underline characters to the desired heading will detect the pattern. The keys to this are consistency and a clear understanding of sections and subsections.

When starting out, it can help to make an explicit reminder sticky note like this:

Character	Level
`=`	1
`-`	2
`^`	3
`~`	4

Example of heading characters

Fill in the various paragraphs. Separate paragraphs (including the section titles) with blank lines. Extra blank lines don't hurt. Omitting blank lines will lead the RST parsers to see a single, long paragraph, which may not be what we intended.
We can use inline markup for emphasis, strong emphasis, code, hyperlinks, and inline math, among other things. If we're planning on using Sphinx, then we have an even larger collection of text roles that we can use. We'll look at these techniques soon.
If the programming editor has a spell checker, use that. This can be frustrating because we'll often have code samples that may include abbreviations that fail spell checking.

How it works...

The docutils conversion programs will examine the document, looking for sections and body elements. A section is identified by a title. The underlines are used to organize the sections into a properly nested hierarchy. The algorithm for deducing this is relatively simple and has these rules:

If the underline character has been seen before, the level is known
If the underline character has not been seen before, then it must be indented one level below the previous outline level
If there is no previous level, this is level one

A properly nested document might have the following sequence of underline characters:

    TITLE
    =====
    SOMETHING
    ---------
    MORE    
    ^^^^
   
    EXTRA
    ^^^^^
    LEVEL 2
    -------
    LEVEL 3
    ^^^^^^^

We can see that the first title underline character, =, will be level one. The next, -, is unknown but appears after a level one, so it must be level two. The third headline has ^, which is previously unknown, is inside level two, and therefore must be level three. The next ^ is still level three. The next two, - and ^, are known to be level two and three respectively.

From this overview, we can see that inconsistency will lead to confusion.

If we change our mind partway through a document, this algorithm can't detect that. If—for inexplicable reasons—we decide to skip over a level and try to have a level four heading inside a level two section, that simply can't be done.

There are several different kinds of body elements that the RST parser can recognize. We've shown a few. The more complete list includes:

Paragraphs of text: These might use inline markup for different kinds of emphasis or highlighting.
Literal blocks: These are introduced with :: and indented four spaces. They may also be introduced with the .. parsed-literal:: directive. A doctest block is indented four spaces and includes the Python >>> prompt.
Lists, tables, and block quotes: We'll look at these later. These can contain other body elements.
Footnotes: These are special paragraphs. When rendered, they may be put on the bottom of a page or at the end of a section. These can also contain other body elements.
Hyperlink targets, substitution definitions, and RST comments: These are specialized text items.

There's more...

For completeness, we'll note here that RST paragraphs are separated by blank lines. There's quite a bit more to RST than this core rule.

In the Including descriptions and documentation recipe, we looked at several different kinds of body elements we might use:

Paragraphs of Text: This is a block of text surrounded by blank lines. Within these, we can make use of inline markup to emphasize words, or to use a font to show that we're referring to elements of our code. We'll look at inline markup in the Using inline markup recipe.
Lists: These are paragraphs that begin with something that looks like a number or a bullet. For bullets, use a simple – or *. Other characters can be used, but these are common. We might have paragraphs like this.
It helps to have bullets because:
They can help clarify
They can help organize
Numbered Lists: There are a variety of patterns that are recognized. We might use a pattern like one of the four most common kinds of numbered paragraphs:
1. Numbers followed by punctuation like . or ).
2. A letter followed by punctuation like . or ).
3. A Roman numeral followed by punctuation.
4. A special case of # with the same punctuation used on the previous items. This continues the numbering from the previous paragraphs.
Literal Blocks: A code sample must be presented literally. The text for this must be indented. We also need to prefix the code with ::. The :: character must either be a separate paragraph or the end of a lead-in to the code example.
Directives: A directive is a paragraph that generally looks like .. directive::. It may have some content that's indented so that it's contained within the directive. It might look like this:

       ..  important::
            Do not flip the bozo bit.

The .. important:: paragraph is the directive. This is followed by a short paragraph of text indented within the directive. In this case, it creates a separate paragraph that includes the admonition of important.

Using directives

Docutils has many built-in directives. Sphinx adds a large number of directives with a variety of features.

Some of the most commonly used directives are the admonition directives: attention, caution, danger, error, hint, important, note, tip, warning, and the generic admonition. These are compound body elements because they can have multiple paragraphs and nested directives within them.

We might have things like this to provide appropriate emphasis:

    ..  note:: Note Title
        We need to indent the content of an admonition.
        This will set the text off from other material.

One of the other common directives is the parsed-literal directive:

    ..  parsed-literal::
        any text
            *almost* any format
        the text is preserved
            but **inline** markup can be used.

This can be handy for providing examples of code where some portion of the code is highlighted. A literal like this is a simple body element, which can only have text inside. It can't have lists or other nested structures.

Using inline markup

Within a paragraph, we have several inline markup techniques we can use:

We can surround a word or phrase with * for *emphasis*. This is commonly typeset as italic.
We can surround a word or phrase with ** for **strong**. This is commonly typeset as bold.
We surround references with single back-ticks (`, it's on the same key as the ~ on most keyboards). Links are followed by an underscore, "_". We might use `section title`_ to refer to a specific section within a document. We don't generally need to put anything around URLs. The docutils tools recognize these. Sometimes we want a word or phrase to be shown and the URL concealed. We can use this: `the Sphinx documentation <http://www.sphinx-doc.org/en/stable/>`_.
We can surround code-related words with a double back-tick (``) to make them look like ``code``.

There's also a more general technique called a text role. A role is a little more complex-looking than simply wrapping a word or phrase in * characters. We use :word: as the role name followed by the applicable word or phrase in single ` back-ticks. A text role looks like this :strong:`this`.

There are a number of standard role names, including :emphasis:, :literal:, :code:, :math:, :pep-reference:, :rfc-reference:, :strong:, :subscript:, :superscript:, and :title-reference:. Some of these are also available with simpler markup like *emphasis* or **strong**. The rest are only available as explicit roles.

Also, we can define new roles with a simple directive. If we want to do very sophisticated processing, we can provide docutils with class definitions for handling roles, allowing us to tweak the way our document is processed. Sphinx adds a large number of roles to support detailed cross-references among functions, methods, exceptions, classes, and modules.