Python Automation Cookbook

Automating Tasks Made Easy

To properly automate tasks, we need a way to make them execute automatically at the proper times. A task that needs to be started manually is not really fully automated.

However, in order to be able to leave them running in the background while worrying about more pressing issues, the task will need to be adequate to run in fire-and-forget mode. We should be able to monitor that it executes correctly, be sure that we capture relevant information (such as receiving notifications if something interesting arises), and know whether there have been any errors while running it.

Ensuring that a piece of software runs consistently with high reliability is actually a very big deal. It is one area that, in order to be done properly, requires specialized knowledge and staff, who typically go by the names of sysadmin, operations, or SRE (Site Reliability Engineering). Big operations, such as Amazon and Google, require huge investment in ensuring that everything works 24/7.

The objective for this book is way more modest than that, as most software doesn't require this kind of high availability. Designing systems with downtimes of less than a few seconds per year is challenging, but executing a task with reasonable reliability is a much easier thing to do. However, be aware that there's maintenance to be done, and plan accordingly.

In this chapter, we'll cover the following recipes:

Preparing a task
Setting up a cron job
Capturing errors and problems
Sending email notifications

We'll begin by going over how we ought to prepare a task before we automate it.

Preparing a task

It all starts with defining precisely the work that needs to be executed, and designing it in a way that doesn't require human intervention to run.

Some ideal characteristic points are as follows:

Single, clear entry point: No confusion on how to start the task.
Clear parameters: If there are any parameters, they should be as explicit as possible.
No interactivity: Stopping the execution to request information from the user is not possible.
The result should be stored: In order to be checked at a different time than when it runs.
Clear result: When we oversee the execution of a program ourselves, we can accept more verbose results, such as unlabeled data or extra debugging information. However, for an automated task, the final result should be as concise and to the point as possible.
Errors should be logged: To analyze what went wrong.

A command-line program has a lot of those characteristics already. It always has a clear entry point, with defined parameters, and the result can be stored, even if just in text format. And it can be improved ensuring a config file that clarifies the parameters, and an output file.

Note that point 6 is the objective of the Capturing errors and problems recipe, and will be covered there.

To avoid interactivity, do not use any function that waits for user input, such as input. Remember to delete debugger breakpoints!

Getting ready

We'll start by following a structure in which a main function will serve as the entry point, and all parameters are supplied to it.

This is the same basic structure that was presented in the Adding command-line arguments recipe in Chapter 1, Let's Begin Our Automation Journey.

The definition of a main function with all of the explicit arguments covers points 1 (single, clear entry point) and 2 (clear parameters). Point 3 (no interactivity) is not difficult to achieve.

To improve points 2 (clear parameters) and 5 (clear result), we'll look at retrieving the configuration from a file and storing the result in another. Another option is to send a notification, such as an email, which will be covered later in this chapter.

How to do it...

Prepare the following command-line program by multiplying two numbers, and save it as prepare_task_step1.py:

import argparse
def main(number, other_number):
    result = number * other_number
    print(f'The result is {result}')
if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('-n1', type=int, help='A number', default=1)
    parser.add_argument('-n2', type=int, help='Another number', default=1)
    args = parser.parse_args()
    main(args.n1, args.n2)

Run prepare_task_step1.py by multiplying two numbers:

$ python3 prepare_task_step1.py -n1 3 -n2 7
The result is 21

Update the file to define a config file that contains both arguments, and save it as prepare_task_step3.py. Note that defining a config file overwrites any command-line parameters:

import argparse
import configparser
def main(number, other_number):
    result = number * other_number
    print(f'The result is {result}')
if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('-n1', type=int, help='A number', default=1)
    parser.add_argument('-n2', type=int, help='Another number', default=1)
    parser.add_argument('--config', '-c', type=argparse.FileType('r'),
                        help='config file')
    args = parser.parse_args()
    if args.config:
        config = configparser.ConfigParser()
        config.read_file(args.config)
        # Transforming values into integers
        args.n1 = int(config['ARGUMENTS']['n1'])
        args.n2 = int(config['ARGUMENTS']['n2'])
    main(args.n1, args.n2)

Create the config file, config.ini. See the ARGUMENTS section and the n1 and n2 values:
```
[ARGUMENTS]
n1=5
n2=7
```

Run the command with the config file. Note that the config file overwrites the command-line parameters, as described in step 2:

$ python3 prepare_task_step3.py -c config.ini
The result is 35
$ python3 prepare_task_step3.py -c config.ini -n1 2 -n2 3
The result is 35

Add a parameter to store the result in a file, and save it as prepare_task_step6.py:

import argparse
import sys
import configparser
def main(number, other_number, output):
    result = number * other_number
    print(f'The result is {result}', file=output)
if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('-n1', type=int, help='A number', default=1)
    parser.add_argument('-n2', type=int, help='Another number', default=1)
    parser.add_argument('--config', '-c', type=argparse.FileType('r'),
                        help='config file')
    parser.add_argument('-o', dest='output', type=argparse.FileType('w'),
                        help='output file',
                        default=sys.stdout)
    args = parser.parse_args()
    if args.config:
        config = configparser.ConfigParser()
        config.read_file(args.config)
        # Transforming values into integers
        args.n1 = int(config['ARGUMENTS']['n1'])
        args.n2 = int(config['ARGUMENTS']['n2'])
    main(args.n1, args.n2, args.output)

Run the result to check that it's sending the output to the defined file. Note that there's no output outside the result files:

$ python3 prepare_task_step6.py -n1 3 -n2 5 -o result.txt
$ cat result.txt
The result is 15
$ python3 prepare_task_step6.py -c config.ini -o result2.txt
$ cat result2.txt
The result is 35

How it works…

Note that the argparse module allows us to define files as parameters, with the argparse.FileType type, and opens them automatically. This is very handy and will raise an error if the file path leads to an invalid location.

Remember to open the file in the correct mode. In step 5, the config file is opened in read mode (r) and the output file in write mode (w), which will overwrite the file if it exists. You may find the append mode (a) useful, which will add the next piece of data at the end of an existing file.

configparser module allows us to use config files with ease. As demonstrated in step 2, the parsing of the file is simple, as follows:

config = configparser.ConfigParser()
config.read_file(file)

The config will then be accessible as a dictionary. This will have the sections of the config file as the keys, and inside another dictionary with each of the config values. So, the value n2 in the ARGUMENTS section is accessed as config['ARGUMENTS']['n2'].

Note that the values are always stored as strings, which are required to be transformed into other types, such as integers.

If you need to obtain Boolean values, do not perform value = bool(config[raw_value]), as any string will be transformed into True no matter what; for instance, the string False is a true string, as it's not empty. Using an empty string is a bad option as well, as they are very confusing. Use the .getboolean method instead, for example, value = config.getboolean(raw_value). There are similar getint() and getfloat() for integers and float values.

Python 3 allows us to pass a file parameter to the print function, which will write to that file. Step 5 shows the usage to redirect all of the printed information to a file.

Note that the default parameter is sys.stdout, which will print the value to the terminal (standard output). This means that calling the script without an -o parameter will display the information on the screen, which is helpful when developing and debugging the script:

$ python3 prepare_task_step6.py -c config.ini
The result is 35
$ python3 prepare_task_step6.py -c config.ini -o result.txt
$ cat result.txt
The result is 35

There's more...

Please refer to the full documentation of configparser in the official Python documentation: https://docs.python.org/3/library/configparser.html.

In most cases, this configuration parser should be good enough, but if more power is needed, you can use YAML files as configuration files. YAML files (https://learn.getgrav.org/advanced/yaml) are very common as configuration files. They are well structured and can be parsed directly, taking into account of various data types:

Add PyYAML to the requirements.txt file:
```
PyYAML==5.3
```
Install the requirements in the virtual environment:
```
$ pip install -r requirements.txt
```

Create the prepare_task_yaml.py file:

import yaml
import argpars
import sys
def main(number, other_number, output):
    result = number * other_number
    print(f'The result is {result}', file=output)
if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('-n1', type=int, help='A number', default=1)
    parser.add_argument('-n2', type=int, help='Another number', default=1)
    parser.add_argument('-c', dest='config', type=argparse.FileType('r'),
 help='config file in YAML format',
 default=None)
    parser.add_argument('-o', dest='output', type=argparse.FileType('w'),
                        help='output file',
                        default=sys.stdout)
    args = parser.parse_args()
    if args.config:
        config = yaml.load(args.config, Loader= yaml.FullLoader)
        # No need to transform values
        args.n1 = config['ARGUMENTS']['n1']
        args.n2 = config['ARGUMENTS']['n2']
    main(args.n1, args.n2, args.output)

Note that the PyYAML yaml.load() function requires a Loader parameter. This is to avoid arbitrary code execution if the YAML file comes from an untrusted source. Always use yaml.SafeLoader unless you need a set of YAML language features. Never use loaders other than yaml.SafeLoader if any part of the data coming from a YAML file comes from an untrusted source (for example, user input). Refer to this article for more information: https://github.com/yaml/pyyaml/wiki/PyYAML-yaml.load(input)-Deprecation.

Define the config file, config.yaml:
```
ARGUMENTS:
    n1: 7
    n2: 4
```

Then, run the following:

$ python3 prepare_task_yaml.py -c config.yaml
The result is 28

There's also the possibility of setting a default config file, as well as a default output file. This can be handy to create a task that requires no input parameters.

As a general rule, try to avoid creating too many input and configuration parameters if the task has a very specific objective in mind. Try to limit the input parameters to different executions of the task. A parameter that never changes is probably fine being defined as a constant. A high number of parameters will make config files or command-line arguments complicated and will create more maintenance in the long run. On the other hand, if your objective is to create a very flexible tool to be used in very different situations, then creating more parameters is probably a good idea. Try to find your own proper balance!

Setting up a cron job

Cron is an old-fashioned but reliable way of executing commands. It has been around since the 1970s in Unix, and it's an old favorite in system administration to perform maintenance tasks such as freeing up disk space, rotating log files, making backups, and other common, repetitive operations.

This recipe is Unix and Unix-like operating systems specific, so it will work in Linux and macOS. While it's possible to schedule a task in Windows, it's very different and uses Task Scheduler, which won't be described here. If you have access to a Linux server, this can be a good way of scheduling periodic tasks.

The main advantages are as follows:

It's present in virtually all Unix or Linux systems and configured to run automatically.
It's easy to use, although a little deceptive at first.
It's well known. Almost anyone involved with admin tasks will have a general idea of how to use it.
It allows for easy periodic commands, with good precision.

However, it also has some disadvantages, including the following:

By default, it may not give much feedback. Retrieving the output, logging execution, and errors are critical.
The task should be as self-contained as possible to avoid problems with environment variables, such as using the wrong Python interpreter, or what path should execute.
It is Unix-specific.
Only fixed periodic times are available.
It doesn't control how many tasks run at the same time. Each time the countdown goes off, it creates a new task. For example, a task that takes 1 hour to complete, and that is scheduled to run once every 45 minutes, will have 15 minutes of overlap where two tasks will be running.

Don't understate the latest effect. Running multiple expensive tasks at the same time can have a bad effect on performance. Having expensive tasks overlapping may result in a race condition where each task stops the others from ever finishing! Allow ample time for your tasks to finish and keep an eye on them. Keep in mind that any other program running in the same host may have their performance affected, which can include any service, such as web servers, databases, and email. Check how loaded the host where the task will execute is so as to avoid surprises.

Getting ready

We will produce a script, called cron.py:

import argparse
import sys
from datetime import datetime
import configparser
def main(number, other_number, output):
    result = number * other_number
    print(f'[{datetime.utcnow().isoformat()}] The result is {result}', 
          file=output)
if __name__ == '__main__':
    parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)
    parser.add_argument('--config', '-c', type=argparse.FileType('r'),
                        help='config file',
                        default='/etc/automate.ini')
    parser.add_argument('-o', dest='output', type=argparse.FileType('w'),
                        help='output file',
                        default=sys.stdout)
    args = parser.parse_args()
    if args.config:
        config = configparser.ConfigParser()
        config.read_file(args.config)
        # Transforming values into integers
        args.n1 = int(config['ARGUMENTS']['n1'])
        args.n2 = int(config['ARGUMENTS']['n2'])
    main(args.n1, args.n2, args.output)

Note the following details:

The config file is, by default, /etc/automate.ini. Reuse config.ini from the previous recipe.
A timestamp has been added to the output. This will make it explicit when the task is run.
The result is being added to the file, as shown with the a mode where the file is open.
The ArgumentDefaultsHelpFormatter parameter automatically adds information about default values when printing the help using the -h argument.

Check that the task is producing the expected result and that you can log to a known file:

$ python3 cron.py
[2020-01-15 22:22:31.436912] The result is 35
$ python3 cron.py -o /path/automate.log
$ cat /path/automate.log
[2020-01-15 22:28:08.833272] The result is 35

How to do it...

Obtain the full path of the Python interpreter. This is the interpreter that's in your virtual environment:
```
$ which python
/your/path/.venv/bin/python
```

Prepare the cron job to be executed. Get the full path and check that it can be executed without any problems. Execute it a couple of times:

$ /your/path/.venv/bin/python /your/path/cron.py -o /path/automate.log
$ /your/path/.venv/bin/python /your/path/cron.py -o /path/automate.log

Check that the result is being added correctly to the result file:

$ cat /path/automate.log
[2020-01-15 22:28:08.833272] The result is 35
[2020-01-15 22:28:10.510743] The result is 35

Edit the crontab file to run the task once every 5 minutes:
```
$ crontab -e
*/5 * * * * /your/path/.venv/bin/python /your/path/cron.py -o /path/automate.log
```
Note that this opens an editing terminal with your default command-line editor.

If you haven't set up your default command-line editor, then, by default, it is likely to be Vim. This can be disconcerting if you don't have experience with Vim. Press I to start inserting text and Esc when you're done. Then, exit after saving the file with wq. For more information about Vim, refer to this introduction: https://null-byte.wonderhowto.com/how-to/intro-vim-unix-text-editor-every-hacker-should-be-familiar-with-0174674.

For information on how to change the default command-line editor, refer to the following link: https://www.a2hosting.com/kb/developer-corner/linux/setting-the-default-text-editor-in-linux.
Check the crontab contents. Note that this displays the crontab contents, but doesn't set it to edit:
```
$ contab -l
*/5 * * * * /your/path/.venv/bin/python /your/path/cron.py -o /path/automate.log
```

Wait and check the result file to see how the task is being executed:

$ tail -F /path/automate.log
[2020-01-17 21:20:00.611540] The result is 35
[2020-01-17 21:25:01.174835] The result is 35
[2020-01-17 21:30:00.886452] The result is 35

How it works...

The crontab line consists of a line describing how often to run the task (the first six elements), plus the task. Each of the initial six elements means a different unit of time to execute. Most of them are stars, meaning any:

* * * * * *
| | | | | | 
| | | | | +-- Year              (range: 1900-3000)
| | | | +---- Day of the Week   (range: 1-7, 1 standing for Monday)
| | | +------ Month of the Year (range: 1-12)
| | +-------- Day of the Month  (range: 1-31)
| +---------- Hour              (range: 0-23)
+------------ Minute            (range: 0-59)

Therefore, our line, */5 * * * * *, means every time the minute is divisible by 5, in all hours, all days... all years.

Here are some examples:

30  15 * * * * means "every day at 15:30"
30   * * * * * means "every hour, at 30 minutes"
0,30 * * * * * means "every hour, at 0 minutes and 30 minutes"
*/30 * * * * * means "every half hour"
0    0 * * 1 * means "every Monday at 00:00"

Do not try to guess too much. Use a cheat sheet such as https://crontab.guru/ for examples and tweaks. Most of the common usages will be described there directly. You can also edit a formula and get a descriptive piece of text of how it's going to run.

After the description of how to run the cron job, include the line to execute the task, as prepared in step 2 of the How to do it… section.

Note that the task is described with all of the full paths for every related file—the interpreter, the script, and the output file. This removes all ambiguity related to the paths and reduces the chances of possible errors. A very common error is for cron to not be able to determine one or more of these three elements.

There's more...

The description of the default output (standard output) can be a bit verbose. When calling python3 cron.py -h, it gets displayed as:

-o OUTPUT   output file (default: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)

This is the description of the standard output (stdout). The format of the parameter can be changed using the formatter_class argument in the ArgumentParser. This means that you can use a custom formatter inheriting from the available default ones to tweak the display of the value. Refer to the documentation at https://docs.python.org/2/library/argparse.html#formatter-class

If there's any problem in the execution of the crontab, you should receive a system mail. This will show up as a message in the terminal like this:

You have mail.
$
This can be read with mail:
$ mail
Mail version 8.1 6/6/93. Type ? for help.
"/var/mail/jaime": 1 message 1 new
>N 1 jaime@Jaimes-iMac-5K Fri Jun 17 21:15 20/914 "Cron <jaime@Jaimes-iM"
? 1
Message 1:
...
/usr/local/Cellar/python/3.8.1/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python: can't open file 'cron.py': [Errno 2] No such file or directory

In the next recipe, we will explore methods to capture the errors independently so that the task can run smoothly.

Capturing errors and problems

An automated task's main characteristic is its fire-and-forget quality. We are not actively looking at the result, but making it run in the background.

Most of the recipes in this book deal with external information, such as web pages or other reports, so the likelihood of finding an unexpected problem when running it is high. This recipe will present an automated task that will safely store unexpected behaviors in a log file that can be checked afterward.

Getting ready

As a starting point, we'll use a task that will divide two numbers, as described in the command line.

This task is very similar to the one presented in step 5 of How it works for the Preparing a task recipe, earlier this chapter. However, instead of multiplying two numbers, we'll divide them.

How to do it...

Create the task_with_error_handling_step1.py file, as follows:

import argparse
import sys
def main(number, other_number, output):
    result = number / other_number
    print(f'The result is {result}', file=output)
if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('-n1', type=int, help='A number', default=1)
    parser.add_argument('-n2', type=int, help='Another number', default=1)      
    parser.add_argument('-o', dest='output', type=argparse.FileType('w'),
                        help='output file', default=sys.stdout)
    args = parser.parse_args()
    main(args.n1, args.n2, args.output)

Execute it a couple of times to see that it divides two numbers:

$ python3 task_with_error_handling_step1.py -n1 3 -n2 2
The result is 1.5
$ python3 task_with_error_handling_step1.py -n1 25 -n2 5
The result is 5.0

Check that dividing by 0 produces an error, and that the error is not logged on the result file:

$ python task_with_error_handling_step1.py -n1 5 -n2 1 -o result.txt
$ cat result.txt
The result is 5.0
$ python task_with_error_handling_step1.py -n1 5 -n2 0 -o result.txt
Traceback (most recent call last):
 File "task_with_error_handling_step1.py", line 20, in <module>
 main(args.n1, args.n2, args.output)
 File "task_with_error_handling_step1.py", line 6, in main  result = number / other_number
ZeroDivisionError: division by zero
$ cat result.txt

Create the task_with_error_handling_step4.py file:

import argparse
import sys
import logging
LOG_FORMAT = '%(asctime)s %(name)s %(levelname)s %(message)s'
LOG_LEVEL = logging.DEBUG
def main(number, other_number, output):
    logging.info(f'Dividing {number} between {other_number}')
    result = number / other_number
    print(f'The result is {result}', file=output)
if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('-n1', type=int, help='A number', default=1)
    parser.add_argument('-n2', type=int, help='Another number', default=1)
    parser.add_argument('-o', dest='output', type=argparse.FileType('w'),
                        help='output file', default=sys.stdout)
    parser.add_argument('-l', dest='log', type=str, help='log file',
                        default=None)
    args = parser.parse_args()
    if args.log:
        logging.basicConfig(format=LOG_FORMAT, filename=args.log,
                            level=LOG_LEVEL)
    else:
        logging.basicConfig(format=LOG_FORMAT, level=LOG_LEVEL)
    try:
        main(args.n1, args.n2, args.output)
    except Exception as exc:
        logging.exception("Error running task")
        exit(1)

Run it to check that it displays the proper INFO and ERROR logs, and that it stores it on the log file:

$ python3 task_with_error_handling_step4.py -n1 5 -n2 0
2020-01-19 14:25:28,849 root INFO Dividing 5 between 0
2020-01-19 14:25:28,849 root ERROR division by zero
Traceback (most recent call last):
  File "task_with_error_handling_step4.py", line 31, in <module>
    main(args.n1, args.n2, args.output)
  File "task_with_error_handling_step4.py", line 10, in main
    result = number / other_number
ZeroDivisionError: division by zero
$ python3 task_with_error_handling_step4.py -n1 5 -n2 0 -l error.log
$ python3 task_with_error_handling_step4.py -n1 5 -n2 0 -l error.log
$ cat error.log
2020-01-19 14:26:15,376 root INFO Dividing 5 between 0
2020-01-19 14:26:15,376 root ERROR division by zero
Traceback (most recent call last):
  File "task_with_error_handling_step4.py", line 33, in <module>
    main(args.n1, args.n2, args.output)
  File "task_with_error_handling_step4.py", line 11, in main
    result = number / other_number
ZeroDivisionError: division by zero
2020-01-19 14:26:19,960 root INFO Dividing 5 between 0
2020-01-19 14:26:19,961 root ERROR division by zero
Traceback (most recent call last):
  File "task_with_error_handling_step4.py", line 33, in <module>
    main(args.n1, args.n2, args.output)
  File "task_with_error_handling_step4.py", line 11, in main
    result = number / other_number
ZeroDivisionError: division by zero

How it works...

To properly capture any unexpected exceptions, the main function should be wrapped in a try-except block, as implemented in step 4 of the How to do it… section. Compare this to how step 1 does not wrap the code:

    try:
        main(...)
    except Exception as exc:
        # Something went wrong
        logging.exception("Error running task")
        exit(1)

Note that logging the exception is important for getting information on what went wrong.

This kind of exception is nicknamed Pokémon Exception because it can catch 'em all. It will capture any unexpected error at the highest level. Do not use it in other areas of the code, as capturing everything can hide unexpected errors. At the very least, any unexpected exception should be logged to allow for further analysis.

The extra step, to exit with status 1 by using the exit(1) call, informs the operating system that something went wrong with our script.

The logging module allows us to log. Note the basic configuration, which includes an optional file to store the logs, the format, and the level of the logs to display.

The available levels for logs are—from less critical to more critical—DEBUG, INFO, WARNING, ERROR, and CRITICAL. The logging level will set the minimum severity required to log the message. For example, an INFO log won't be stored if the severity is set to WARNING.

Creating logs is easy. You can do this by making a call to the logging.<logging level> method (where the logging level is debug, info, and so on). For example:

>>> import logging
>>> logging.basicConfig(level=logging.INFO)
>>> logging.warning('a warning message')
WARNING:root:a warning message
>>> logging.info('an info message')
INFO:root:an info message
>>> logging.debug('a debug message')
>>>

Note how logs with a severity lower than INFO are not displayed. Use the level definition to tweak how much information to display. This may change, for example, how DEBUG logs may be used only while developing the task, but not be displayed when running it. Notice that task_with_error_handling_step4.py defines the logging level to be DEBUG, by default.

A good definition of log levels is key to displaying relevant information while reducing noise. It is not easy to set up sometimes, but especially if more than one person is involved, try to agree on exactly what WARNING versus ERROR means to avoid misinterpretations.

logging.exception() is a special case that will create an ERROR log, but it will also include information about the exception, such as the stack trace.

Remember to check logs to discover errors. A useful reminder is to add a note to the results file, like this:

try:
    main(args.n1, args.n2, args.output)
except Exception as exc:
    logging.exception(exc)
    print('There has been an error. Check the logs', file=args.output)

There's more...

The Python logging module has many capabilities, including the following:

It provides further tweaks to the format of the log, for example, including the file and line number of the log that was produced.
It defines different logger objects, each one with its own configuration, such as logging level and format. This allows us to publish logs to different systems in different ways, though, normally, a single logger object is used to keep things simple.
It sends logs to multiple places, such as the standard output and file, or even a remote logger.
It automatically rotates logs, creating new log files after a certain time or size. This is handy for keeping logs organized by day or week. It also allows for the compression or removal of old logs. Logs take up space when they accumulate.
It reads standard logging configurations from files.

Instead of creating complex rules on what to log, try to log extensively using the proper level for each, and then filter by level.

For comprehensive details, refer to the Python documentation of the module at https://docs.python.org/3.8/library/logging.html, or the tutorial at https://docs.python.org/3.8/howto/logging.html.

Sending email notifications

Email has become an inescapable tool for everyday use. It's arguably the best place to send a notification if an automated task has detected something. On the other hand, email inboxes are already too full up with spam messages, so be careful.

Spam filters are also a reality. Be careful with whom to send emails to and the number of emails to be sent. An email server or address can be labeled as a spam source, and all emails may be quietly dropped by the internet.

This recipe will show you how to send a single email using an existing email account.

This approach is viable for spare emails sent to a couple of people, as a result of an automated task, but no more than that. Refer to Chapter 9, Dealing with Communication Channels, for more ideas on how to send emails, including groups.

Getting ready

For this recipe, we require a valid email account set up, which includes the following:

A valid email server using SMTP; SMTP is the standard email protocol
A port to connect to
An address
A password

These four elements should be enough to be able to send an email.

Some email services, for example, Gmail, will encourage you to set up 2FA, meaning that a password is not enough to send an email. Typically, they'll allow you to create a specific password for apps to use, bypassing the 2FA request. Check your email provider's information for options.

The email service you use should indicate what the SMTP server is and what port to use in their documentation. This can be retrieved from email clients as well, as they are the same parameters. Check your provider documentation. In the following example, we will use a Gmail account.

How to do it...

Create the email_task.py file, as follows:

import argparse
import configparser
import smtplib 
from email.message import EmailMessage
def main(to_email, server, port, from_email, password):
    print(f'With love, from {from_email} to {to_email}')
    # Create the message
    subject = 'With love, from ME to YOU'
    text = '''This is an example test'''
    msg = EmailMessage()
    msg.set_content(text)
    msg['Subject'] = subject
    msg['From'] = from_email
    msg['To'] = to_email
    # Open communication and send
    server = smtplib.SMTP_SSL(server, port)
    server.login(from_email, password)
    server.send_message(msg)
    server.quit()
if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('email', type=str, help='destination email')
    parser.add_argument('-c', dest='config', type=argparse.FileType('r'),
                        help='config file', default=None)
    args = parser.parse_args()
    if not args.config:
        print('Error, a config file is required')
        parser.print_help()
        exit(1)
    config = configparser.ConfigParser()
    config.read_file(args.config)
    main(args.email,
         server=config['DEFAULT']['server'],
         port=config['DEFAULT']['port'],
         from_email=config['DEFAULT']['email'],
         password=config['DEFAULT']['password'])

Create a configuration file called email_conf.ini with the specifics of your email account. For example, for a Gmail account, fill in the following template. The template is available in GitHub at https://github.com/PacktPublishing/Python-Automation-Cookbook-Second-Edition/blob/master/Chapter02/email_conf.ini but be sure to fill it in with your data:
```
[DEFAULT]
email = EMAIL@gmail.com
server = smtp.gmail.com
port = 465
password = PASSWORD
```
Ensure that the file cannot be read or written by other users on the system, setting the permissions of the file to allow only our user. 600 permissions mean read and write access for the file owner, and no access to anyone else:
```
$ chmod 600 email_conf.ini
```

Run the script to send a test email:

$ python3 email_task.py -c email_conf.ini destination_email@server.com

Check the inbox of the destination email; an email should be received with the subject With love, from ME to YOU.

How it works...

There are two key steps in the preceding scripts—the generation of the message and the sending.

The message needs to contain mainly the To and From email addresses, as well as the Subject. If the content is pure text, as in this case, calling .set_content() is enough. The whole message can then be sent.

It is technically possible to send an email that is tagged as coming from a different email address than the one you used to send it. This is discouraged, though, as it can be considered by your email provider as trying to impersonate a different email. You can use the reply-to header to allow answering to a different account.

Sending the email requires you to connect to the specified server and start an SMTP connection. SMTP is the standard for email communication.

The steps are quite straightforward—configure the server, log into it, send the prepared message, and quit.

If you need to send more than one message, you can log in, send multiple emails, and then quit, instead of connecting each time.

There's more...

If the objective is a bigger operation, such as a marketing campaign, or even production emails, such as confirming a user's email, please refer to Chapter 9, Dealing with Communication Channels.

The email message content used in this recipe is very simple, but emails can be much more complicated than that.

The To field can contain multiple recipients. Separate them with commas, like this:

message['To'] = ','.join(recipients)

Emails can be defined in HTML, with an alternative plain text, and have attachments. The basic operation is to set up a MIMEMultipart and then attach each of the MIME parts that compose the email:

from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.mime.image import MIMEImage
message = MIMEMultipart()
part1 = MIMEText('some text', 'plain')
message.attach(part1)
with open('path/image', 'rb') as image:
    part2 = MIMEImage(image.read())
message.attach(part2)

The most common SMTP connection is SMTP_SSL, which is more secure, as all communication with the server is encrypted and always requires a login and password. However, plain, unauthenticated SMTP exists—check your email provider documentation.

Remember that this recipe is aimed toward simple notifications. Emails can grow quite complex if attaching different information. If your objective is an email for customers or any general group, try to use the ideas in Chapter 9, Dealing with Communication Channels.