Before you start reading this book, you should know the basics of Python programming, including its basic syntax, variable types, data type tuples, list dictionaries, functions, strings, and methods. We will work with Python version 3.7, available at www.python.org/downloads.

The examples and source code for this chapter are available in the GitHub repository at https://github.com/PacktPublishing/Mastering-Python-for-Networking-and-Security-Second-Edition.

Check out the following video to see the Code in Action:

https://bit.ly/3mXDJld

Python has many advantages when it comes to picking it for scripting. Before we dig deep into the Python scripting landscape, let’s take a look at these advantages and new features available in Python 3.

Why choose Python?

There are many reasons to choose Python as your main programming language. Importantly, many security tools are written in Python. This language offers many opportunities for extending and adding features to tools that are already written. Let’s look at what else Python has to offer us:

• It is a multi-platform and open source language.
• It is a simple, fast, robust, and powerful language.
• Many libraries, modules, and projects focused on computer security are written in Python.
• A lot of documentation is available, along with a very large user community.
• It is a language designed to make robust programs with a few lines of code, something that is only possible in other languages after including many characteristics of each language.
• It is ideal for prototypes and rapid-concept tests (Proof of Concept).

Multi-platform capabilities and versions

The Python interpreter is available on many platforms (Linux, DOS, Windows, and macOS X). The code that we create in Python is translated into bytecode when it is executed for the first time. For that reason, in systems in which we are going to execute our programs or scripts developed in Python, we need the interpreter to be installed.

In this book, we will work with Python version 3.7. If you’re starting to write some new Python code today, you should use Python 3. It’s important to be aware that Python 2 is end of life and will no longer receive security patches, so users should upgrade their code to Python 3.

If you have Python 2 code that you can upgrade to Python 3, you should do that as well. But if you’re like most companies with an existing Python 2 code base, your best option might well be to upgrade incrementally, which means having code that works under 2 and 3 simultaneously. Once you’ve converted all of your code, and it passes tests under both Python 2 and 3, you can flip the switch, joining the world of Python 3 and all of its goodness.

Tip

PEPs (Python Enhancement Proposals) are the main forums in the Python community for proposing new features or improvements to the Python core language. They enable the community to review, discuss, and improve proposals. Popular tools such as pep8 and flake8 enforce these rules when run on a Python file. The main PEP index can be found at http://python.org/dev/peps.

Python 3 features

Much has been written about the changes in Python 2 and 3. An extensive collection of such information is available at https://python-future.org. This site offers the futurize and pasteurize packages, as well as a great deal of documentation describing the changes between versions, techniques for upgrading, and other things to watch out for.

Some of the most important new features that Python 3 offers are as follows:

• Unicode is supported throughout the standard library and is the default type for any strings defined.
• The input function has been renewed.
• The modules have been restructured.
• The new asyncio library, which is part of the standard library, gives a defined way to execute asynchronous programming in Python. This makes it easy to write concurrent programs enabling you to make the most of your new-generation hardware.
• Better exception handling: in Python 2.X, there were lots of ways to throw and catch exceptions; with Python 3, error handling is cleaner and improved.
• Virtualenv is now part of the standard Python distribution.

  Tip

  If you are new to Python, you should start with Python 3 since many things have been improved and more thoughtfully designed. If you want to use old code or specific packages and libraries that are still based on Python 2, you should, of course, use this version, especially in those cases where porting would be complex. Exploring old Python 2 code with tools such as 2to3 and porting, if necessary, is a good place to start.

Now that you know the reason for choosing Python as a scripting language and the main features of Python 3, let’s move on to learning about the main data structures available in Python.

In this section, we will review different types of data structures, including lists, tuples, and dictionaries. We will see methods and operations for managing these data structures and practical examples where we review the main use cases.

Lists

Lists in Python are equivalent to structures as dynamic vectors in programming languages such as C. We can express literals by enclosing their elements between a pair of brackets and separating them with commas. The first element of a list has index 0.

Consider the following example: a programmer can create a list using the append() method by adding objects, printing the objects, and then sorting them before printing again. We describe a list of protocols in the following example, and use the key methods of a Python list as add, index, and remove:

>>> protocolList = []
>>> protocolList.append(“ftp”)
>>> protocolList.append(“ssh”)
>>> protocolList.append(“smtp”)
>>> protocolList.append(“http”)
>>> print(protocolList)
[‘ftp’,’ssh’,’smtp’,’http’]
>>> protocolList.sort()
>>> print(protocolList)
[‘ftp’,’http’,’smtp’,’ssh’]
>>> type(protocolList)
<type ‘list’>
>>> len(protocolList)
4

To access specific positions, we can use the index() method, and to delete an element, we can use the remove() method:

>>> position = protocolList.index(“ssh”)
>>> print(“ssh position”+str(position))
ssh position 3
>>> protocolList.remove(“ssh”)
>>> print(protocolList)
[‘ftp’,’http’,’smtp’]
>>> count = len(protocolList)
>>> print(“Protocol elements “+str(count))
Protocol elements 3

To print out the whole protocol list, use the following instructions. This will loop through all the elements and print them:

>>> for protocol in protocolList:
>>> 	print (protocol)
ftp
http
smtp

Lists also have methods that help manipulate the values within them and allow us to store more than one variable within them and provide a better way to sort object arrays in Python. These are the techniques commonly used to control lists:

• .append(value): Appends an element at the end of the list
• .count(‘x’): Gets the number of ‘x’ in the list
• .index(‘x’): Returns the index of ‘x’ in the list
• .insert(‘y’,’x’): Inserts ‘x’ at location ‘y’
• .pop(): Returns the last element and also removes it from the list
• .remove(‘x’): Removes the first ‘x’ from the list
• .reverse(): Reverses the elements in the list
• .sort(): Sorts the list in ascending order

The indexing operator allows access to an element and is expressed syntactically by adding its index in brackets to the list, list [index]. You can change the value of a chosen element in the list using the index between brackets:

protocols[4] = ‘ssh’
print(“New list content: “, protocols)

Also, you can copy the value of a specific position to another position in the list:

protocols[1] = protocols[4]
print(“New list content:”, protocols)

The value inside the brackets that selects one element of the list is called an index, while the operation of selecting an element from the list is known as indexing.

Adding elements to a list

We can add elements to a list by means of the following methods:

• list.append(value): This method allows an element to be inserted at the end of the list. It takes its argument’s value and puts it at the end of the list that owns the method. The list’s length then increases by one.
• list.insert(location, value): The insert() method is a bit smarter since it can add a new element at any place in the list, and not just at the end. It takes as arguments first the required location of the element to be inserted and then the element to be inserted.

Reversing a list

Another interesting operation that we perform in lists is the one that offers the possibility of getting elements in a reverse way in the list through the reverse() method:

>>> protocolList.reverse()
>>> print(protocolList)
[‘smtp’,’http’,’ftp’]

Another way to do the same operation is to use the -1 index. This quick and easy technique shows how you can access all the elements of a list in reverse order:

>>> protocolList[::-1]
>>> print(protocolList)
[‘smtp’,’http’,’ftp’]

Searching elements in a list

In this example, we can see the code for finding the location of a given element inside a list. We use the range function to get elements inside protocolList and we compare each element with the element to find. When both elements are equal, we break the loop and return the element.

You can find the following code in the search_element_list.py file:

protocolList = [“FTP”, “HTTP”, “SNMP”, “SSH”]
toFind = “SSH”
found = False
for i in range(len(protocolList)):
        found = protocolList[i] == toFind
        if found:
                break
if found:
        print(“Element found at index”, i)
else:
        print(“Element not found”)

Now that you know how to add, reverse, and search for elements in a list, let’s move on to learning about tuples in Python.

Tuples

A tuple is like a list, except its size cannot change and cannot add more elements than originally specified. The parentheses delimit a tuple. If we try to modify a tuple element, we get an error that indicates that the tuple object does not support element assignment:

>>>tuple=(“ftp”,”ssh”,”http”,”snmp”)
>>>tuple[0]
‘ftp’
>>>tuple[0]=”FTP”
Traceback (most recent call last):
    File “<stdin>”, line 1, in <module>
TypeError: ‘tuple’ object does not support item assignment

Now that you know the basic data structures for working with Python, let’s move on to learning about Python dictionaries in order to organize information in the key-value format.

Python dictionaries

The Python dictionary data structure is probably the most important in the entire language and allows us to associate values with keys. A key is any immutable object. The value associated with a key can be accessed with the indexing operator. In Python, dictionaries are implemented using hash tables.

A Python dictionary is a way of storing information in the format of key: value pairs. Python dictionaries have curly brackets, {}. Let’s look at a protocols dictionary, with names and numbers, for example:

>>> services = {“ftp”:21, “ssh”:22, “smtp”:25, “http”:80}

The limitation with dictionaries is that we cannot use the same key to create multiple values. This will overwrite the duplicate key preceding value.

Using the update method, we can combine two distinct dictionaries into one. In addition, the update method will merge existing elements if they conflict:

>>> services = {“ftp”:21, “ssh”:22, “smtp”:25, “http”:80}
>>> services2 = {“ftp”:21, “ssh”:22, “snmp”:161, “ldap”:389}
>>> services.update(services2)
>>> print(services)
{“ftp”:21, “ssh”:22, “smtp”:25, “http”:80,”snmp”:161, “ldap”:389}

The first value is the key, and the second the key value. We can use any unchangeable value as a key. We can use numbers, sequences, Booleans, or tuples, but not lists or dictionaries, since they are mutable.

The main difference between dictionaries and lists or tuples is that values contained in a dictionary are accessed by their name and not by their index. You may also use this operator to reassign values, as in the lists and tuples:

>>> services[“http”]= 8080

This means that a dictionary is a set of key-value pairs with the following conditions:

• Each key must be unique: That means it is not possible to have more than one key of the same value.
• A key may be data of any type: It may be a number or a string.
• A dictionary is not a list: A list contains a set of numbered values, while a dictionary holds pairs of values.
• The len() function: This works for dictionaries and returns the number of key-value elements in the dictionary.

  Important note

  In Python 3.7, dictionaries have become ordered collections by default.

When building a dictionary, each key is separated from its value by a colon, and we separate items by commas. The .keys() method will return a list of all keys of a dictionary and the .items() method will return a complete list of elements in the dictionary. The following are examples involving these methods:

• services.keys() is a method that will return all the keys in the dictionary.
• services.items() is a method that will return the entire list of items in a dictionary:

>>> keys = services.keys()
>>> print(keys)
[‘ftp’, ‘smtp’, ‘ssh’, ‘http’, ‘snmp’]

Another way is based on using a dictionary’s method called items(). The method returns a list of tuples (this is the first example where tuples are something more than just an example of themselves) where each tuple is a key-value pair:

1. Enter the following command:

   >>> items = services.items()
   >>> print(items)
   [(‘ftp’, 21), (‘smtp’,25), (‘ssh’, 22), (‘http’, 80), (‘snmp’, 161)]

   From the performance point of view, when it is stored, the key inside a dictionary is converted to a hash value to save space and boost efficiency when searching or indexing the dictionary. The dictionary may also be printed, and the keys browsed in a particular order.

2. The following code sorts the dictionary elements in ascending order by key using the sort() method:

   >>> items.sort()
   >>> print(items)
   [(‘ftp’, 21), (‘http’, 80), (‘smtp’, 25), (‘snmp’, 161), (‘ssh’, 22)]

3. Finally, you might want to iterate over a dictionary and extract and display all the key-value pairs with a classical for loop:

   >>> for key,value in services.items():
   >>>	print(key,value)
   ftp 21
   smtp 25
   ssh 22
   http 80
   snmp 16

Assigning a new value to an existing key is simple due to dictionaries being fully mutable. There are no obstacles to modify them:

1. In this example, we’re going to replace the value of the http key:

   >>> services[‘http’] = 8080
   >>> print(services)
   {“ftp”:21, “ssh”:22, “smtp”:25, “http”:8080,”snmp”:161}

2. Adding a new key-value pair to a dictionary is as easy as modifying a value. Only a new, previously non-existent key needs to be assigned to one:

   >>> services[‘ldap’] = 389
   >>> print(services)
   {“ftp”:21, “ssh”:22, “smtp”:25, “http”:8080,”snmp”:161, “ldap”:389}

Note that this is very different behavior compared to lists, which don’t allow you to assign values to non-existing indices.

Now that you know the main data structures for working with Python, let’s move on to learning how to structure our Python code with functions and classes.

In this section, we will review Python functions, classes, and how to manage exceptions in Python scripts. We will review some examples for declaring and using both in our script code. We’ll also review the main exceptions we can find in Python for inclusion in our scripts.

Python functions

A function is a block of code that performs a specific task when the function is called (invoked). You can use functions to make your code reusable, better organized, and more readable. Functions can have parameters and return values.

There are at least four basic types of functions in Python:

• Built-in functions: These are an integral part of Python. You can see a complete list of Python’s built-in functions at https://docs.python.org/3/library/functions.html.
• Functions that come from pre-installed modules.
• User-defined functions: These are written by developers in their own code and they use them freely in Python.
• The lambda function: This allow us to create anonymous functions that are built using expressions such as product = lambda x,y : x * y, where lambda is a Python keyword and x and y are the function parameters.

With the builtins module, we can see all classes and methods available by default in Python:

>>> import builtins
>>> dir(builtins)
[‘ArithmeticError’, ‘AssertionError’, ‘AttributeError’, ‘BaseException’, ‘BlockingIOError’, ‘BrokenPipeError’, ‘BufferError’, ‘BytesWarning’, ‘ChildProcessError’, ‘ConnectionAbortedError’, ‘ConnectionError’, ‘ConnectionRefusedError’, ‘ConnectionResetError’, ‘DeprecationWarning’, ‘EOFError’, ‘Ellipsis’, ‘EnvironmentError’, ‘Exception’, ‘False’, ‘FileExistsError’, ‘FileNotFoundError’, ‘FloatingPointError’, ‘FutureWarning’, ‘GeneratorExit’, ‘IOError’, ‘ImportError’, ‘ImportWarning’, ‘IndentationError’, ‘IndexError’, ‘InterruptedError’, ‘IsADirectoryError’, ‘KeyError’, ‘KeyboardInterrupt’, ‘LookupError’, ‘MemoryError’, ‘ModuleNotFoundError’, ‘NameError’, ‘None’, ‘NotADirectoryError’, ‘NotImplemented’, ‘NotImplementedError’, ‘OSError’, ‘OverflowError’, ‘PendingDeprecationWarning’, ‘PermissionError’, ‘ProcessLookupError’, ‘RecursionError’, ‘ReferenceError’, ‘ResourceWarning’, ‘RuntimeError’, ‘RuntimeWarning’, ‘StopAsyncIteration’, ‘StopIteration’, ‘SyntaxError’, ‘SyntaxWarning’, ‘SystemError’, ‘SystemExit’, ‘TabError’, ‘TimeoutError’, ‘True’, ‘TypeError’, ‘UnboundLocalError’, ‘UnicodeDecodeError’, ‘UnicodeEncodeError’, ‘UnicodeError’, ‘UnicodeTranslateError’, ‘UnicodeWarning’, ‘UserWarning’, ‘ValueError’, ‘Warning’, ‘ZeroDivisionError’, ‘__build_class__’, ‘__debug__’, ‘__doc__’, ‘__import__’, ‘__loader__’, ‘__name__’, ‘__package__’, ‘__spec__’, ‘abs’, ‘all’, ‘any’, ‘ascii’, ‘bin’, ‘bool’, ‘breakpoint’, ‘bytearray’, ‘bytes’, ‘callable’, ‘chr’, ‘classmethod’, ‘compile’, ‘complex’, ‘copyright’, ‘credits’, ‘delattr’, ‘dict’, ‘dir’, ‘divmod’, ‘enumerate’, ‘eval’, ‘exec’, ‘exit’, ‘filter’, ‘float’, ‘format’, ‘frozenset’, ‘getattr’, ‘globals’, ‘hasattr’, ‘hash’, ‘help’, ‘hex’, ‘id’, ‘input’, ‘int’, ‘isinstance’, ‘issubclass’, ‘iter’, ‘len’, ‘license’, ‘list’, ‘locals’, ‘map’, ‘max’, ‘memoryview’, ‘min’, ‘next’, ‘object’, ‘oct’, ‘open’, ‘ord’, ‘pow’, ‘print’, ‘property’, ‘quit’, ‘range’, ‘repr’, ‘reversed’, ‘round’, ‘set’, ‘setattr’, ‘slice’, ‘sorted’, ‘staticmethod’, ‘str’, ‘sum’, ‘super’, ‘tuple’, ‘type’, ‘vars’, ‘zip’]

In Python, functions include reusable code-ordered blocks. This allows a programmer usually to write a block of code to perform a single, connected action. Although Python offers several built-in features, a programmer may build user-defined functionality.

In addition to helping us program and debug by dividing the program into small parts, the functions also allow us to manage code in a more reusable manner.

Python functions are defined using the def keyword with the function name, followed by the function parameters. The function’s body is composed of Python statements to be executed. You have the option to return a value to the function caller at the end of the function, or if you do not assign a return value, it will return the None object by default.

For instance, we can define a function that returns True if the element is within the sequence given a sequence of numbers and an item passed by a parameter, and False otherwise:

>>> def contains(sequence,item):
>>>	for element in sequence:
>>>		if element == item:
>>>			return True
>>>		return False
>>> print contains([100,200,300,400],200)
True
>>> print contains([100,200,300,400],300)
True
>>> print contains([100,200,300,400],350)
False

Two important factors make parameters different and special:

• Parameters only exist within the functions in which they were described, and the only place where the parameter can be specified is a space between a pair of parentheses in the def state.
• Assigning a value to the parameter is done at the time of the function’s invocation by specifying the corresponding argument.

Python classes

Python is an object-oriented language that allows you to create classes from such descriptions and instantiate them. The functions specified inside the class are instance methods, also known as member functions.

Python’s way of constructing objects is via the class keyword. A Python object is an assembly of methods, variables, and properties. Lots of objects can be generated with the same class description.

Here is a simple example of a protocol object definition. You can find the following code in the protocol.py file:

class protocol(object):
def __init__(self, name, number,description):
                 self.name = name
        self.number = number
        self.description = description
def getProtocolInfo(self):
         return self.name+ “ “+str(self.number)+ “ “+self.description

In the previous code, we can see a method with the name __init__, which represents the class constructor. If a class has a constructor, it is invoked automatically and implicitly when the object of the class is instantiated.

The init method is a special method that acts as a constructor method to perform the necessary initialization operation. The method’s first parameter is a special keyword, and we use the self-identifier for the current object reference. Basically, the self keyword is a reference to the object itself and provides a way for its attributes and methods to access it.

The constructor method has to have the self parameter and may have more parameters than just self; if this happens, the way in which the class name is used to create the object must reflect the __init__ definition. This method is used to set up the object, in other words, properly initialize its internal state, create instance variables, instantiate any other objects if their existence is needed, and so on.

Important note

In Python, self is a reserved language word and is mandatory. It is the first parameter of traditional methods and through it you can access the class attributes and methods. This parameter is equivalent to the pointer that can be found in languages such as C ++ or Java.

An object is a set of the requirements and qualities assigned to a specific class. Classes form a hierarchy, which means that an object belonging to a specific class belongs to all the superclasses at the same time.

To build an object, write the class name followed by any parameter needed in parentheses. These are the parameters that will be transferred to the init method, which is the process that is called when the class is instantiated:

>>> protocol_http= protocol(“HTTP”, 80, “Hypertext transfer protocol”)

Now that we have created our object, we can access its attributes and methods through the object.attribute and object.method() syntax:

>>> protocol_http.name
>>> protocol_http.number
>>> protocol_http.description
>>> protocol_http.getProtocolInfo()

In summary, object programming is the art of defining and expanding classes. A class is a model of a very specific part of reality, reflecting properties and methods found in the real world. The new class may add new properties and new methods, and therefore may be more useful in specific applications.

Python inheritance

Let’s define one of the fundamental concepts of object programming, named inheritance. Any object bound to a specific level of a class hierarchy inherits all the traits (as well as the requirements and qualities) defined inside any of the superclasses.

The core principles of the languages of object-oriented programming are encapsulation, inheritance, and polymorphism. In an object-oriented language, by creating hierarchies, objects are related to others, and it is conceivable that some objects inherit the properties and methods of other objects, expanding their actions and/or specializing.

Inheritance allows us to create a new class from another, inherit its attributes and methods, and adapt or extend them as required. This facilitates the reuse of the code since you can implement the basic behaviors and data in a base class and specialize them in the derived classes.

To implement inheritance in Python, we need to add the name of the class that is inherited within parentheses to show that a class inherits from another class, as we can see in the following code:

>>>class MyList(list):
>>>	def max_min(self):
>>>		return max(self),min(self)
>>>myList= MyList()
>>>myList.extend([100,200,300,500])
>>>print(myList)
[100, 200, 300, 500]
>>>print(myList.max_min())
(500, 100)

As we can see in the previous example, inheritance is a common practice of passing attributes and methods from the superclass to a newly created class. The new class inherits all the already existing methods and attributes, but is able to add some new ones if needed.

Managing exceptions

Each time your code tries to do something wrong, Python stops your program, and it creates a special kind of data, called an exception. Both of these activities are known as raising an exception. We can say that Python always raises an exception (or that an exception has been raised) when it has no idea what to do with your code.

Exceptions are errors that Python detects during execution of the program. If the interpreter experiences an unusual circumstance, such as attempting to divide a number by 0 or attempting to access a file that does not exist, an exception is created or thrown, telling the user that there is a problem.

When the exception is not detected, the execution flow is interrupted, and the console shows the information associated with the exception so that the developer can solve the problem with the information returned by the exception.

Let’s see a Python code throwing an exception while attempting to divide 1 by 0. We’ll get the following error message if we execute it:

>>>def division(a,b):
>>>	return a/b
>>>def calculate():
>>>division(1,0)
>>>calculate()
Traceback (most recent call last):
    File “<stdin>”, line 1, in <module>
    File “<stdin>”, line 2, in calculate
    File “<stdin>”, line 2, in division
ZeroDivisionError: division by zero

In the previous example, we can see traceback, which consists of a list of the calls that caused the exception. As we see in the stack trace, the error was caused by the call to the calculate() method, which, in turn, calls division (1, 0), and ultimately the execution of the a/b sentence of division in line 2.

Important note

Python provides effective tools that allow you to observe exceptions, identify them, and handle them efficiently. This is possible due to the fact that all potential exceptions have their unambiguous names, so you can categorize them and react appropriately.

In Python, we can use a try/except block to resolve situations related to exception handling. Now, the program tries to run the division by zero. When the error happens, the exceptions manager captures the error and prints a message that is relevant to the exception:

>>>try:
>>>	print(“10/0 = “,str(10/0))
>>>except Exception as exception:
>>>	print(“Error =”,str(exception))
Error = division by zero

The try keyword begins a block of the code that may or may not be performing correctly. Next, Python tries to perform some operations; if it fails, an exception is raised and Python starts to look for a solution.

At this point, the except keyword starts a piece of code that will be executed if anything inside the try block goes wrong – if an exception is raised inside a previous try block, it will fail here, so the code located after the except keyword should provide an adequate reaction to the raised exception.

In the following example, we try to create a file-type object. If the file is not found in the filesystem, an exception of the IOError type is thrown, which we can capture thanks to our try except block:

>>>try:
>>>	f = open(‘file.txt’,”r”)
>>>except Exception as exception:
>>>	print(“File not found:”,str(exception)) 
File not found: [Errno 2] No such file or directory: ‘file.txt’

In the first block, Python tries to perform all instructions placed between the try: and except: statements; if nothing is wrong with the execution and all instructions are performed successfully, the execution jumps to the point after the last line of the except: block, and the block’s execution is considered complete.

The following code raises an exception related to accessing an element that does not exist in the list:

>>> list = []
>>> x = list[0]
Traceback (most recent call last):
IndexError: list index out of range

Python 3 defines 63 built-in exceptions, and all of them form a tree-shaped hierarchy. Some of the built-in exceptions are more general (they include other exceptions), while others are completely concrete. We can say that the closer to the root an exception is located, the more general (abstract) it is.

Some of the exceptions available by default are listed here (the class from which they are derived is in parentheses):

• BaseException: The class from which all exceptions inherit.
• Exception (BaseException): An exception is a special case of a more general class named BaseException.
• ZeroDivisionError (ArithmeticError): An exception raised when the second argument of a division is 0. This is a special case of a more general exception class named ArithmeticError.
• EnvironmentError (StandardError): This is a parent class of errors related to input/output.
• IOError (EnvironmentError): This is an error in an input/output operation.
• OSError (EnvironmentError): This is an error in a system call.
• ImportError (StandardError): The module or the module element that you wanted to import was not found.

All the built-in Python exceptions form a hierarchy of classes. The following script dumps all predefined exception classes in the form of a tree-like printout.

You can find the following code in the get_exceptions_tree.py file:

def printExceptionsTree(ExceptionClass, level = 0):
        if level > 1:
                print(“     |” * (level - 1), end=””)
        if level > 0:
                print(“     +---”, end=””)
        print(ExceptionClass.__name__)
        for subclass in ExceptionClass.__subclasses__():
                printExceptionsTree(subclass, level + 1)
printExceptionsTree(BaseException)

As a tree is a perfect example of a recursive data structure, a recursion seems to be the best tool to traverse through it. The printExceptionsTree() function takes two arguments:

• A point inside the tree from which we start traversing the tree
• A level to build a simplified drawing of the tree’s branches

This could be a partial output of the previous script:

BaseException
     +---Exception
     |     +---TypeError
     |     +---StopAsyncIteration
     |     +---StopIteration
     |     +---ImportError
     |     |     +---ModuleNotFoundError
     |     |     +---ZipImportError
     |     +---OSError
     |     |     +---ConnectionError
     |     |     |     +---BrokenPipeError
     |     |     |     +---ConnectionAbortedError
     |     |     |     +---ConnectionRefusedError
     |     |     |     +---ConnectionResetError
     |     |     +---BlockingIOError
     |     |     +---ChildProcessError
     |     |     +---FileExistsError
     |     |     +---FileNotFoundError
     |     |     +---IsADirectoryError
     |     |     +---NotADirectoryError
     |     |     +---InterruptedError
     |     |     +---PermissionError
     |     |     +---ProcessLookupError
     |     |     +---TimeoutError
     |     |     +---UnsupportedOperation
     |     |     +---herror
     |     |     +---gaierror
     |     |     +---timeout
     |     |     +---Error
     |     |     |     +---SameFileError
     |     |     +---SpecialFileError
     |     |     +---ExecError
     |     |     +---ReadError

In the output of the previous script, we can see that the root of Python’s exception classes is the BaseException class (this is a superclass of all the other exceptions). For each of the encountered classes, performs the following set of operations:

• Print its name, taken from the __name__ property.
• Iterate through the list of subclasses delivered by the __subclasses__() method, and recursively invoke the printExceptionsTree() function, incrementing the nesting level, respectively.

Now that you know the functions, classes, and exceptions for working with Python, let’s move on to learning how to manage modules and packages. Also, we will review the use of some modules for managing parameters, including argparse and OptionParse.

In this section, you will learn how Python provides modules that are built in a modular way and offers the possibility to developers to create their own modules.

What is a module in Python?

A module is a collection of functions, classes, and variables that we can use from a program. There is a large collection of modules available with the standard Python distribution.

A module can be specified as a file containing definitions and declarations from Python. The filename is the module name attached with the .py suffix. We can start by defining a simple module in a .py file. We’ll define a simple test() function inside this my module.py file that will print “This is my first module”:

You can find the following code in the my_module.py file:

def test():
print(“This is my first module”)

Within our main.py file, we can then import this file as a module and use our newly-defined test() method, like so:

You can find the following code in the main.py file:

import my_module
def main():
my_module.test()
if __name__ == ‘__main__’:
main()

When a module is imported, its content is implicitly executed by Python. It gives the module the chance to initialize some of its internal aspects. The initialization takes place only once, when the first import occurs, so the assignments done by the module aren’t repeated unnecessarily. That’s all we need in order to define a very simple Python module within our Python scripts.

Getting information from standard modules

We continue through some standard Python modules. We could get more information about methods and other entities from a specific module using the dir() method. The module has to have been previously imported as a whole (for example, using the import module instruction):

>>>import <module_name>
>>>dir(module_name)

The dir() method returns an alphabetically sorted list containing all entities’ names available in the module identified by a name passed to the function as an argument. For example, you can run the following code to print the names of all entities within the math module. You can find the following code in the get_entities_module.py file:

import math
for name in dir(math):
        print(name, end=”\t”)

In the previous script, we are using the dir() method to get all name entities from the math module.

Difference between a Python module and a Python package

Writing your own modules doesn’t differ much from writing ordinary scripts. There are some specific aspects you must be aware of, but it definitely isn’t rocket science. When we are working with Python, it is important to understand the difference between a Python module and a Python package. It is important to differentiate between them; a package is a module that includes one or more modules.

Let’s summarize some important concepts:

• A module is a kind of container filled with functions – you can pack as many functions as you want into one module and distribute it across the world.
• Of course, it’s generally a good idea not to mix functions with different application areas within one module, so group your functions carefully and name the module containing them in a clear and intuitive way.

Python Module Index

Python comes with a robust standard library that includes everything from built-in modules for easy I/O access to platform-specific API calls. Python’s modules make up their own universe, in which Python itself is only a galaxy, and we would venture to say that exploring the depths of these modules can take significantly more time than getting acquainted with “pure” Python. You can read about all standard Python modules here: https://docs.python.org/3/py-modindex.html.

Managing parameters in Python

Often in Python, scripts that are used on the command line as arguments are used to give users options when they run a certain command. Each argument that is provided to a Python script is exposed through the sys.argv array, which can be accessed by importing the sys module.

However, to develop this task, the best option is to use the argparse module, which comes installed by default when you install Python. For more information, you can check out the official website: https://docs.python.org/3/library/argparse.html.

You can find the following code in the testing_parameters.py file:

import argparse
parser = argparse.ArgumentParser(description=’Testing parameters’)
parser.add_argument(“-p1”, dest=”param1”, help=”parameter1”)
parser.add_argument(“-p2”, dest=”param2”, help=”parameter2”) 
params = parser.parse_args()
print(“Parameter 1”,params.param1)
print(“Parameter 2”,params.param2)

One of the interesting choices is that the type of parameter can be indicated using the type attribute. For example, if we want to treat a certain parameter as if it were an integer, then we might do so as follows:

parser.add_argument(“-param”, dest=”param”, type=”int”)

Another thing that could help us to have a more readable code is to declare a class that acts as a global object for the parameters. For example, if we want to pass several parameters at the same time to a function, we could use this global object, which is the one that contains the global execution parameters.

You can find the following code in the params_global_argparse.py file:

import argparse
class Parameters:
“””Global parameters”””
def __init__(self, **kwargs):
     self.param1 = kwargs.get(“param1”)
     self.param2 = kwargs.get(“param2”)
def view_parameters(input_parameters):
print(input_parameters.param1)
print(input_parameters.param2)
parser = argparse.ArgumentParser(description=’Passing parameters in an object’)
parser.add_argument(“-p1”, dest=”param1”, help=”parameter1”)
parser.add_argument(“-p2”, dest=”param2”, help=”parameter2”)
params = parser.parse_args()
input_parameters = Parameters(param1=params.param1,param2=params.param2)
view_parameters(input_parameters)

In the previous script, we can see that with the argparse module, we obtain parameters and we encapsulate these parameters in an object with the Parameters class.

Python provides another class called OptionParser for managing command-line arguments. OptionParser is part of the optparse module that is provided by the standard library. OptionParser allows you to do a range of very useful things with command-line arguments:

• Specify a default if a certain argument is not provided.
• It supports both argument flags (either present or not) and arguments with values.
• It supports different formats of passing arguments.

Let’s use OptionParser to manage parameters in the same way we have seen before with the argparse module. In the code provided here, command-line arguments are used to pass in these variables:

You can find the following code in the params_global_OptionsParser.py file:

from optparse import OptionParser
class Parameters:
        “””Global parameters”””
        def __init__(self, **kwargs):
                self.param1 = kwargs.get(“param1”)
                self.param2 = kwargs.get(“param2”)
def view_parameters(input_parameters):
        print(input_parameters.param1)
        print(input_parameters.param2)
parser = OptionParser()
parser.add_option(“--p1”, dest=”param1”, help=”parameter1”)
parser.add_option(“--p2”, dest=”param2”, help=”parameter2”)
(options, args) = parser.parse_args()
input_parameters = Parameters(param1=options.param1,param2=options.param2)
view_parameters(input_parameters)

The previous script demonstrates the use of the OptionParser class. It provides a simple interface for command-line arguments, allowing you to define certain properties for each command-line option. It also allows you to specify default values. If certain arguments are not provided, it allows you to throw specific errors.

Now that you know how Python manages modules and packages, let’s move on to learning how to manage dependencies and create a virtual environment with the virtualenv utility.

In this section, you will be able to identify how to manage dependencies and the execution environment with pip and virtualenv.

Managing dependencies in a Python project

If our project has dependencies with other libraries, the goal will be to have a file where we have such dependencies, so that our module is built and distributed as quickly as possible. For this function, we will build a file called requirements.txt, which will have all the dependencies that the module in question requires if we invoke it with the pip utility.

To install all the dependencies, use the pip command:

$ pip -r requirements.txt

Here, pip is the Python package and dependency manager where requirements.txt is the file where all the dependencies of the project are saved.

Tip

Within the Python ecosystem, we can find new projects to manage the dependencies and packages of a Python project. For example, poetry (https://python-poetry.org) is a tool to handle dependency installation as well as build and package Python packages.

Generating the requirements.txt file

We also have the possibility to create the requirements.txt file from the project source code. For this task, we can use the pipreqs module, whose code can be downloaded from the GitHub repository at https://github.com/bndr/pipreqs.

In this way, the module can be installed either with the pip install pipreqs command or through the GitHub code repository using the python setup.py install command.

For more information about the module, you can refer to the official PyPI page:

https://pypi.python.org/pypi/pipreqs

To generate the requirements.txt file, you have to execute the following command:

$ pipreqs <path_project>

Working with virtual environments

When operating with Python, it’s strongly recommended that you use virtual environments. A virtual environment provides a separate environment for installing Python modules and an isolated copy of the Python executable file and associated files.

You can have as many virtual environments as you need, which means that you can have multiple module configurations configured, and you can easily switch between them.

From version 3, Python includes a venv module, which provides this functionality. The documentation and examples are available at https://docs.python.org/3.8/using/.

There is also a standalone tool available for earlier versions, which can be found at https://virtualenv.pypa.io/en/latest.

Configuring virtualenv

When you install a Python module on your local computer without having to use a virtual environment, you install it on the operating system globally. Typically, this installation requires a user root administrator and the Python module is configured for each user and project.

The best approach at this point is to create a Python virtual environment if you need to work on many Python projects, or if you are working with several projects that are sharing some modules.

virtualenv is a Python module that enables you to build isolated, virtual environments. Essentially, you must create a folder that contains all the executable files and modules needed for a project. You can install virtualenv as follows:

1. Type in the following command:

   $ sudo pip install virtualenv

2. To create a new virtual environment, create a new folder and enter the folder from the command line:

   $ cd your_new_folder
   $ virtualenv name-of-virtual-environment
   $ source bin/activate

3. Once we have it active, we will have a clean environment of modules and libraries and we will have to download the dependencies of our project so that they are copied in this directory using the following command:

   (venv) > pip install -r requirements.txt

   Executing this command will initiate a folder with the name indicated in your current working directory with all the executable files of Python and the pip module that allows you to install different packages in your virtual environment.

   Important note

   If you are working with Python 3.3+, virtualenv is included in stdlib. You can get an installation update for virtualenv in the Python documentation: https://docs.python.org/3/library/venv.html.

virtualenv is like a sandbox where all the dependencies of the project will be installed when you are working, and all modules and dependencies are kept separate. If users have the same version of Python installed on their machine, the same code will work from the virtual environment without requiring any change.

Now that you know how you can install your own virtual environment, let’s move on to review development environments for Python scripting, including Python IDLE and PyCharm.

In this section, we will review PyCharm and Python IDLE as development environments for Python scripting.

Setting up a development environment

In order to rapidly develop and debug Python applications, it is absolutely necessary to use an Integrated Development Environment (IDE). If you want to try different options, we recommend you check out the list that is on the official site of Python, where you can see the tools according to your operating systems and needs:

https://wiki.python.org/moin/IntegratedDevelopmentEnvironments

Between all the environments, the following two are what we will look at:

• PyCharm: http://www.jetbrains.com/pycharm
• Python IDLE: https://docs.python.org/3/library/idle.html

PyCharm

PyCharm is an IDE developed by Jetbrains, based on the company’s IntelliJ IDEA, the same company’s IDE, but focused on Java, and is the Android Studio base.

PyCharm is multi-platform and we can find binaries for operating systems running Windows, Linux, and macOS X. There are two versions of PyCharm – community and technical, with variations in functionality relating to web framework integration and support for databases. In the following URL, we can see a comparison between both editions:

http://www.jetbrains.com/pycharm

The main advantages of this development environment are as follows:

• Autocomplete, syntax highlighter, analysis tool, and refactoring
• Integration with web frameworks such as Django and Flask
• An advanced debugger
• Connection with version-control systems, such as Git, CVS, and SVN

In the following screenshot, we can see how to configure virtualenv in PyCharm:

Figure 1.1 – Configuring virtualenv in PyCharm

In the preceding screenshot, we are setting the configuration related to establishing a new environment for the project using virtualenv.

Debugging with PyCharm

In this example, we are debugging a Python script that accepts two input parameters. An interesting topic is the possibility of adding a breakpoint to our script.

In the following screenshot, we are setting a breakpoint in the view_parameters method:

Figure 1.2 – Setting a breakpoint in PyCharm

With the View Breakpoint option, we can see the breakpoint established in the script:

Figure 1.3 – Viewing breakpoints in PyCharm

In the following screenshot, we can visualize the values of the parameters that contain the values we are debugging:

Figure 1.4 – Debugging variables in PyCharm

In this way, we can know the state of each of the variables at runtime, as well as modify their values to change the logic of our script.

Debugging with Python IDLE

Python IDLE is the default IDE that comes installed by default when you install Python in your operating system. When executing Python IDLE, it offers the possibility to debug your script and see errors and exceptions in the Python shell console:

Figure 1.5 – Running a script in the Python shell

In the preceding screenshot, we can see the output in the Python shell and the exception is related to File not found.

In this chapter, we learned how to install Python on the Windows and Linux operating systems. We reviewed the main data structures and collections, such as lists, tuples, and dictionaries. We also reviewed functions, managing exceptions, and how to create classes and objects, as well as the use of attributes and special methods. Then we looked at development environments and a methodology to introduce into programming with Python. Finally, we reviewed the main development environments, PyCharm and PythonIDLE, for script development in Python.

In the next chapter, we will explore programming system packages for working with operating systems and filesystems, threads, and concurrency.

As we conclude, here is a list of questions for you to test your knowledge regarding this chapter’s material. You will find the answers in the Assessments section of the Appendix:

1. What data structure in Python allows us to associate values with keys?
2. How can we debug variables in Python development environments?
3. What is the Python class from which all exceptions inherit?
4. Which method returns an alphabetically sorted list containing all entities’ names that are available in a specific module?
5. Which class does Python provide from the optparse module for managing command-line arguments?

In these links, you will find more information about theafore mentioned tools and the official Python documentation for some of the modules we have analyzed:

• Python 3.7 version library: https://docs.python.org/3.7/library/
• Virtualenv documentation: https://virtualenv.pypa.io/en/latest/
• Python Integrated Development Environments: https://wiki.python.org/moin/IntegratedDevelopmentEnvironments