This section primarily provides a basic understanding of the Python language constructs. If you feel you already have a solid grasp of Python, feel free to skip ahead.

Let's get down to the nuts and bolts.

A Python program is written as source code in one or more .py files and consists of statements and expressions as shown in the following screenshot:

Both statements and expressions tell Python to do something. The difference is that expressions can be combined to form more complex expressions, while statements can be combined with expressions, but not with other statements.

For example, a statement looks like this:

if 2 > 1: 

An expression looks like this:

print ("One is the loneliest number") 

Python source code files are executed from top to bottom as soon as they're loaded by the Python runtime. This means that for simple programs, we could just write a series of statements in a .py file and then tell Python to run them. In the preceding example, the if and else parts are statements or a single statement with two parts, if you prefer to think of it that way. Everything else is an expression. For more complex programs, we need a more structured approach.

Like most programming languages, Python lets us create functions and classes in order to organize our code.

To organize our code, we can divide it into four basic building blocks. We'll discuss each of these separately for understanding their role and importance in the Python code. These are as follows:

• Functions
• Variables
• Expressions
• Classes

We'll start with a brief look at functions. Functions are created using a def statement, which is a statement using the def keyword as its identifying component. As I said earlier, Python executes the statements in a .py file, starting from the top, as shown in the following screenshot:

When Python executes a def statement, it creates a function as a result. This means that the code that runs before the def statement does not see the function because it doesn't exist yet. The part of the def line inside parentheses is called the parameter list:

example_function(name, radius): 

The parameter list is a list of internal names for data values that are given to the function as the input. Outside the function, these values might have different names or no names at all, but inside, they'll be stored in these variables.

The indented block of code immediately after the def line is called the function body, and you could think of it as the source code of the function:

def example_function(name, radius): 
    area = math.pi * radius ** 2 
    return "The area of {} is {}" .format(name, area) 

The following screenshot shows the output of the preceding example:

The code inside the function body is an exception to the rule about running Python code from the top to the bottom of the file. This code is stored away and then executed later, when we tell the function to run.

If you're more familiar with C++ or Java, you may be wondering where the function parameter types and return types are. In Python, the data type is inherent in each data value, so the runtime always knows what type of data we're working with and whether what we're trying to do is a valid operation for that data type. Thus, for the most part, we don't need explicit data types.

Python programmers sometimes talk about duck typing, which is a reference to the following saying:

What they mean by this saying is that if the operations we're trying to perform on a data value work, it doesn't really matter if it's precisely the kind of data we expected. It's probably close enough. If they don't work, Python will tell us what went wrong and where, which is often more useful to know than the kind of information that can be determined by comparing data types alone.

Function decorators, which we'll discuss in later chapters, can provide a convenient way of enforcing these annotations.

The second major building block of a Python program is called a variable. A variable is pretty much just a box for storing a data value. The variable has a name and we can use that name to access the data stored in the variable or to replace the data with a new value.

The function parameters in the previous examples were variables, as was area:

(name, radius):  

To set the data stored in a variable, we use an assignment statement. An assignment is a statement, so remember this means that it can't be combined with any other statement. It gets a line of source code all for itself and the expressions that are part of it.

An assignment statement consists of the variable's name on the left-hand side of an equal to symbol and the value we want to store in the variable on the right-hand side, as shown in the following code:

outer = "Hello world"  

If the variable didn't already exist, it will be created. Irrespective of whether the variable existed before or not, the value is stored in the variable.

Variables that are created inside a function are only visible inside that function and each time the function runs they're created a new.

The following code provides an example of this in action:

outer = "Hello world"  
def example_function(param): 
    inner = "Hello function: {}".format(param) 
    print(inner, outer) 
example_function("first") 
example_function("second") 
print(inner) 

The last line of the preceding example demonstrates that the variable created inside the function does not exist for code outside the function, as shown in the following output of the code:

This code example also shows what happens when we try to ask Python to do something impossible. It tells us what we did wrong and gives us the information about where the problem occurred and how we got there.

The third major building block of Python programs is expressions. We've seen expressions in every example so far because it's nearly impossible to do anything in Python without using expressions.

Expressions consist of data values and operations to perform on those data values. The very simple expressions are a single data value and with no operations, for example, a single number. More complex expressions involve at least one operation and probably more data values as well, for example, adding two numbers or calculating the area, as shown in the following code example:

import math 
 
def example_function(name: str, radius: float) -> str: 
  area = math.pi * radius ** 2 
  return "The area of {} is {}" .format(name, area)  
 
print(example_function('Bob', 5)) 

All expressions produce a resulting data value of some sort; for example, adding two numbers produces the sum as another number, while concatenating two text strings produces the concatenation as another text string. Using a name variable to look up the stored value is an expression, so is running a function.

Anywhere we need a value, we can use any expression that produces the needed value. It doesn't matter whether the expression is a simple number, such as 55, a variable name, a complex combination of values and operators, a function call, or any other expression. At least, it doesn't matter as far as the final result is concerned. Some expressions take less time to execute than others, so speed can be a factor.

The final fundamental building block we're going to discuss in this section is classes. The word class is a synonym for category or type; in this case, it is referring to data values.

A class defines a new kind of data value by describing a set of internal data and operations for that type of data value. This is done primarily by defining a group of functions that make up the class. A special function called __init__ is used to set up the internal data for a new data value of this type, and the rest of the functions define the operations on an existing data value of this type:

class Frood: 
    def __init__(self, age): 
        self.age = age 
        print("Frood initialized") 
 
    def anniversary(self): 
        self.age += 1 
        print("Frood is now {} years old".format(self.age)) 
 
f1 = Frood(12) 
f2 = Frood(97) 
f1.anniversary() 
f2.anniversary() 
f1.anniversary() 
f2.anniversary() 

All the functions of a class receive a parameter called self, as shown in the preceding code example for classes. This parameter is the data value being operated on. That's different from C++ or Java because while those languages do basically the same thing, the parameter is implicit instead of being an explicit part of the function's parameter list.

Class functions, including __init__, should store and retrieve data from self when they want to manipulate the data value that they're connected to.

In the preceding example, we created a new data type called Frood and then made two separate data values of that type. Then, we used the anniversary function that we created as part of the class to modify each of them.

The output of the code example for classes is as follows:

Python has several flow control statements that will be familiar to people who know another language in the C family. For example, Python has loops and if, elif, and else branches (shown in the following code example):

selector = 5 
 
if selector < 3: 
    print("less than three") 
elif selector < 6: 
    print("less than six") 
else: 
    print("six or more") 
while selector > 0" 
    print('selector is {}' .format(selector)) 
    selector -=1 
 
for x in ['a', 'b', 'c', 'd']: 
    print(x) 
for x in range(5): 
    print(x) 

Python also has a for loop statement, but it's not like the for loops in C, C++, or Java. Instead of counting through numbers, the for loop iterates through the values. If we actually want to count through numbers with a for loop, that's easily done using a range iterator, as shown in the following screenshot in the output of the preceding code example:

Before we wrap-up this section, there's one last thing I should comment on and that's Python's views on indentation to signify the block structure.

Most other programming languages have explicit symbols that indicate the beginning of a block and the end of a block. However, it's a common practice in all of those languages to indent blocks so that humans find the code easier to read. In fact, failure to do so is often taken as a sign that a programmer is an amateur. This means that the block structure in most languages is actually represented in two different ways: the symbols and the indentation. By incorporating indentation into syntax without the need for explicit symbols, Python both removes this duplication and ensures that the code is readable.

With that, we've come to the end of this section. In the next section, we'll look at some of Python's built-in data structures and the data processing syntax.

Now, let's take a look at the core data structure types of Python. These aren't the only data structures available, of course, because it's fairly easy to create data structures using classes. However, these data structures are built right into the heart of Python and they're highly efficient, so it's a good idea to be very familiar with them.

The first thing to understand is that data structures are themselves data values similar to a filing cabinet-they're one thing that contains many things. Like any other data value, they can be stored in a variable or used as part of an expression.

The first data structure we're going to look at is Python's dictionary. A dictionary consists of any number of key-value pairs. The key can be used to get or set the value or remove the pair from the dictionary entirely.

There are several ways to create a dictionary in Python. The simplest is to use a dictionary expression, which is just a pair of curly brackets surrounding the key-value pairs we want in the dictionary. Each key-value pair is marked with a colon between the key and value, and each pair is separated by a comma, as shown in the following code example:

example_dict = {'a' :1, 'b' :2, 'c' :3} 

When this expression runs, the result is a dictionary object containing the keys and their values. We can also use the dict class to create dictionary objects:

another_dict = dict() 

If we don't want to use the special syntax to access one of the stored values in a dictionary, we use a lookup expression. This means that we place square brackets containing the key we want to look up after an expression that gives us the dictionary. Usually, this means, the name of the variable containing the dictionary, an open square bracket, a sub-expression that gives us the key, and then a closing square bracket:

example_dict['b'] 
2 

We can also use the dict.get function if we prefer not to use the special syntax:

example_dict.get('c') 
3 

The next data type we're going to look at is a list, which can be created with the list expression. A list expression is just a pair of square brackets surrounding the data values we want to store in the list, with each value separated by a comma. It is not necessary that each of the values be of the same type. The code example for a list is as follows:

In the preceding example, they're strings, but they could be numbers, or a list, or any other kind of data mixed together. We can use a lookup expression to retrieve data values.

Unlike with a dictionary though, the keys for a list are integers. That's because instead of associating key values with data values, a list just stores its data values in order. The key for the very first item in the list is 0. The key for the next item is 1, and so on. We can also use negative integers for the key. We still get a data value out, but it's counted from the end of the list instead of the beginning, with the item at -1 being the last item in the list.

We can use the list.append function to add a new item at the end of the list or its insert function to add a new item anywhere as shown in the following code:

The list will automatically grow to be large enough to hold all of the data we put into it.

The next data structure we'll look at is the tuple. A tuple expression is any sequence of value expressions separated by commas if it happens in the place where the language wasn't already expecting to see a comma.

However, it's common and smart to put parentheses around most tuple expressions because it avoids ambiguity. The code example for tuple is as follows:

Like a list, data values can be retrieved from a tuple using numbers. However, we can't add more data to a tuple, and we cannot replace one data value with another.

Why would we want a data structure like that?

Well, there are several reasons. We can list them as follows:

• First, because they are constant, tuples make good dictionary keys or set members, but we'll get to that in a while.
• Second, they fill a different conceptual role than lists. We tend to expect every member of a list to be the same type, such as a list of names or a list of ages. Lists are sort of like columns of a database in that way. We tend to expect tuples to contain different types of data in each element, but we expect them to be related to each other, such as a name in the first element and an age in the second. To continue our analogy, tuples are something like rows in a database.
• Third, tuples tend to be slightly more efficient for the computer to work with, both in terms of time and memory usage. So, in optimization situations, they are preferable to lists when they are sufficient for the task.

The final data structure we'll look at is the set. A set is a collection of data values without keys; like a list, but in no particular order, like a dictionary. We can create a set using a set expression, which is a pair of curly brackets around comma-separated values, as shown in the following code example:

Locating a specific value in a set is fast, as is adding or removing a value, as shown in the following example:

Each value can only be in the set once. Sets support a bunch of mathematical operations, such as union and intersection, and are generally more useful than might be obvious at first, though we can't really prove that here in this section.

Python has a special kind of expression called a comprehension. Comprehensions are variations of the special syntax for creating dictionaries, lists, and sets.

Let's look at some examples. Here we see a list comprehension:

capitals = [x.upper() for x in example_list] 

What this expression does is that it creates a new list containing the uppercase versions of the words in the old list.

The first part after the opening square bracket is an x.upper() expression. This expression describes how to derive a member of the new list from a member of the old list. After that is the for keyword, then the name of the x variable we used in the first expression to represent the values from the old list. Then, the keyword is followed by the example_list expression that gives us the old list and the closing square bracket. The code output is as follows:

The dictionary and set comprehensions are very similar. If we want to use both the key and the value of an existing dictionary in a comprehension, we need to use the dict.items function, and dictionary comprehensions need to specify both the key and value separated by a colon, as shown in this example:

squares = {k: v ** 2 for k, v in example_dict.items()} 

As shown in the following screenshot, notice that the resulting data type depends on what sort of comprehension we used, not on what sort of data structure we used as the source of data:

We can use a list comprehension to create a list of data pulled from the values of a dictionary, for example, or, as we did here, we can use the set comprehension to create a set.

Tuples are slightly different, but only slightly. A tuple comprehension would look exactly like a different syntactic element called a generator expression. The code example for tuple comprehension is as follows:

Python's designers hate ambiguity; so instead, if we want the equivalent of a tuple comprehension, we pass a generator expression to a tuple constructor.

That's it for this quick introduction to Python's built-in data structures. In the next section, we're going to look at some useful, but possibly surprising, traits of functions and classes that are significantly different from C, C++, or Java.

In Python, functions and classes are first-class objects. The phrase first-class object is a fancy way of saying that the data values can be accessed, modified, stored, and otherwise manipulated by the program they are a part of. In Python, a function is just as much a data value as a text string is. The same goes for classes.

When a function definition statement is executed, it stores the resulting function in a variable with the name that was specified in the def statement, as shown in the following screenshot:

This variable isn't special; it's just like any other variable holding the value. This means that we can use it in expressions, assign the value to other values, or even store a different value in place of the original function.

The function value itself contains quite a few attribute variables, which we can access. More usefully, most of the time, we can add attributes to a function object, allowing us to store custom information about a function as part of the function and access that information later, as shown in the following code example:

One common task that first-class functions make easy is assigning handlers to events. To bind the handler function to an event in Python, we just pass the function object as a parameter when we call the binding function, as shown here:

That's a significant improvement over the hoops that C++ or Java imposes on us to do something similar. As function definition statements, class definition statements create a class object and store it in a variable. This can be confusing at first. Classes describe the type of object, how can they be objects themselves?

Think of it this way-a blueprint for a house describes the type of building, but the blueprint is still a thing in its own, right? It's the same with class objects. This means that like function objects, class objects can be stored in variables, and otherwise, be treated as data values. Most interestingly, they could be used as parameters to function calls.

As an example of why that's interesting, consider this-Python's standard library contains a data structure class called defaultdict, which is like a dictionary except, when we try to look up a key that isn't already in the dictionary. It creates a new value and adds it to the dictionary, before returning it to the code that tried the lookup, as shown here:

How does the defaultdict class know how to create the default value?

The defaultdict class knows because we gave it class as a parameter when we created the defaultdict class. Thus, if we want a dictionary of list, we can give the defaultdict class the list class, as its how to make a default parameter. As an aside, defaultdict can also work with a function, as its how to make a default parameter.

The defaultdict class actually doesn't care what that parameter is, as long as the object we passed can create a new object whenever the defaultdict class needs a new default. This is an example of the duck typing we mentioned in the previous section. It doesn't matter whether the parameter is a function, a class, or anything else, so long as it behaves properly. If it doesn't behave properly, we'll be told what went wrong and where.

We discussed a little while ago that we could add attributes to function objects, which is often handy. We could do something similar with classes, with one big difference- attributes that we add to functions are only visible to the code that has access to that function object, which usually doesn't include the code of the function itself, but attributes that we add to class objects are visible to any code that has access to the class object or to an object of the type described by the class.

This means that if we add an attribute to a class, the functions defined in that class will be able to access that attribute through the self parameter, as shown in the following code example:

We need to be careful when adding attributes to classes because if we accidentally overwrite one of the class' attributes, we could break the class.

We have a greater ability to manipulate classes than functions. So, we need to use that ability more thoughtfully. Also, notice that, in this example, one of the attributes we added to the class is a function, which then proceeded to work exactly as if it had been defined as a part of the class from the beginning.

Next, let's take a short tour of some of the highlights of Python's standard library.

The library of code that comes pre-installed with Python is extensive, so we're not going into the details. The goal here is to come away with an understanding of the breadth of quality tools we have available, so if we need them in the future then we know where to look. Thus, we're going to just briefly touch on many useful things. You can find the official documentation on the standard library at https://docs.python.org/3/library/index.html.

The index page contains a list of the different packages available to you in Python's standard library. Let's briefly run through them in order.

First of all, there is the Collections package, which contains even more data structures: https://docs.python.org/3/library/collections.html.

The Collections package contains the defaultdict class that we spoke about in the previous section. The Collections package also contains an OrderedDict parameter that remembers the order in which the items were inserted and gives them back in the same order when we iterate over it. A deque class is a variation on tuples that uses names to access the element and a PseudoDict parameter that provides a composite view of several other dictionaries.

There are a few other data structures in there as well. One common data structure missing from the collections package is a PriorityQueue parameter, but that's just because it has its own package called heapq:

https://docs.python.org/3/library/heapq.html

Python's PriorityQueue operations are implemented as functions that work with built-in lists to add and remove items according to the heap property.

Storing and retrieving data is an extremely common need for programs and the pickle package makes it easy:

https://docs.python.org/3/library/pickle.html

Inside the pickle package are classes and functions that facilitate transforming arbitrary Python data into a sequence of bytes that can be stored in a file, sent across the network, or whatever you need. The pickle package also has the tools to reverse the process and transform those bytes back into fully-fledged Python data objects.

Also, in the vein of storing data, the sqlite3 package provides complete access to the SQLite database manager, allowing us to take advantage of a complete transactional relational database:

https://docs.python.org/3/library/sqlite.html

Third-party packages to access other database systems follow pretty much the same interface, so it's easy to make the switch to a different database later, if needed.

The json package is also relevant to data handling. It parses or generates the de facto standard Internet Data Exchange (IDX) format:

https://docs.python.org/3/library/json.html

The json package is smart, so it handles JSON (JavaScript Object Notation) objects, arrays, strings, numbers, null values, and so on, in a sensible way.

Mapping them onto the proper Python datatypes, the base64 package encodes bytes into base64, or decodes base64 into bytes:

https://docs.python.org/3/library/base64.html

There are several other similar packages for binhex, uu code, and so on, as well.

The html and xml packages provide all sorts of utilities for dealing with the major internet markup languages, including parsers and the document object model:

https://docs.python.org/3/library/html.html

The urllib package provides us with convenient ways to retrieve data from URLs or to send data to them:

https://docs.python.org/3/library/urllib.html

In particular, the urllib.request.url open function is extremely useful.

The itertools and functools packages provide an assortment of utilities having to do with functional programming paradigms:

https://docs.python.org/3/library/itertools.html

In particular, the functools package allows for us to create partially applied functions and the itertools package lets us concatenate iterators.

The enum package contains support for creating and using named enumerations:

https://docs.python.org/3/library/enum.html

Each enumeration is a distinct data type, like a class.

The pathlib package contains classes and functions that provide a cross-platform abstraction of file and file path operations:

https://docs.python.org/3/library/pathlib.html

The inspect package is interesting and quite useful. It provides us with functions that can be used to gather information about data objects, particularly about functions and classes. If we want to know the names of functions, parameters, or we want to access an object's documentation, or any number of things along those lines, the inspect package will get us there:

https://docs.python.org/3/library/inspect.html

The packages we just looked at are by no means the complete list of what's available in the standard library, but hopefully they give some idea of the depth and breadth of what we get just by installing Python. Taking a look through the library documentation on https://www.python.org/ is highly recommended for people who want to get the most out of Python. There are some particularly useful packages that we didn't mention at all. That's because there are whole other sections devoted to them later in the book.

So, that brings us to the end of our overview of the standard library.

In this section, we're going to take a look at a few of the changes that have occurred in the latest releases of Python, specifically we will look at these:

• The syntactic changes
• The changes in the packages
• Other changes

Let's get started!

Since version 3.5, Python has three new groups of syntactic editions. The first of these groups is the introduction of keywords for describing coroutines. Python already supported coroutines, but the keywords make things clear and sometimes simpler. We'll be discussing coroutines in depth in a later chapter, so we won't go into this any further now.

The second piece of the new syntax is the introduction of the @ symbol as an infix binary operator. This means that placing an @ symbol between two sub-expressions is now a valid Python expression, just like placing a + symbol between the sub-expressions would be as shown in the following screenshot:

However, since no built-in data type supports the @ symbol operator yet, we won't be finding much use for it in this book. The intended semantic meaning of the @ symbol is that it should represent matrix multiplication and it was added to improve support for an interoperability between third-party packages that implement matrixes and matrix operations.

The third piece of new syntax is an expansion of Python's pre-existing syntax for using lists and dictionaries to provide the parameter values when invoking a function.

Before, it was possible to put an asterisk (*) before a list of values to indicate that those values should be assigned to the parameters in the same order that they appeared in the list. Here is the code example for a single asterisk:

Similarly, * before two values was used to indicate that the values in a dictionary with text string keys should be assigned to the function's parameters by name, as shown here:

The new syntax is just that we can now use more than one list or dictionary in this way, and that we can use the same asterisk and double asterisk syntax for constructing tuples, lists, dictionaries, and sets.

We mentioned earlier that while Python attaches data types to data values rather than variables, it is possible to use function annotations to describe the expected types of function parameters' return values.

Python now includes a package called typing in the standard library that contains classes and functions supporting the usage of type hints.

Python also includes a package called zipapp in the standard library.

The zipapp package makes it easy to construct .pyz files. A .pyz file is an archive file containing Python code and arbitrary read-only data, which the Python runtime is able to execute as a self-contained program. Once a program is debugged and ready for distribution, packaging it into a .pyz file is a simple and smart way to hand it to the users.

Some low-level improvements have been made in Python since version 3.5, such as faster reading of filesystem directories, automatic retrying of interrupted operating system calls, and a math.isclose function for checking whether two numbers are approximately equal.

There are also a bunch of more minor additions that improve things throughout the standard library, all backwards compatible with earlier Python 3 versions.

In the rare cases where something is added that breaks backwards compatibility, it's not enabled by default. For such a change, if we want to use it, we'd have to specifically mark our code as supporting the change. Those changes will not become standard until two versions later, so a breaking change in Python 3.5 would not become the default until Python version 3.7, with Python 3.5 and 3.6 issuing warnings when they encounter code that depends on the changing feature.

In Python 3.5, there was only one such change-a small and smart alteration in the iteration protocol. It shouldn't have any effect on code that works properly, but technically it's a change in the interface and so it gets the full wait two versions treatment.

If you want more detail about any of these changes I've mentioned, or if you ever want to find out what's changed between versions of Python, the documentation on https://docs.python.org/3/ always contains a what's new document that goes into some detail about new features and provides links to the full documentation.

I always look forward to reading the what's new document for each release of Python, to find out what new toys I've just been handed.

So, we've now taken a high-level view of the Python standard library, introducing us to some of the more useful items it contains. That brings us to the end of our Python primer.

In this chapter, we looked at some fundamentals of the Python programming language. We have seen how to create and access those data structures, and how to use comprehensions to create and transform data structures based on existing ones.

We looked briefly at what it means for Python to have first-class functions and classes, and how that can affect the possibilities open to us as programmers.

We briefly talked about some of the high points of the Python standard library. We also quickly covered the syntax, basic assumptions, and fundamental tools of the Python programming language.

In the next chapter, we're going to see how to set up a Python programming environment for us to work in for the remainder of the course and learn a bit about how to integrate third-party code.