Most users of the pandas library will use an import alias so they can refer to it as pd. In general, in this book, we will not show the pandas and NumPy imports, but they look like this:

import pandas as pd
import numpy as np

While it is an optional dependency in the 2.x series of pandas, many examples in this book will also leverage the PyArrow library, which we assume to be imported as:

import pyarrow as pa

The basic building block in pandas is a pd.Series, which is a one-dimensional array of data paired with a pd.Index. The index labels can be used as a simplistic way to look up values in the pd.Series, much like the Python dictionary built into the language uses key/value pairs (we will expand on this and much more pd.Index functionality in Chapter 2, Selection and Assignment).

The following section demonstrates a few ways of creating a pd.Series directly.

How to do it

The easiest way to construct a pd.Series is to provide a sequence of values, like a list of integers:

pd.Series([0, 1, 2])

0    0
1    1
2    2
dtype: int64

A tuple is another type of sequence, making it valid as an argument to the pd.Series constructor:

pd.Series((12.34, 56.78, 91.01))

0    12.34
1    56.78
2    91.01
dtype: float64

When generating sample data, you may often reach for the Python range function:

pd.Series(range(0, 7, 2))

0    0
1    2
2    4
3    6
dtype: int64

In all of the examples so far, pandas will try and infer a proper data type from its arguments for you. However, there are times when you will know more about the type and size of your data than can be inferred. Providing that information explicitly to pandas via the dtype= argument can be useful to save memory or ensure proper integration with other typed systems, like SQL databases.

To illustrate this, let’s use a simple range argument to fill a pd.Series with a sequence of integers. When we did this before, the inferred data type was a 64-bit integer, but we, as developers, may know that we never expect to store larger values in this pd.Series and would be fine with only 8 bits of storage (if you do not know the difference between an 8-bit and 64-bit integer, that topic will be covered in Chapter 3, Data Types). Passing dtype="int8" to the pd.Series constructor will let pandas know we want to use the smaller data type:

pd.Series(range(3), dtype="int8")

0    0
1    1
2    2
dtype: int8

A pd.Series can also have a name attached to it, which can be specified via the name= argument (if not specified, the name defaults to None):

pd.Series(["apple", "banana", "orange"], name="fruit")

0     apple
1     banana
2     orange
Name: fruit, dtype: object

While pd.Series is the building block, pd.DataFrame is the main object that comes to mind for users of pandas. pd.DataFrame is the primary and most commonly used object in pandas, and when people think of pandas, they typically envision working with a pd.DataFrame.

In most analysis workflows, you will be importing your data from another source, but for now, we will show you how to construct a pd.DataFrame directly (input/output will be covered in Chapter 4, The pandas I/O System).

How to do it

The most basic construction of a pd.DataFrame happens with a two-dimensional sequence, like a list of lists:

pd.DataFrame([
    [0, 1, 2],
    [3, 4, 5],
    [6, 7, 8],
])

    0   1   2
0   0   1   2
1   3   4   5
2   6   7   8

With a list of lists, pandas will automatically number the row and column labels for you. Typically, users of pandas will at least provide labels for columns, as it makes indexing and selecting from a pd.DataFrame much more intuitive (see Chapter 2, Selection and Assignment, for an introduction to indexing and selecting). To label your columns when constructing a pd.DataFrame from a list of lists, you can provide a columns= argument to the constructor:

pd.DataFrame([
    [1, 2],
    [4, 8],
], columns=["col_a", "col_b"])

     col_a    col_b
0    1          2
1    4          8

Instead of using a list of lists, you could also provide a dictionary. The keys of the dictionary will be used as column labels, and the values of the dictionary will represent the values placed in that column of the pd.DataFrame:

pd.DataFrame({
    "first_name": ["Jane", "John"],
    "last_name": ["Doe", "Smith"],
})

            first_name      last_name
0           Jane            Doe
1           John            Smith

In the above example, our dictionary values were lists of strings, but the pd.DataFrame does not strictly require lists. Any sequence will work, including a pd.Series:

ser1 = pd.Series(range(3), dtype="int8", name="int8_col")
ser2 = pd.Series(range(3), dtype="int16", name="int16_col")
pd.DataFrame({ser1.name: ser1, ser2.name: ser2})

             int8_col         int16_col
0            0                0
1            1                1
2            2                2

When constructing both the pd.Series and pd.DataFrame objects in the previous sections, you likely noticed the values to the left of these objects starting at 0 and incrementing by 1 for each new row of data. The object responsible for those values is the pd.Index, highlighted in the following image:

Figure 1.1: Default pd.Index, highlighted in red

In the case of a pd.DataFrame, you have a pd.Index not only to the left of the object (often referred to as the row index or even just index) but also above (often referred to as the column index or columns):

Figure 1.2: A pd.DataFrame with a row and column index

Unless explicitly provided, pandas will create an auto-numbered pd.Index for you (technically, this is a pd.RangeIndex, a subclass of the pd.Index class). However, it is very rare to use pd.RangeIndex for your columns, as referring to a column named City or Date is more expressive than referring to a column in the n^th position. The pd.RangeIndex appears more commonly in the row index, although you may still want custom labels to appear there as well. More advanced selection operations with the default pd.RangeIndex and custom pd.Index values will be covered in Chapter 2, Selection and Assignment, to help you understand different use cases, but for now, let’s just look at how you would override the construction of the row and column pd.Index objects during pd.Series and pd.DataFrame construction.

How to do it

When constructing a pd.Series, the easiest way to change the row index is by providing a sequence of labels to the index= argument. In this example, the labels dog, cat, and human will be used instead of the default pd.RangeIndex numbered from 0 to 2:

pd.Series([4, 4, 2], index=["dog", "cat", "human"])

dog          4
cat          4
human        2
dtype: int64

If you want finer control, you may want to construct the pd.Index yourself before passing it as an argument to index=. In the following example, the pd.Index is given the name animal, and the pd.Series itself is named num_legs, providing more context to the data:

index = pd.Index(["dog", "cat", "human"], name="animal")
pd.Series([4, 4, 2], name="num_legs", index=index)

animal
dog          4
cat          4
human        2
Name: num_legs, dtype: int64

A pd.DataFrame uses a pd.Index for both dimensions. Much like with the pd.Series constructor, the index= argument can be used to specify the row labels, but you now also have the columns= argument to control the column labels:

pd.DataFrame([
    [24, 180],
    [42, 166],
], columns=["age", "height_cm"], index=["Jack", "Jill"])

         age    height_cm
Jack     24     180
Jill     42     166

Once you have a pd.Series, there are quite a few attributes you may want to inspect. The most basic attributes can tell you the type and size of your data, which is often the first thing you will inspect when reading in data from a data source.

How to do it

Let’s start by creating a pd.Series that has a name, alongside a custom pd.Index, which itself has a name. Although not all of these elements are required, having them will help us more clearly understand what the attributes we access through this recipe are actually showing us:

index = pd.Index(["dog", "cat", "human"], name="animal")
ser = pd.Series([4, 4, 2], name="num_legs", index=index)
ser

animal
dog      4
cat      4
human    2
Name: num_legs, dtype: int64

The first thing users typically want to know about their data is the type of pd.Series. This can be inspected via the pd.Series.dtype attribute:

ser.dtype

dtype('int64')

The name may be inspected via the pd.Series.name attribute. The data we constructed in this recipe was created with the name="num_legs" argument, which is what you will see when accessing this attribute (if not provided, this will return None):

ser.name

num_legs

The associated pd.Index can be accessed via pd.Series.index:

ser.index

Index(['dog', 'cat', 'human'], dtype='object', name='animal')

The name of the associated pd.Index can be accessed via pd.Series.index.name:

ser.index.name

animal

The shape can be accessed via pd.Series.shape. For a one-dimensional pd.Series, the shape is returned as a one-tuple where the first element represents the number of rows:

ser.shape

3

The size (number of elements) can be accessed via pd.Series.size:

ser.size

3

The Python built-in function len can show you the length (number of rows):

len(ser)

3

The pd.DataFrame shares many of the attributes of the pd.Series, with some slight differences. Generally, pandas tries to share as many attributes as possible between the pd.Series and pd.DataFrame, but the two-dimensional nature of the pd.DataFrame makes it more natural to express some things in plural form (for example, the .dtype attribute becomes .dtypes) and gives us a few more attributes to inspect (for example, .columns exists for a pd.DataFrame but not for a pd.Series).

How to do it

Much like we did in the previous section, we are going to construct a pd.DataFrame with a custom pd.Index in the rows, while also using custom labels in the columns. This will be more helpful when inspecting the various attributes:

index = pd.Index(["Jack", "Jill"], name="person")
df = pd.DataFrame([
    [24, 180, "red"],
    [42, 166, "blue"],
], columns=["age", "height_cm", "favorite_color"], index=index)
df

           age    height_cm    favorite_color
person
Jack       24     180          red
Jill       42     166          blue

The types of each column can be inspected via the pd.DataFrame.dtypes attribute. This attribute returns a pd.Series where each row shows the data type corresponding to each column in our pd.DataFrame:

df.dtypes

age                int64
height_cm          int64
favorite_color     object
dtype: object

The row index can be accessed via pd.DataFrame.index:

df.index

Index(['Jack', 'Jill'], dtype='object', name='person')

The column index can be accessed via pd.DataFrame.columns:

df.columns

Index(['age', 'height_cm', 'favorite_color'], dtype='object')

The shape can be accessed via pd.DataFrame.shape. For a two-dimensional pd.DataFrame, the shape is returned as a two-tuple where the first element represents the number of rows and the second element represents the number of columns:

df.shape

2     3

The size (number of elements) can be accessed via pd.DataFrame.size:

df.size

6

The Python built-in function len can show you the length (number of rows):

len(df)

2

Join our community on Discord

Join our community’s Discord space for discussions with the authors and other readers:

https://packt.link/pandas