One of the simplest and common formats for storing data is the TXT format; many IoT sensors log sensor readings with different timestamps in the simple .txt file format. Python provides built-in functions for creating, reading, and writing into TXT files.

We can access TXT files in Python itself without using any module; the data, in this case, is of the string type, and you will need to transform it to other types to use it. Alternatively, we can use NumPy or pandas.

Python has built-in functions that read and write into TXT files. The complete functionality is provided using four sets of functions: open(), read(), write(), and close(). As the names suggest, they are used to open a...

Comma-separated value (CSV) files are the most popular formats for storing tabular data generated by IoT systems. In a .csv file, the values of the records are stored in plain-text rows, with each row containing the values of the fields separated by a separator. The separator is a comma by default but can be configured to be any other character. In this section, we will learn how to use data from CSV files with Python's csv, numpy, and pandas modules. We will use the household_power_consumption data file. The file can be downloaded from the following GitHub link: https://github.com/ahanse/machlearning/blob/master/household_power_consumption.csv. To access the data files, we define the following variables:

data_folder = '../../data/household_power_consumption' 
data_file = 'household_power_consumption.csv'

Generally, to quickly read the data from...

Excel, a component of the Microsoft Office pack, is one of the popular formats in which data is stored and visualized. Since 2010, Office has supported the .xlsx format. We can read XLSX files using the OpenPyXl and pandas functions.

OpenPyXl is a Python library for reading and writing Excel files. It is an open source project. A new workbook is created using the following command:

wb = Workbook()

We can access the currently active sheet by using the following command:

ws = wb.active()

To change the sheet name, use the title command:

ws.title = "Demo Name"

A single row can be added to the sheet using the append method:

ws.append()

A new sheet can be created using the create_sheet...

JavaScript Object Notation (JSON) is another popular data format in IoT systems. In this section, we will learn how to read JSON data with Python's JSON, NumPy, and pandas packages.

For this section, we will use the zips.json file, which contains US ZIP codes with city codes, geolocation details, and state codes. The file has JSON objects recorded in the following format:

{ "_id" : "01001", "city" : "AGAWAM", "loc" : [ -72.622739, 42.070206 ], "pop" : 15338, "state" : "MA" }

To load and decode JSON data, use the json.load() or json.loads() functions. As an example, the following...

Hierarchical Data Format (HDF) is a specification put together by the HDF Group, a consortium of academic and industry organizations (https://support.hdfgroup.org/HDF5/). In HDF5 files, data is organized into groups and datasets. A group is a collection of groups or datasets. A dataset is a multidimensional homogeneous array.

In Python, PyTables and h5py are two major libraries for handling HDF5 files. Both these libraries require HDF5 to be installed. For the parallel version of HDF5, a version of MPI is also required to be installed. Installation of HDF5 and MPI is beyond the scope of this book. Installation instructions for parallel HDF5 can be found at the following link: https://support.hdfgroup.org/ftp/HDF5/current/src/unpacked/release_docs/INSTALL_parallel.

Most databases are organized using relational models. A relational database consists of one or more related tables of information, and the relationship between information in different tables is described using keys. Conventionally, these databases are managed using the Database Management System (DBMS), software which interacts with end users, different applications, and the database itself to capture and analyze data. Commercially available DBMSes use Structured Query Language (SQL) to access and manipulate databases. We can also use Python to access relational databases. In this section, we will explore SQLite and MySQL, two very popular database engines that work with Python.

According...

The Not Only Structured Query Language (NoSQL) database is not a relational database; instead, data can be stored in key-value, JSON, document, columnar, or graph formats. They are frequently used in big data and real-time applications. We will learn here how to access NoSQL data using MongoDB, and we assume you have the MongoDB server configured properly and on:

1. We will need to establish a connection with the Mongo daemon using the MongoClient object. The following code establishes the connection to the default host, localhost , and port (27017). And it gives us access to the database:

from pymongo import MongoClient
client = MongoClient()
db = client.test

2. In this example, we try to load the cancer dataset available in scikit-learn to the Mongo database. So, we first get the breast cancer dataset and convert it to a pandas DataFrame:

from sklearn.datasets import...