Introduction

Statistics refers to the mathematics and techniques with which we understand data. It is a vast field which plays a key role in the areas of data mining and artificial intelligence, intersecting with the areas of engineering and other disciplines. Statistics helps in describing data, that is, descriptive statistics reveals the distribution of the data for each variable. Also, statistics is widely used for the purpose of prediction.

In this chapter, we'll see how to apply various statistical measures and functions on large datasets using Spark.

Working with Pandas

Pandas is an open source Python library for highly specialized data analysis. It is the reference point that all professionals using the Python language need to study and analyze data sets for statistical purposes of analysis and decision-making. Pandas arises from the need to have a specific library for the analysis of the data which provides tools for data processing , data extraction and data manipulation...