Search icon CANCEL
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Conferences
Free Learning
Arrow right icon
Hands-On Data Analysis with Pandas
Hands-On Data Analysis with Pandas

Hands-On Data Analysis with Pandas: Efficiently perform data collection, wrangling, analysis, and visualization using Python

eBook
€29.99
Paperback
€36.99
Subscription
Free Trial
Renews at €18.99p/m

What do you get with Print?

Product feature icon Instant access to your digital eBook copy whilst your Print order is Shipped
Product feature icon Paperback book shipped to your preferred address
Product feature icon Download this book in EPUB and PDF formats
Product feature icon Access this title in our online reader with advanced features
Product feature icon DRM FREE - Read whenever, wherever and however you want
Product feature icon AI Assistant (beta) to help accelerate your learning
Table of content icon View table of contents Preview book icon Preview Book

Hands-On Data Analysis with Pandas

Introduction to Data Analysis

Before we can begin our hands-on introduction to data analysis with pandas, we need to learn about the fundamentals of data analysis. Those who have ever looked at the documentation for a software library know how overwhelming it can be if you have no clue what you are looking for. Therefore, it is essential that we not only master the coding aspect, but also the thought process and workflow required to analyze data, which will prove the most useful in augmenting our skill set in the future.

Much like the scientific method, data science has some common workflows that we can follow when we want to conduct an analysis and present the results. The backbone of this process is statistics, which gives us ways to describe our data, make predictions, and also draw conclusions about it. Since prior knowledge of statistics is not a prerequisite, this chapter...

Chapter materials

All the files for this book are on GitHub at https://github.com/stefmolin/Hands-On-Data-Analysis-with-Pandas. While having a GitHub account isn't necessary to work through this book, it is a good idea to create one, as it will serve as a portfolio for any data/coding projects. In addition, working with Git will provide a version control system and make collaboration easy.

In order to get a local copy of the files, we have a few options (ordered from least useful to most useful):

  • Download the ZIP file and extract the files locally
  • Clone the repository without forking it
  • Fork the repository and then clone it

This book includes exercises for every chapter; therefore, for those who want to keep a copy of their solutions...

Fundamentals of data analysis

Data analysis is a highly iterative process involving collection, preparation (wrangling), exploratory data analysis (EDA), and drawing conclusions. During an analysis, we will frequently revisit each of these steps. The following diagram depicts a generalized workflow:

In practice, this process is heavily skewed towards the data preparation side. Surveys have found that, although data scientists enjoy the data preparation side of their job the least, it makes up 80% of their work (https://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consuming-least-enjoyable-data-science-task-survey-says/#419ce7b36f63). This data preparation step is where pandas really shines.

Data collection

...

Statistical foundations

When we want to make observations about the data we are analyzing, we are often, if not always, turning to statistics in some fashion. The data we have is referred to as the sample, which was observed from (and is a subset of) the population. Two broad categories of statistics are descriptive and inferential statistics. With descriptive statistics, as the name implies, we are looking to describe the sample. Inferential statistics involves using the sample statistics to infer, or deduce, something about the population, such as the underlying distribution.

The sample statistics are used as estimators of the population parameters, meaning that we have to quantify their bias and variance. There are a multitude of methods for this; some will make assumptions on the shape of the distribution (parametric) and others won't (non-parametric). This is all well...

Setting up a virtual environment

This book was written using Python 3.6.4, but the code should work for Python 3.6+, which is available on all major operating systems. In this section, we will go over how to set up the virtual environment in order to follow along with this book. If Python isn't already installed on your computer, read through the following sections on virtual environments first, and then decide whether to install Anaconda, since it will also install Python. To install Python without Anaconda, download it here: https://www.python.org/downloads/. Then, continue with the section on venv.

To check if Python is already installed, run where python3 from the command line on Windows or which python3 from the command line on Linux/macOS. If this returns nothing, try running it with just python (instead of python3). If Python is installed, check the version by running...

Summary

In this chapter, we learned about the main processes in conducting data analysis: data collection, wrangling, EDA, and drawing conclusions. We followed that up with an overview of descriptive statistics and learned how to describe the central tendency and spread of our data; how to summarize it both numerically and visually using the 5-number summary, box plots, histograms, and kernel density estimates; how to scale our data; and how to quantify relationships between variables in our dataset.

We got an introduction to prediction and time series analysis. Then, we had a very brief overview of some core topics in inferential statistics that can be explored after mastering the contents of this book. Note that, while all the examples in this chapter were of one or two variables, real-life data is often high-dimensional. Chapter 10, Making Better Predictions – Optimizing...

Exercises

Run through the introduction_to_data_analysis.ipynb notebook for a review of this chapter's content, and then complete the following exercises to practice working with JupyterLab and calculating summary statistics in Python:

  1. Explore the JupyterLab interface and look at some of the shortcuts that are available. Don't worry about memorizing them for now (eventually, they will become second nature and save you a lot of time)—just get comfortable using Jupyter Notebooks.
  2. Is all data normally distributed? Explain why or why not.
  1. When would it make more sense to use the median instead of the mean for the measure of center?
  2. Run the code in the first cell of the exercises.ipynb notebook. It will give you a list of 100 values to work with for the rest of the exercises in this chapter.
  3. Using the data from exercise #4, calculate the following statistics without...

Further reading

The following are some resources that you can use to become more familiar with Jupyter:

The following resource shows you how to use conda to manage virtual environments instead of the venv solution that was explained earlier in this chapter:

Left arrow icon Right arrow icon

Key benefits

  • Perform efficient data analysis and manipulation tasks using pandas
  • Apply pandas to different real-world domains with the help of step-by-step demonstrations
  • Get accustomed to using pandas as an effective data exploration tool.

Description

Data analysis has become a necessary skill in a variety of domains where knowing how to work with data and extract insights can generate significant value. Hands-On Data Analysis with Pandas will show you how to analyze your data, get started with machine learning, and work effectively with Python libraries often used for data science, such as pandas, NumPy, matplotlib, seaborn, and scikit-learn. Using real-world datasets, you will learn how to use the powerful pandas library to perform data wrangling to reshape, clean, and aggregate your data. Then, you will be able to conduct exploratory data analysis by calculating summary statistics and visualizing the data to find patterns. In the concluding chapters, you will explore some applications of anomaly detection, regression, clustering, and classification using scikit-learn to make predictions based on past data. By the end of this book, you will be equipped with the skills you need to use pandas to ensure the veracity of your data, visualize it for effective decision-making, and reliably reproduce analyses across multiple datasets.

Who is this book for?

This book is for data analysts, data science beginners, and Python developers who want to explore each stage of data analysis and scientific computing using a wide range of datasets. You will also find this book useful if you are a data scientist looking to implement pandas in machine learning. Working knowledge of Python programming language will be beneficial.

What you will learn

  • Understand how data analysts and scientists gather and analyze data
  • Perform data analysis and data wrangling using Python
  • Combine, group, and aggregate data from multiple sources
  • Create data visualizations with pandas, matplotlib, and seaborn
  • Apply machine learning (ML) algorithms to identify patterns and make predictions
  • Use Python data science libraries to analyze real-world datasets
  • Use pandas to solve common data representation and analysis problems
  • Build Python scripts, modules, and packages for reusable analysis code
Estimated delivery fee Deliver to Austria

Premium delivery 7 - 10 business days

€17.95
(Includes tracking information)

Product Details

Country selected
Publication date, Length, Edition, Language, ISBN-13
Publication date : Jul 26, 2019
Length: 740 pages
Edition : 1st
Language : English
ISBN-13 : 9781789615326
Category :
Languages :
Concepts :
Tools :

What do you get with Print?

Product feature icon Instant access to your digital eBook copy whilst your Print order is Shipped
Product feature icon Paperback book shipped to your preferred address
Product feature icon Download this book in EPUB and PDF formats
Product feature icon Access this title in our online reader with advanced features
Product feature icon DRM FREE - Read whenever, wherever and however you want
Product feature icon AI Assistant (beta) to help accelerate your learning
Estimated delivery fee Deliver to Austria

Premium delivery 7 - 10 business days

€17.95
(Includes tracking information)

Product Details

Publication date : Jul 26, 2019
Length: 740 pages
Edition : 1st
Language : English
ISBN-13 : 9781789615326
Category :
Languages :
Concepts :
Tools :

Packt Subscriptions

See our plans and pricing
Modal Close icon
€18.99 billed monthly
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Simple pricing, no contract
€189.99 billed annually
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Choose a DRM-free eBook or Video every month to keep
Feature tick icon PLUS own as many other DRM-free eBooks or Videos as you like for just €5 each
Feature tick icon Exclusive print discounts
€264.99 billed in 18 months
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Choose a DRM-free eBook or Video every month to keep
Feature tick icon PLUS own as many other DRM-free eBooks or Videos as you like for just €5 each
Feature tick icon Exclusive print discounts

Frequently bought together


Stars icon
Total 115.97
Mastering pandas
€36.99
Python Machine Learning
€41.99
Hands-On Data Analysis with Pandas
€36.99
Total 115.97 Stars icon

Table of Contents

19 Chapters
Section 1: Getting Started with Pandas Chevron down icon Chevron up icon
Introduction to Data Analysis Chevron down icon Chevron up icon
Working with Pandas DataFrames Chevron down icon Chevron up icon
Section 2: Using Pandas for Data Analysis Chevron down icon Chevron up icon
Data Wrangling with Pandas Chevron down icon Chevron up icon
Aggregating Pandas DataFrames Chevron down icon Chevron up icon
Visualizing Data with Pandas and Matplotlib Chevron down icon Chevron up icon
Plotting with Seaborn and Customization Techniques Chevron down icon Chevron up icon
Section 3: Applications - Real-World Analyses Using Pandas Chevron down icon Chevron up icon
Financial Analysis - Bitcoin and the Stock Market Chevron down icon Chevron up icon
Rule-Based Anomaly Detection Chevron down icon Chevron up icon
Section 4: Introduction to Machine Learning with Scikit-Learn Chevron down icon Chevron up icon
Getting Started with Machine Learning in Python Chevron down icon Chevron up icon
Making Better Predictions - Optimizing Models Chevron down icon Chevron up icon
Machine Learning Anomaly Detection Chevron down icon Chevron up icon
Section 5: Additional Resources Chevron down icon Chevron up icon
The Road Ahead Chevron down icon Chevron up icon
Solutions Chevron down icon Chevron up icon
Other Books You May Enjoy Chevron down icon Chevron up icon

Customer reviews

Top Reviews
Rating distribution
Full star icon Full star icon Full star icon Full star icon Half star icon 4.7
(11 Ratings)
5 star 90.9%
4 star 0%
3 star 0%
2 star 9.1%
1 star 0%
Filter icon Filter
Top Reviews

Filter reviews by




Ali Assad Apr 06, 2021
Full star icon Full star icon Full star icon Full star icon Full star icon 5
Excellent book! If you are looking to sharpen up on pandas and data analysis in python in general (not only pandas) this is a very solid. In my case I needed a strong refresher and this book was a great choice going beyond that. The accompanying notebooks are great too.
Amazon Verified review Amazon
Felipe M. Sep 21, 2019
Full star icon Full star icon Full star icon Full star icon Full star icon 5
If you are new to Pandas and looking for a way to get up to speed fast this is a great book. Lots of good examples to see how to apply the concepts in real life, which helps a lot to understand how the different dataframe methods work. You can see the author payed a lot of attention to the detail, from the way to breakdown the chapters to the choice of exercises at the end of each chapter.As a Python programmer Learning Pandas I also appreciate a lot the quality of the code. Usually the code you find in data science examples get things done but it’s far from being of good quality. It’s very refreshing to see good examples written in a very pythonic way.I keep this book next to my desk and I have the feeling I will be coming back to use it as a reference a lot in the coming weeks. The only way this could be improved would be to have the author sit next to you every time you need to use Pandas :)
Amazon Verified review Amazon
Amazon Customer Nov 16, 2021
Full star icon Full star icon Full star icon Full star icon Full star icon 5
Having just started on my Pandas journey and coming from a non technical background, I find this book simple and the exercises relatable .
Amazon Verified review Amazon
CryptoKnight Aug 30, 2019
Full star icon Full star icon Full star icon Full star icon Full star icon 5
I really enjoyed this book. This book is great for python programmers who want to learn/implement something cool with data using Pandas (of-course). With some programming background in python/R/C, this book is perfect for the data scientist wannabe. Every chapter has a very interesting exercise set. Easy to follow. I am glad that the author has provided the github repo for this too, way better than reading code from the book alone! The book starts from easy steps e.g., import data, transform, basic computations, more fancy topics like plotting, to more advanced and application-specific like finance, machine learning (yes the book also talked about this in practical aspect!, using sklearn), security analytics!I can't say enough good things about this book and about how effective it is. I started 10 days ago with a skill of python but no data science background. Now, nearly two weeks later, I'm making my scripts do all kinds of things that are complicated if you don't know Pandas e.g., aggregate statistics based on duration (day, week, ..). I genuinely found this book and the exercises a truly beneficial experience/resource. Honestly I usually don't trust this publisher. Most Packt books I own I think weren't worth their price. This book is surprisingly good quality. Finally they have a good author.
Amazon Verified review Amazon
Adam Mar 23, 2021
Full star icon Full star icon Full star icon Full star icon Full star icon 5
Amazing read. Very thorough, covering a broad range of topics in the data science space. I would highly recommend starting with this book for anyone interested in building a foundation in Data Science.
Amazon Verified review Amazon
Get free access to Packt library with over 7500+ books and video courses for 7 days!
Start Free Trial

FAQs

What is the delivery time and cost of print book? Chevron down icon Chevron up icon

Shipping Details

USA:

'

Economy: Delivery to most addresses in the US within 10-15 business days

Premium: Trackable Delivery to most addresses in the US within 3-8 business days

UK:

Economy: Delivery to most addresses in the U.K. within 7-9 business days.
Shipments are not trackable

Premium: Trackable delivery to most addresses in the U.K. within 3-4 business days!
Add one extra business day for deliveries to Northern Ireland and Scottish Highlands and islands

EU:

Premium: Trackable delivery to most EU destinations within 4-9 business days.

Australia:

Economy: Can deliver to P. O. Boxes and private residences.
Trackable service with delivery to addresses in Australia only.
Delivery time ranges from 7-9 business days for VIC and 8-10 business days for Interstate metro
Delivery time is up to 15 business days for remote areas of WA, NT & QLD.

Premium: Delivery to addresses in Australia only
Trackable delivery to most P. O. Boxes and private residences in Australia within 4-5 days based on the distance to a destination following dispatch.

India:

Premium: Delivery to most Indian addresses within 5-6 business days

Rest of the World:

Premium: Countries in the American continent: Trackable delivery to most countries within 4-7 business days

Asia:

Premium: Delivery to most Asian addresses within 5-9 business days

Disclaimer:
All orders received before 5 PM U.K time would start printing from the next business day. So the estimated delivery times start from the next day as well. Orders received after 5 PM U.K time (in our internal systems) on a business day or anytime on the weekend will begin printing the second to next business day. For example, an order placed at 11 AM today will begin printing tomorrow, whereas an order placed at 9 PM tonight will begin printing the day after tomorrow.


Unfortunately, due to several restrictions, we are unable to ship to the following countries:

  1. Afghanistan
  2. American Samoa
  3. Belarus
  4. Brunei Darussalam
  5. Central African Republic
  6. The Democratic Republic of Congo
  7. Eritrea
  8. Guinea-bissau
  9. Iran
  10. Lebanon
  11. Libiya Arab Jamahriya
  12. Somalia
  13. Sudan
  14. Russian Federation
  15. Syrian Arab Republic
  16. Ukraine
  17. Venezuela
What is custom duty/charge? Chevron down icon Chevron up icon

Customs duty are charges levied on goods when they cross international borders. It is a tax that is imposed on imported goods. These duties are charged by special authorities and bodies created by local governments and are meant to protect local industries, economies, and businesses.

Do I have to pay customs charges for the print book order? Chevron down icon Chevron up icon

The orders shipped to the countries that are listed under EU27 will not bear custom charges. They are paid by Packt as part of the order.

List of EU27 countries: www.gov.uk/eu-eea:

A custom duty or localized taxes may be applicable on the shipment and would be charged by the recipient country outside of the EU27 which should be paid by the customer and these duties are not included in the shipping charges been charged on the order.

How do I know my custom duty charges? Chevron down icon Chevron up icon

The amount of duty payable varies greatly depending on the imported goods, the country of origin and several other factors like the total invoice amount or dimensions like weight, and other such criteria applicable in your country.

For example:

  • If you live in Mexico, and the declared value of your ordered items is over $ 50, for you to receive a package, you will have to pay additional import tax of 19% which will be $ 9.50 to the courier service.
  • Whereas if you live in Turkey, and the declared value of your ordered items is over € 22, for you to receive a package, you will have to pay additional import tax of 18% which will be € 3.96 to the courier service.
How can I cancel my order? Chevron down icon Chevron up icon

Cancellation Policy for Published Printed Books:

You can cancel any order within 1 hour of placing the order. Simply contact customercare@packt.com with your order details or payment transaction id. If your order has already started the shipment process, we will do our best to stop it. However, if it is already on the way to you then when you receive it, you can contact us at customercare@packt.com using the returns and refund process.

Please understand that Packt Publishing cannot provide refunds or cancel any order except for the cases described in our Return Policy (i.e. Packt Publishing agrees to replace your printed book because it arrives damaged or material defect in book), Packt Publishing will not accept returns.

What is your returns and refunds policy? Chevron down icon Chevron up icon

Return Policy:

We want you to be happy with your purchase from Packtpub.com. We will not hassle you with returning print books to us. If the print book you receive from us is incorrect, damaged, doesn't work or is unacceptably late, please contact Customer Relations Team on customercare@packt.com with the order number and issue details as explained below:

  1. If you ordered (eBook, Video or Print Book) incorrectly or accidentally, please contact Customer Relations Team on customercare@packt.com within one hour of placing the order and we will replace/refund you the item cost.
  2. Sadly, if your eBook or Video file is faulty or a fault occurs during the eBook or Video being made available to you, i.e. during download then you should contact Customer Relations Team within 14 days of purchase on customercare@packt.com who will be able to resolve this issue for you.
  3. You will have a choice of replacement or refund of the problem items.(damaged, defective or incorrect)
  4. Once Customer Care Team confirms that you will be refunded, you should receive the refund within 10 to 12 working days.
  5. If you are only requesting a refund of one book from a multiple order, then we will refund you the appropriate single item.
  6. Where the items were shipped under a free shipping offer, there will be no shipping costs to refund.

On the off chance your printed book arrives damaged, with book material defect, contact our Customer Relation Team on customercare@packt.com within 14 days of receipt of the book with appropriate evidence of damage and we will work with you to secure a replacement copy, if necessary. Please note that each printed book you order from us is individually made by Packt's professional book-printing partner which is on a print-on-demand basis.

What tax is charged? Chevron down icon Chevron up icon

Currently, no tax is charged on the purchase of any print book (subject to change based on the laws and regulations). A localized VAT fee is charged only to our European and UK customers on eBooks, Video and subscriptions that they buy. GST is charged to Indian customers for eBooks and video purchases.

What payment methods can I use? Chevron down icon Chevron up icon

You can pay with the following card types:

  1. Visa Debit
  2. Visa Credit
  3. MasterCard
  4. PayPal
What is the delivery time and cost of print books? Chevron down icon Chevron up icon

Shipping Details

USA:

'

Economy: Delivery to most addresses in the US within 10-15 business days

Premium: Trackable Delivery to most addresses in the US within 3-8 business days

UK:

Economy: Delivery to most addresses in the U.K. within 7-9 business days.
Shipments are not trackable

Premium: Trackable delivery to most addresses in the U.K. within 3-4 business days!
Add one extra business day for deliveries to Northern Ireland and Scottish Highlands and islands

EU:

Premium: Trackable delivery to most EU destinations within 4-9 business days.

Australia:

Economy: Can deliver to P. O. Boxes and private residences.
Trackable service with delivery to addresses in Australia only.
Delivery time ranges from 7-9 business days for VIC and 8-10 business days for Interstate metro
Delivery time is up to 15 business days for remote areas of WA, NT & QLD.

Premium: Delivery to addresses in Australia only
Trackable delivery to most P. O. Boxes and private residences in Australia within 4-5 days based on the distance to a destination following dispatch.

India:

Premium: Delivery to most Indian addresses within 5-6 business days

Rest of the World:

Premium: Countries in the American continent: Trackable delivery to most countries within 4-7 business days

Asia:

Premium: Delivery to most Asian addresses within 5-9 business days

Disclaimer:
All orders received before 5 PM U.K time would start printing from the next business day. So the estimated delivery times start from the next day as well. Orders received after 5 PM U.K time (in our internal systems) on a business day or anytime on the weekend will begin printing the second to next business day. For example, an order placed at 11 AM today will begin printing tomorrow, whereas an order placed at 9 PM tonight will begin printing the day after tomorrow.


Unfortunately, due to several restrictions, we are unable to ship to the following countries:

  1. Afghanistan
  2. American Samoa
  3. Belarus
  4. Brunei Darussalam
  5. Central African Republic
  6. The Democratic Republic of Congo
  7. Eritrea
  8. Guinea-bissau
  9. Iran
  10. Lebanon
  11. Libiya Arab Jamahriya
  12. Somalia
  13. Sudan
  14. Russian Federation
  15. Syrian Arab Republic
  16. Ukraine
  17. Venezuela