Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletter Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds
Practical Data Analysis
Practical Data Analysis

Practical Data Analysis: For small businesses, analyzing the information contained in their data using open source technology could be game-changing. All you need is some basic programming and mathematical skills to do just that.

eBook
$9.99 $32.99
Paperback
$54.99
Subscription
Free Trial
Renews at $19.99p/m

What do you get with eBook?

Product feature icon Instant access to your Digital eBook purchase
Product feature icon Download this book in EPUB and PDF formats
Product feature icon Access this title in our online reader with advanced features
Product feature icon DRM FREE - Read whenever, wherever and however you want
OR
Modal Close icon
Payment Processing...
tick Completed

Billing Address

Table of content icon View table of contents Preview book icon Preview Book

Practical Data Analysis

Chapter 2. Working with Data

Building real world's data analytics requires accurate data. In this chapter we discuss how to obtain, clean, normalize, and transform raw data into a standard format such as Comma-Separated Values (CSV) or JavaScript Object Notation (JSON) using OpenRefine.

In this chapter we will cover:

  • Datasource

    • Open data

    • Text files

    • Excel files

    • SQL databases

    • NoSQL databases

    • Multimedia

    • Web scraping

  • Data scrubbing

    • Statistical methods

    • Text parsing

    • Data transformation

  • Data formats

    • CSV

    • JSON

    • XML

    • YAML

  • Getting started with OpenRefine

Datasource


Datasource is a term used for all the technology related to the extraction and storage of data. A datasource can be anything from a simple text file to a big database. The raw data can come from observation logs, sensors, transactions, or user's behavior.

In this section we will take a look into the most common forms for datasource and datasets.

A dataset is a collection of data, usually presented in tabular form. Each column represents a particular variable, and each row corresponds to a given member of the data, as is shown in the following figure:

A dataset represents a physical implementation of a datasource; the common features of a dataset are as follows:

  • Dataset characteristics (such as multivariate or univariate)

  • Number of instances

  • Area (for example life, business, and so on)

  • Attribute characteristics (namely, real, categorical, and nominal)

  • Number of attributes

  • Associated tasks (such as classification or clustering)

  • Missing Values

Open data

Open data is data that can be used, re...

Data scrubbing


Data scrubbing, also called data cleansing, is the process of correcting or removing data in a dataset that is incorrect, inaccurate, incomplete, improperly formatted, or duplicated.

The result of the data analysis process not only depends on the algorithms, it also depends on the quality of the data. That's why the next step after obtaining the data, is data scrubbing. In order to avoid dirty data our dataset should possess the following characteristics:

  • Correct

  • Completeness

  • Accuracy

  • Consistency

  • Uniformity

The dirty data can be detected by applying some simple statistical data validation also by parsing the texts or deleting duplicate values. Missing or sparse data can lead you to highly misleading results.

Statistical methods

In this method we need some context about the problem (knowledge domain) to find values that are unexpected and thus erroneous, even if the data type match but the values are out of the range, it can be resolved by setting the values to an average or mean value...

Data formats


When we are working with data for human consumption the easiest way to store it is through text files. In this section, we will present parsing examples of the most common formats such as CSV, JSON, and XML. These examples will be very helpful in the next chapters.

Tip

The dataset used for these examples is a list of Pokémon characters by National Pokedex number, obtained at the URL http://bulbapedia.bulbagarden.net/.

All the scripts and dataset files can be found in the author's GitHub repository available at the URL https://github.com/hmcuesta/PDA_Book/tree/master/Chapter3/.

CSV

CSV is a very simple and common open format for table, such as data, which can be exported and imported by most of the data analysis tools. CSV is a plain text format this means that the file is a sequence of characters, with no data that has to be interpreted instead, for example, binary numbers.

There are many ways to parse a CSV file from Python, and in a moment we will discuss two of them:

The first eight...

Getting started with OpenRefine


OpenRefine (formerly known as Google Refine) is a formatting tool very useful in data cleansing, data exploration, and data transformation. It is an open source web application which runs directly in your computer, skipping the problem of uploading your delicate information to an external server.

To start working with OpenRefine just run the application and open a browser in the URL available at http://127.0.0.1:3333/.

Refer to Appendix, Setting Up the Infrastructure.

Firstly, we need to upload our data and click on Create Project. In the following screenshot, we can observe our dataset, in this case, we will use monthly sales of an alcoholic beverages company. The dataset format is an MS Excel (.xlsx) worksheet with 160 rows.

We can download the original MS Excel file and the OpenRefine project from the author's GitHub repository available at the following URL:

https://github.com/hmcuesta/PDA_Book/tree/master/Chapter2

Text facet

Text facet is a very useful tool...

Summary


In this chapter we explored the common datasources and implemented a web scraping example. Next, we introduced the basic concepts of data scrubbing such as statistical methods and text parsing. Then we learned about how to parse the most used text formats with Python. Finally, we presented an introduction to OpenRefine which is an excellent tool for data cleansing and data formatting. Working with data is not just code or clicks, we also need to play with the data and follow our intuition to get our data in great shape. We need to get involved in the knowledge domain of our data to find inconsistencies. Global vision of data helps us to discover what we need to know about our data.

In the next chapter, we will explore our data through some visualization techniques and we will present a fast introduction to D3js.

Left arrow icon Right arrow icon

Key benefits

  • Explore how to analyze your data in various innovative ways and turn them into insight
  • Learn to use the D3.js visualization tool for exploratory data analysis
  • Understand how to work with graphs and social data analysis
  • Discover how to perform advanced query techniques and run MapReduce on MongoDB

Description

Plenty of small businesses face big amounts of data but lack the internal skills to support quantitative analysis. Understanding how to harness the power of data analysis using the latest open source technology can lead them to providing better customer service, the visualization of customer needs, or even the ability to obtain fresh insights about the performance of previous products. Practical Data Analysis is a book ideal for home and small business users who want to slice and dice the data they have on hand with minimum hassle.Practical Data Analysis is a hands-on guide to understanding the nature of your data and turn it into insight. It will introduce you to the use of machine learning techniques, social networks analytics, and econometrics to help your clients get insights about the pool of data they have at hand. Performing data preparation and processing over several kinds of data such as text, images, graphs, documents, and time series will also be covered.Practical Data Analysis presents a detailed exploration of the current work in data analysis through self-contained projects. First you will explore the basics of data preparation and transformation through OpenRefine. Then you will get started with exploratory data analysis using the D3js visualization framework. You will also be introduced to some of the machine learning techniques such as, classification, regression, and clusterization through practical projects such as spam classification, predicting gold prices, and finding clusters in your Facebook friends' network. You will learn how to solve problems in text classification, simulation, time series forecast, social media, and MapReduce through detailed projects. Finally you will work with large amounts of Twitter data using MapReduce to perform a sentiment analysis implemented in Python and MongoDB. Practical Data Analysis contains a combination of carefully selected algorithms and data scrubbing that enables you to turn your data into insight.

Who is this book for?

This book is for developers, small business users, and analysts who want to implement data analysis and visualization for their company in a practical way. You need no prior experience with data analysis or data processing; however, basic knowledge of programming, statistics, and linear algebra is assumed.

What you will learn

  • Work with data to get meaningful results from your data analysis projects Visualize your data to find trends and correlations Build your own image similarity search engine Learn how to forecast numerical values from time series data Create an interactive visualization for your social media graphExplore the MapReduce framework in MongoDB Create interactive simulations with D3js

Product Details

Country selected
Publication date, Length, Edition, Language, ISBN-13
Publication date : Oct 22, 2013
Length: 360 pages
Edition : 1st
Language : English
ISBN-13 : 9781783281008
Category :
Languages :
Concepts :
Tools :

What do you get with eBook?

Product feature icon Instant access to your Digital eBook purchase
Product feature icon Download this book in EPUB and PDF formats
Product feature icon Access this title in our online reader with advanced features
Product feature icon DRM FREE - Read whenever, wherever and however you want
OR
Modal Close icon
Payment Processing...
tick Completed

Billing Address

Product Details

Publication date : Oct 22, 2013
Length: 360 pages
Edition : 1st
Language : English
ISBN-13 : 9781783281008
Category :
Languages :
Concepts :
Tools :

Packt Subscriptions

See our plans and pricing
Modal Close icon
$19.99 billed monthly
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Simple pricing, no contract
$199.99 billed annually
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Choose a DRM-free eBook or Video every month to keep
Feature tick icon PLUS own as many other DRM-free eBooks or Videos as you like for just $5 each
Feature tick icon Exclusive print discounts
$279.99 billed in 18 months
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Choose a DRM-free eBook or Video every month to keep
Feature tick icon PLUS own as many other DRM-free eBooks or Videos as you like for just $5 each
Feature tick icon Exclusive print discounts

Frequently bought together


Stars icon
Total $ 170.97
Building Machine Learning Systems with Python
$54.99
Machine Learning with R
$60.99
Practical Data Analysis
$54.99
Total $ 170.97 Stars icon
Banner background image

Table of Contents

14 Chapters
Getting Started Chevron down icon Chevron up icon
Working with Data Chevron down icon Chevron up icon
Data Visualization Chevron down icon Chevron up icon
Text Classification Chevron down icon Chevron up icon
Similarity-based Image Retrieval Chevron down icon Chevron up icon
Simulation of Stock Prices Chevron down icon Chevron up icon
Predicting Gold Prices Chevron down icon Chevron up icon
Working with Support Vector Machines Chevron down icon Chevron up icon
Modeling Infectious Disease with Cellular Automata Chevron down icon Chevron up icon
Working with Social Graphs Chevron down icon Chevron up icon
Sentiment Analysis of Twitter Data Chevron down icon Chevron up icon
Data Processing and Aggregation with MongoDB Chevron down icon Chevron up icon
Working with MapReduce Chevron down icon Chevron up icon
Online Data Analysis with IPython and Wakari Chevron down icon Chevron up icon

Customer reviews

Top Reviews
Rating distribution
Full star icon Full star icon Full star icon Half star icon Empty star icon 3.6
(7 Ratings)
5 star 28.6%
4 star 42.9%
3 star 0%
2 star 14.3%
1 star 14.3%
Filter icon Filter
Top Reviews

Filter reviews by




Carlos Rodriguez Contreras Feb 19, 2014
Full star icon Full star icon Full star icon Full star icon Full star icon 5
This a very useful text for all people trying to get into Big Data Analysis. Concepts are clearly explained and readers do not need to be experts in any topic covered, this is why I chose the Cuesta's book over a lot of books on Big Data that apparently try to show mainly the expertise of authors. If you, like me, are interested in Big Data, this is a must on your shelf.
Amazon Verified review Amazon
José Carlos Dec 07, 2013
Full star icon Full star icon Full star icon Full star icon Full star icon 5
This book is not about theories of data analysis, is about how move your hacking skills into the data analysis world.If you are a programmer/hacker who want to understanding a problem from a data-oriented perspective, this book isfor you.This book is a fast introduction to data analysis methods including some of the most used techniques forclassification, regression and clustering. The book provides a wide range of tools like Python, mlpy, Pandas, D3jsand MongoDB. The recipes are clear and easy to follow you can get into data analysis in fast way if you alreadyhave some programming skills.I can highly recommend chapters 10 and 11 which focus on Social Networks Analytics and Social NetworksGraph’s Visualization.
Amazon Verified review Amazon
Mark Kerzner Nov 27, 2013
Full star icon Full star icon Full star icon Full star icon Empty star icon 4
This is a very practical book, which teaches you how to "make data talk to you," that is, how to extract information, quantitative and qualitative, out of your data, and make it useful beyond just numbers.Following the by now ubiquitous quote by Hal Varian of Google that "the sexy job in the next ten years will be statisticians" [...] the book teaches not the theory and not the programming languages, but methods and operations on the data.Programming languages do come in (Python with its mathematical and word analysis packages), but only as tools for the practical applications. So, if you are not looking for the theoretical mathematical proofs or for computers science implementation details but are rather interested in the answers that the data can provide, you have come to the right place. Here are some of the the areas that the books covers:Data formats and visualizationText classificationFinding similar imagesSimulation of stock price and predicting the prices of goldMachine learningModeling infectious diseasesWorking with social graphsSentiment analysis of Twitter dataThe reader will do well to go deeper and to read the description of the algorithms mentioned in the books. As mentioned, the books is practical in that it explains the benefits of the analysis but not the analysis itself. However, it gives you a good list of areas you need to go deeper into, and sets you on the right track with that. Later, you will be able to use it as handbook and a cheat sheet.
Amazon Verified review Amazon
View2 Nov 24, 2013
Full star icon Full star icon Full star icon Full star icon Empty star icon 4
This books gives a very practical introduction to data analysis. It covers a wide range of topics, including data visualization, text analysis (spam recognition, sentiment analysis), image analysis, social graph analysis, Bayes classification, SVM, etc. The examples are very practical, and teaches the user how to use popular languages and libraries like d3.js, python3, nltk, mlpy etc. to do basic data analysis.The book is a great read for beginners. To read and fully appreciate it, no data analysis is required. The books provides an introductory to the very basic techniques. Some basic understanding of python and javascript would be necessary, though.What I like of this book is its hand-on style: while reading, you can easily get started with your first data analyses. The examples are very simple, the code easy to read, and a very detailed appendix helps to install the tools used. This book is a great help to learn data analysis by doing.What may be improved is precision. I found some grammar mistakes. Not so big a problem, but not perfect, either. For instance reading sentences like "we will use Pillow due to its compatibility with Python 3.2 and can be downloaded ..." [p. 97] does hurt a little. More problematic is the section "Classifier accuracy" [p. 90]. It simply uses the ratio of correctly predicted emails to be a measure of accuracy, although actually every discussion of classification accuracy must contain the rations of false positives and false negatives as well.Overall, this book is a very practical introduction to data analysis for beginners.
Amazon Verified review Amazon
R. Friesel Jr. Dec 09, 2013
Full star icon Full star icon Full star icon Full star icon Empty star icon 4
I just finished up reading "Practical Data Analysis" by Hector Cuesta (Packt Publishing, 2013) and overall, it was a pretty good overview and recommends some good tools. I would say that the book is a good place for someone to get started if they have no real experience performing these kinds of analyses, and though Cuesta doesn't go deep into the math behind it all, he isn't afraid to use the technical names for different formulae, which should make it easy for you to do your own follow-up research.Jeff Leek's Data Analysis on Coursera provides the lens through which I read this book. That being said, I found myself doing a lot of comparing and contrasting between the two. For example, they both use practical, reasonably small "real world" sample problems to highlight specific analytical techniques and/or features of their chosen toolkits. However, whereas Leek's course focused exclusively on using R, Cuesta assembles his own all-star team of tools using Python and D3.js. Perhaps it goes without saying, but there are pros and cons to each approach (e.g., Leek's "pure R" vs. Cuesta's "Python plus D3.js"), and I felt that it was best to consider them together.Cuesta's approach with this book is to present a sample scenario in each chapter that introduces a class of problem, a solution to that problem, and his recommended toolkit. For example, chapter six creates a stock price simulation, introducing simple simulation problems (especially for apparently stochastic data), time series data and Monte Carlo methods, and then how to simulate the data using Python and visualizing it in D3.js. Although the book is not strictly a "cookbook", the chapters very much feel like macro-level "recipes". There's quite a bit of code and some decent discussion around the concepts that govern the analytical model, and (true to the "practical" in the title) the emphasis is on the "how" and not the "why".While I did not read the entire book cover-to-cover, I would definitely recommend it to anyone that wants an introduction to some basic data analysis techniques and tools. You'll get more out of this book if you have some base to compare it to -- e.g., some experience in R (academic or otherwise); and you'll get the most out of this book if you also have a solid foundation in the mathematics and/or statistics that underlie these analytical approaches.DISCLOSURE: I was given an electronic copy of this book from the publisher in exchange for writing a review.
Amazon Verified review Amazon
Get free access to Packt library with over 7500+ books and video courses for 7 days!
Start Free Trial

FAQs

How do I buy and download an eBook? Chevron down icon Chevron up icon

Where there is an eBook version of a title available, you can buy it from the book details for that title. Add either the standalone eBook or the eBook and print book bundle to your shopping cart. Your eBook will show in your cart as a product on its own. After completing checkout and payment in the normal way, you will receive your receipt on the screen containing a link to a personalised PDF download file. This link will remain active for 30 days. You can download backup copies of the file by logging in to your account at any time.

If you already have Adobe reader installed, then clicking on the link will download and open the PDF file directly. If you don't, then save the PDF file on your machine and download the Reader to view it.

Please Note: Packt eBooks are non-returnable and non-refundable.

Packt eBook and Licensing When you buy an eBook from Packt Publishing, completing your purchase means you accept the terms of our licence agreement. Please read the full text of the agreement. In it we have tried to balance the need for the ebook to be usable for you the reader with our needs to protect the rights of us as Publishers and of our authors. In summary, the agreement says:

  • You may make copies of your eBook for your own use onto any machine
  • You may not pass copies of the eBook on to anyone else
How can I make a purchase on your website? Chevron down icon Chevron up icon

If you want to purchase a video course, eBook or Bundle (Print+eBook) please follow below steps:

  1. Register on our website using your email address and the password.
  2. Search for the title by name or ISBN using the search option.
  3. Select the title you want to purchase.
  4. Choose the format you wish to purchase the title in; if you order the Print Book, you get a free eBook copy of the same title. 
  5. Proceed with the checkout process (payment to be made using Credit Card, Debit Cart, or PayPal)
Where can I access support around an eBook? Chevron down icon Chevron up icon
  • If you experience a problem with using or installing Adobe Reader, the contact Adobe directly.
  • To view the errata for the book, see www.packtpub.com/support and view the pages for the title you have.
  • To view your account details or to download a new copy of the book go to www.packtpub.com/account
  • To contact us directly if a problem is not resolved, use www.packtpub.com/contact-us
What eBook formats do Packt support? Chevron down icon Chevron up icon

Our eBooks are currently available in a variety of formats such as PDF and ePubs. In the future, this may well change with trends and development in technology, but please note that our PDFs are not Adobe eBook Reader format, which has greater restrictions on security.

You will need to use Adobe Reader v9 or later in order to read Packt's PDF eBooks.

What are the benefits of eBooks? Chevron down icon Chevron up icon
  • You can get the information you need immediately
  • You can easily take them with you on a laptop
  • You can download them an unlimited number of times
  • You can print them out
  • They are copy-paste enabled
  • They are searchable
  • There is no password protection
  • They are lower price than print
  • They save resources and space
What is an eBook? Chevron down icon Chevron up icon

Packt eBooks are a complete electronic version of the print edition, available in PDF and ePub formats. Every piece of content down to the page numbering is the same. Because we save the costs of printing and shipping the book to you, we are able to offer eBooks at a lower cost than print editions.

When you have purchased an eBook, simply login to your account and click on the link in Your Download Area. We recommend you saving the file to your hard drive before opening it.

For optimal viewing of our eBooks, we recommend you download and install the free Adobe Reader version 9.