Search icon CANCEL
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Conferences
Free Learning
Arrow right icon
Learning Apache Spark 2
Learning Apache Spark 2

Learning Apache Spark 2: A beginner's guide to real-time Big Data processing using the Apache Spark framework

eBook
Mex$561.99 Mex$803.99
Paperback
Mex$1004.99
Subscription
Free Trial

What do you get with a Packt Subscription?

Free for first 7 days. $19.99 p/m after that. Cancel any time!
Product feature icon Unlimited ad-free access to the largest independent learning library in tech. Access this title and thousands more!
Product feature icon 50+ new titles added per month, including many first-to-market concepts and exclusive early access to books as they are being written.
Product feature icon Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.
Product feature icon Thousands of reference materials covering every tech concept you need to stay up to date.
Subscribe now
View plans & pricing
Table of content icon View table of contents Preview book icon Preview Book

Learning Apache Spark 2

Chapter 2. Transformations and Actions with Spark RDDs

Now that we have had a basic overview of the architecture of Spark and key software components, we will cover Spark RDD's in this chapter. During the course of this chapter, we'll walk through the following topics:

  • How to construct RDDs
  • Operations on RDDs, such as transformations and actions
  • Passing functions to Spark (Scala, Java, and Python)
  • Transformations such as map, filter, flatMap, and sample
  • Set operations such as distinct, intersection, and union
  • Actions such as reduce, collect, count, take, and first
  • PairRDDs
  • Shared and broadcast variables

Let's get cracking!

What is an RDD?

What's in a name might be true for a rose, but perhaps not for Resilient Distributed Datasets (RDD) which, in essence, describes what an RDD is.

They are basically datasets, which are distributed across a cluster (remember the Spark framework is inherently based on an MPP architecture), and provide resilience (automatic failover) by nature.

Before we go into any further detail, let's try to understand this a little bit, and again we are trying to be as abstract as possible. Let us assume that you have a sensor data from aircraft sensors and you want to analyze the data irrespective of its size and locality. For example, an Airbus A350 has roughly 6000 sensors across the entire plane and generates 2.5 TB data per day, while the newer model expected to launch in 2020 will generate roughly 7.5 TB per day. From a data engineering point of view, it might be important to understand the data pipeline, but from an analyst and a data scientist point of view, the major concern...

Operations on RDD

Two major operation types can be performed on an RDD. They are called:

  • Transformations
  • Actions

Transformations

Transformations are operations that create a new dataset, as RDDs are immutable. They are used to transform data from one to another, which could result in amplification of the data, reduction of the data, or a totally different shape altogether. These operations do not return any value back to the driver program, and hence are lazily evaluated, which is one of the main benefits of Spark.

An example of a transformation would be a map function that will pass through each element of the RDD and return a totally new RDD representing the results of application of the function on the original dataset.

Actions

Actions are operations that return a value to the driver program. As previously discussed, all transformations in Spark are lazy, which essentially means that Spark remembers all the transformations carried out on an RDD, and applies them in the most optimal fashion...

Passing functions to Spark (Scala)

As you have seen in the previous example, passing functions is a critical functionality provided by Spark. From a user's point of view you would pass the function in your driver program, and Spark would figure out the location of the data partitions across the cluster memory, running it in parallel. The exact syntax of passing functions differs by the programming language. Since Spark has been written in Scala, we'll discuss Scala first.

In Scala, the recommended ways to pass functions to the Spark framework are as follows:

  • Anonymous functions
  • Static singleton methods

Anonymous functions

Anonymous functions are used for short pieces of code. They are also referred to as lambda expressions, and are a cool and elegant feature of the programming language. The reason they are called anonymous functions is because you can give any name to the input argument and the result would be the same.

For example, the following code examples would produce the same...

Passing functions to Spark (Java)

In Java, to create a function you will have to implement the interfaces available in the org.apache.spark.api.java function package. There are two popular ways to create such functions:

  • Implement the interface in your own class, and pass the instance to Spark.
  • Starting Java 8, you can use Lambda expressions to pass off the functions to the Spark framework.

Let's implement the preceding word count examples in Java:

Passing functions to Spark (Java)

Figure 2.13: Code example of Java implementation of word count (inline functions)

If you belong to a group of programmers who feel that writing inline functions makes the code complex and unreadable (a lot of people do agree to that assertion), you may want to create separate functions and call them as follows:

Passing functions to Spark (Java)

Figure 2.14: Code example of Java implementation of word count

Passing functions to Spark (Python)

Python provides a simple way to pass functions to Spark. The Spark programming guide available at spark.apache.org suggests there are three recommended ways to do this:

  • Lambda expressions is the ideal way for short functions that can be written inside a single expression
  • Local defs inside the function calling into Spark for longer code
  • Top-level functions in a module

While we have already looked at the lambda functions in some of the previous examples, let's look at local definitions of the functions. We can encapsulate our business logic which is splitting of words, and counting into two separate functions as shown below.

def splitter(lineOfText): 
     words = lineOfText.split(" ") 
     return len(words) 
def aggregate(numWordsLine1, numWordsLineNext): 
     totalWords = numWordsLine1 + numWordsLineNext 
     return totalWords 

Let's see the working code example:

Passing functions to Spark (Python)

Figure 2.15: Code example of Python word count (local definition of...

What is an RDD?


What's in a name might be true for a rose, but perhaps not for Resilient Distributed Datasets (RDD) which, in essence, describes what an RDD is.

They are basically datasets, which are distributed across a cluster (remember the Spark framework is inherently based on an MPP architecture), and provide resilience (automatic failover) by nature.

Before we go into any further detail, let's try to understand this a little bit, and again we are trying to be as abstract as possible. Let us assume that you have a sensor data from aircraft sensors and you want to analyze the data irrespective of its size and locality. For example, an Airbus A350 has roughly 6000 sensors across the entire plane and generates 2.5 TB data per day, while the newer model expected to launch in 2020 will generate roughly 7.5 TB per day. From a data engineering point of view, it might be important to understand the data pipeline, but from an analyst and a data scientist point of view, the major concern is to...

Operations on RDD


Two major operation types can be performed on an RDD. They are called:

  • Transformations
  • Actions

Transformations

Transformations are operations that create a new dataset, as RDDs are immutable. They are used to transform data from one to another, which could result in amplification of the data, reduction of the data, or a totally different shape altogether. These operations do not return any value back to the driver program, and hence are lazily evaluated, which is one of the main benefits of Spark.

An example of a transformation would be a map function that will pass through each element of the RDD and return a totally new RDD representing the results of application of the function on the original dataset.

Actions

Actions are operations that return a value to the driver program. As previously discussed, all transformations in Spark are lazy, which essentially means that Spark remembers all the transformations carried out on an RDD, and applies them in the most optimal fashion...

Passing functions to Spark (Scala)


As you have seen in the previous example, passing functions is a critical functionality provided by Spark. From a user's point of view you would pass the function in your driver program, and Spark would figure out the location of the data partitions across the cluster memory, running it in parallel. The exact syntax of passing functions differs by the programming language. Since Spark has been written in Scala, we'll discuss Scala first.

In Scala, the recommended ways to pass functions to the Spark framework are as follows:

  • Anonymous functions
  • Static singleton methods

Anonymous functions

Anonymous functions are used for short pieces of code. They are also referred to as lambda expressions, and are a cool and elegant feature of the programming language. The reason they are called anonymous functions is because you can give any name to the input argument and the result would be the same.

For example, the following code examples would produce the same output:

val...

Passing functions to Spark (Java)


In Java, to create a function you will have to implement the interfaces available in the org.apache.spark.api.java function package. There are two popular ways to create such functions:

  • Implement the interface in your own class, and pass the instance to Spark.
  • Starting Java 8, you can use Lambda expressions to pass off the functions to the Spark framework.

Let's implement the preceding word count examples in Java:

Figure 2.13: Code example of Java implementation of word count (inline functions)

If you belong to a group of programmers who feel that writing inline functions makes the code complex and unreadable (a lot of people do agree to that assertion), you may want to create separate functions and call them as follows:

Figure 2.14: Code example of Java implementation of word count

Passing functions to Spark (Python)


Python provides a simple way to pass functions to Spark. The Spark programming guide available at spark.apache.org suggests there are three recommended ways to do this:

  • Lambda expressions is the ideal way for short functions that can be written inside a single expression
  • Local defs inside the function calling into Spark for longer code
  • Top-level functions in a module

While we have already looked at the lambda functions in some of the previous examples, let's look at local definitions of the functions. We can encapsulate our business logic which is splitting of words, and counting into two separate functions as shown below.

def splitter(lineOfText): 
     words = lineOfText.split(" ") 
     return len(words) 
def aggregate(numWordsLine1, numWordsLineNext): 
     totalWords = numWordsLine1 + numWordsLineNext 
     return totalWords 

Let's see the working code example:

Figure 2.15: Code example of Python word count (local definition...

Transformations


We've used few transformation functions in the examples in this chapter, but I would like to share with you a list of the most commonly used transformation functions in Apache Spark. You can find a complete list of functions in the official documentation http://bit.ly/RDDTransformations.

Most Common Transformations

 

map(func)

coalesce(numPartitions)

filter(func)

repartition(numPartitions)

flatMap(func)

repartitionAndSortWithinPartitions(partitioner)

mapPartitions(func)

join(otherDataset, [numTasks])

mapPartitionsWithIndex(func)

cogroup(otherDataset, [numTasks])

sample(withReplacement, fraction, seed)

cartesian(otherDataset)

Map(func)

The map transformation is the most commonly used and the simplest of transformations on an RDD. The map transformation applies the function passed in the arguments to each of the elements of the source RDD. In the previous examples, we have seen the usage of map() transformation where we have passed the split() function...

Set operations in Spark


For those of you who are from the database world and have now ventured into the world of big data, you're probably looking at how you can possibly apply set operations on Spark datasets. You might have realized that an RDD can be a representation of any sort of data, but it does not necessarily represent a set based data. The typical set operations in a database world include the following operations, and we'll see how some of these apply to Spark. However, it is important to remember that while Spark offers some of the ways to mimic these operations, spark doesn't allow you to apply conditions to these operations, which is common in SQL operations:

  • Distinct: Distinct operation provides you a non-duplicated set of data from the dataset
  • Intersection: The intersection operations returns only those elements that are available in both datasets
  • Union: A union operation returns the elements from both datasets
  • Subtract: A subtract operation returns the elements from one dataset...
Left arrow icon Right arrow icon
Download code icon Download Code

Key benefits

  • Familiarize yourself with the new features introduced in Apache Spark 2, as well as its components for Big Data processing and analytics
  • Manipulate your data, perform stream analytics and machine learning, and deploy your Spark models to production using practical examples
  • If you are new to Apache Spark and want to quickly get started with it, this book will help you

Description

Apache Spark has seen an unprecedented growth in terms of its adoption over the last few years, mainly because of its speed, diversity and real-time data processing capabilities. It has quickly become the preferred choice of tool for many Big Data professionals looking to find quick insights from large chunks of data. This book introduces you to the Apache Spark framework, and familiarizes you with all the latest features and capabilities introduced in Spark 2. Starting with a detailed introduction to Spark’s architecture and the installation procedure, this book covers everything you need to know about the Spark framework in the most practical manner. You will learn how to perform the basic ETL activities using Spark, and work with different components of Spark such as Spark SQL, as well as the Dataset and DataFrame APIs for manipulating your data. Then, you will perform machine learning using Spark MLlib, as well as perform streaming analytics and graph processing using the Spark Streaming and GraphX modules respectively. The book also gives special emphasis on deploying your Spark models, and how they can be operated in a clustered mode. During the course of the book, you will come across implementations of different real-world use-cases and examples, giving you the hands-on knowledge you need to use Apache Spark in the best possible manner.

Who is this book for?

This book is intended for aspiring Big Data professionals and anyone who wants to get started with Apache Spark for Big Data processing and analytics. If you’ve worked with Apache Spark before and want to get familiarized with the new features introduced in Spark 2, this book will also help you. Some fundamental understanding of Big Data concepts and knowledge of Scala programming is required to get the best out of this book.

What you will learn

  • Get a thorough overview of Big Data processing and analytics, and its importance to organizations and data professionals
  • Get familiarized with the Apache Spark ecosystem, and the new features released in Spark 2 for data processing and analysis
  • Get a thorough understanding of different modules of Apache Spark such as Spark SQL, Spark RDD, Spark Streaming, Spark MLlib and GraphX
  • Work with data of different file formats, and learn how to process it with Apache Spark
  • Introduce yourself to SparkR and walk through the details of data munging including selecting, aggregating and grouping data using R studio
  • Realize how to deploy Spark with YARN, MESOS or a Stand-alone cluster manager
  • Build effective recommendation engines with Spark using collaborative filtering

Product Details

Country selected
Publication date, Length, Edition, Language, ISBN-13
Publication date : Mar 28, 2017
Length: 356 pages
Edition : 1st
Language : English
ISBN-13 : 9781785885136
Category :
Languages :
Concepts :

What do you get with a Packt Subscription?

Free for first 7 days. $19.99 p/m after that. Cancel any time!
Product feature icon Unlimited ad-free access to the largest independent learning library in tech. Access this title and thousands more!
Product feature icon 50+ new titles added per month, including many first-to-market concepts and exclusive early access to books as they are being written.
Product feature icon Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.
Product feature icon Thousands of reference materials covering every tech concept you need to stay up to date.
Subscribe now
View plans & pricing

Product Details

Publication date : Mar 28, 2017
Length: 356 pages
Edition : 1st
Language : English
ISBN-13 : 9781785885136
Category :
Languages :
Concepts :

Packt Subscriptions

See our plans and pricing
Modal Close icon
$19.99 billed monthly
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Simple pricing, no contract
$199.99 billed annually
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Choose a DRM-free eBook or Video every month to keep
Feature tick icon PLUS own as many other DRM-free eBooks or Videos as you like for just Mex$85 each
Feature tick icon Exclusive print discounts
$279.99 billed in 18 months
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Choose a DRM-free eBook or Video every month to keep
Feature tick icon PLUS own as many other DRM-free eBooks or Videos as you like for just Mex$85 each
Feature tick icon Exclusive print discounts

Frequently bought together


Stars icon
Total Mex$ 3,261.97
Learning PySpark
Mex$1004.99
Learning Apache Spark 2
Mex$1004.99
Mastering Spark for Data Science
Mex$1251.99
Total Mex$ 3,261.97 Stars icon

Table of Contents

11 Chapters
1. Architecture and Installation Chevron down icon Chevron up icon
2. Transformations and Actions with Spark RDDs Chevron down icon Chevron up icon
3. ETL with Spark Chevron down icon Chevron up icon
4. Spark SQL Chevron down icon Chevron up icon
5. Spark Streaming Chevron down icon Chevron up icon
6. Machine Learning with Spark Chevron down icon Chevron up icon
7. GraphX Chevron down icon Chevron up icon
8. Operating in Clustered Mode Chevron down icon Chevron up icon
9. Building a Recommendation System Chevron down icon Chevron up icon
10. Customer Churn Prediction Chevron down icon Chevron up icon
Theres More with Spark Chevron down icon Chevron up icon

Customer reviews

Top Reviews
Rating distribution
Full star icon Full star icon Full star icon Half star icon Empty star icon 3.8
(6 Ratings)
5 star 50%
4 star 0%
3 star 33.3%
2 star 16.7%
1 star 0%
Filter icon Filter
Top Reviews

Filter reviews by




Shambhu Nath Mar 13, 2018
Full star icon Full star icon Full star icon Full star icon Full star icon 5
Delivery is awesome ! this book is used simple english language, So good for beginner, also explanations is good but I felt screenshot print is not so good !Thanks,Shambhu Nath
Amazon Verified review Amazon
Ivan Falcão Oct 17, 2017
Full star icon Full star icon Full star icon Full star icon Full star icon 5
Excelente livro. Apresenta uma base teórica considerável, além de diversos exemplos práticos. Certamente uma das melhores opções pra quem quer aprender mais spark
Amazon Verified review Amazon
Kalaiselvan Dec 25, 2017
Full star icon Full star icon Full star icon Full star icon Full star icon 5
Simple language, good book for hands on development
Amazon Verified review Amazon
Deepak May 20, 2019
Full star icon Full star icon Full star icon Empty star icon Empty star icon 3
Not much in details. Tells only on high level and gives the link to refer for further details. Returned it. Ordered learning spark from orielly.
Amazon Verified review Amazon
Jose VL Jun 21, 2017
Full star icon Full star icon Full star icon Empty star icon Empty star icon 3
Nice reading to learn about Spark. I'd have liked to see more information for developers ... maybe next edition :-)
Amazon Verified review Amazon
Get free access to Packt library with over 7500+ books and video courses for 7 days!
Start Free Trial

FAQs

What is included in a Packt subscription? Chevron down icon Chevron up icon

A subscription provides you with full access to view all Packt and licnesed content online, this includes exclusive access to Early Access titles. Depending on the tier chosen you can also earn credits and discounts to use for owning content

How can I cancel my subscription? Chevron down icon Chevron up icon

To cancel your subscription with us simply go to the account page - found in the top right of the page or at https://subscription.packtpub.com/my-account/subscription - From here you will see the ‘cancel subscription’ button in the grey box with your subscription information in.

What are credits? Chevron down icon Chevron up icon

Credits can be earned from reading 40 section of any title within the payment cycle - a month starting from the day of subscription payment. You also earn a Credit every month if you subscribe to our annual or 18 month plans. Credits can be used to buy books DRM free, the same way that you would pay for a book. Your credits can be found in the subscription homepage - subscription.packtpub.com - clicking on ‘the my’ library dropdown and selecting ‘credits’.

What happens if an Early Access Course is cancelled? Chevron down icon Chevron up icon

Projects are rarely cancelled, but sometimes it's unavoidable. If an Early Access course is cancelled or excessively delayed, you can exchange your purchase for another course. For further details, please contact us here.

Where can I send feedback about an Early Access title? Chevron down icon Chevron up icon

If you have any feedback about the product you're reading, or Early Access in general, then please fill out a contact form here and we'll make sure the feedback gets to the right team. 

Can I download the code files for Early Access titles? Chevron down icon Chevron up icon

We try to ensure that all books in Early Access have code available to use, download, and fork on GitHub. This helps us be more agile in the development of the book, and helps keep the often changing code base of new versions and new technologies as up to date as possible. Unfortunately, however, there will be rare cases when it is not possible for us to have downloadable code samples available until publication.

When we publish the book, the code files will also be available to download from the Packt website.

How accurate is the publication date? Chevron down icon Chevron up icon

The publication date is as accurate as we can be at any point in the project. Unfortunately, delays can happen. Often those delays are out of our control, such as changes to the technology code base or delays in the tech release. We do our best to give you an accurate estimate of the publication date at any given time, and as more chapters are delivered, the more accurate the delivery date will become.

How will I know when new chapters are ready? Chevron down icon Chevron up icon

We'll let you know every time there has been an update to a course that you've bought in Early Access. You'll get an email to let you know there has been a new chapter, or a change to a previous chapter. The new chapters are automatically added to your account, so you can also check back there any time you're ready and download or read them online.

I am a Packt subscriber, do I get Early Access? Chevron down icon Chevron up icon

Yes, all Early Access content is fully available through your subscription. You will need to have a paid for or active trial subscription in order to access all titles.

How is Early Access delivered? Chevron down icon Chevron up icon

Early Access is currently only available as a PDF or through our online reader. As we make changes or add new chapters, the files in your Packt account will be updated so you can download them again or view them online immediately.

How do I buy Early Access content? Chevron down icon Chevron up icon

Early Access is a way of us getting our content to you quicker, but the method of buying the Early Access course is still the same. Just find the course you want to buy, go through the check-out steps, and you’ll get a confirmation email from us with information and a link to the relevant Early Access courses.

What is Early Access? Chevron down icon Chevron up icon

Keeping up to date with the latest technology is difficult; new versions, new frameworks, new techniques. This feature gives you a head-start to our content, as it's being created. With Early Access you'll receive each chapter as it's written, and get regular updates throughout the product's development, as well as the final course as soon as it's ready.We created Early Access as a means of giving you the information you need, as soon as it's available. As we go through the process of developing a course, 99% of it can be ready but we can't publish until that last 1% falls in to place. Early Access helps to unlock the potential of our content early, to help you start your learning when you need it most. You not only get access to every chapter as it's delivered, edited, and updated, but you'll also get the finalized, DRM-free product to download in any format you want when it's published. As a member of Packt, you'll also be eligible for our exclusive offers, including a free course every day, and discounts on new and popular titles.