Packt+ | Advance your knowledge in tech

GO TO TOP

You're reading from Apache Spark 2: Data Processing and Real-Time Analytics Master complex big data processing, stream analytics, and machine learning with Apache Spark

Product type Course

Published in Dec 2018

Publisher Packt

ISBN-13 9781789959208

Length 616 pages

Edition 1st Edition

Languages

Processing

Tools

Apache Spark

Concepts

Big Data

Authors (7):

Sridhar Alla

Sridhar?Alla?is the co-founder and CTO of Blue Whale Consulting and is expert at helping companies (big and small) define their vision for systems and capabilities that will allow them to establish a strategic execution plan to deal with the ever-growing data collected to support analytics and product teams. He has very experienced at dealing with all aspects of data collection, security, governance, and processing as part of end-to-end big data analytics and machine learning initiatives (including predictive modeling, deep learning, and ML automation).nSridhar?is a published book author and an avid presenter at numerous conferences, including Strata, Hadoop World, and Spark Summit.? He also has several patents filed with the US PTO on large-scale computing and distributed systems.?nHe has over 18 years' experience writing code in Scala, Java, C, C++, Python, R, and Go, and has extensive hands-on knowledge of Spark, Flink, TensorFlow, Keras, Hadoop, Cassandra, HBase, MongoDB, Riak, Redis, Zeppelin, Mesos, Docker, Kafka, ElasticSearch, Solr, H2O, machine learning, text analytics, distributed computing, and high-performance computing.nSridhar lives with his wife and daughter in New Jersey and in his spare time loves blogging and coaching organizations on next-generation advancements in technology and their alignment with business goals.

See other products by Sridhar Alla

Romeo Kienzler

Romeo Kienzler works as the chief data scientist in the IBM Watson IoT worldwide team, helping clients to apply advanced machine learning at scale on their IoT sensor data. He holds a Master's degree in computer science from the Swiss Federal Institute of Technology, Zurich, with a specialization in information systems, bioinformatics, and applied statistics.

See other products by Romeo Kienzler

Siamak Amirghodsi

Siamak Amirghodsi (Sammy) is interested in building advanced technical teams, executive management, Spark, Hadoop, big data analytics, AI, deep learning nets, TensorFlow, cognitive models, swarm algorithms, real-time streaming systems, quantum computing, financial risk management, trading signal discovery, econometrics, long-term financial cycles, IoT, blockchain, probabilistic graphical models, cryptography, and NLP.

See other products by Siamak Amirghodsi

Broderick Hall

Broderick Hall is a hands-on big data analytics expert and holds a masters degree in computer science with 20 years of experience in designing and developing complex enterprise-wide software applications with real-time and regulatory requirements at a global scale. He is a deep learning early adopter and is currently working on a large-scale cloud-based data platform with deep learning net augmentation.

See other products by Broderick Hall

Md. Rezaul Karim

Md. Rezaul Karim is a researcher, author, and data science enthusiast with a strong computer science background, coupled with 10 years of research and development experience in machine learning, deep learning, and data mining algorithms to solve emerging bioinformatics research problems by making them explainable. He is passionate about applied machine learning, knowledge graphs, and explainable artificial intelligence (XAI).nCurrently, he is working as a research scientist at Fraunhofer FIT, Germany. He is also a PhD candidate at RWTH Aachen University, Germany. Before joining FIT, he worked as a researcher at the Insight Centre for Data Analytics, Ireland. Previously, he worked as a lead software engineer at Samsung Electronics, Korea.

See other products by Md. Rezaul Karim

Meenakshi Rajendran

Meenakshi Rajendran is experienced in the end-to-end delivery of data analytics and data science products for leading financial institutions. Meenakshi holds a master's degree in business administration and is a certified PMP with over 13 years of experience in global software delivery environments. Her areas of research and interest are Apache Spark, cloud, regulatory data governance, machine learning, Cassandra, and managing global data teams at scale.

See other products by Meenakshi Rajendran

Shuen Mei

Shuen Mei is a big data analytic platforms expert with 15+ years of experience in designing, building, and executing large-scale, enterprise-distributed financial systems with mission-critical low-latency requirements. He is certified in the Apache Spark, Cloudera Big Data platform, including Developer, Admin, and HBase. He is also a certified AWS solutions architect with emphasis on peta-byte range real-time data platform systems.

See other products by Shuen Mei

+3 more Show less

View More author details

Table of Contents (23) Chapters

Title Page

About Packt

Contributors

Preface

1. A First Taste and What's New in Apache Spark V2 FREE CHAPTER

Spark machine learning
Spark Streaming
Spark SQL
Spark graph processing
Extended ecosystem
What's new in Apache Spark V2?
Cluster design
Cluster management
Cloud-based deployments
Performance
Cloud
Summary

2. Apache Spark Streaming

Overview
Errors and recovery
Streaming sources
Summary

3. Structured Streaming

The concept of continuous applications
Windowing
Increased performance with good old friends
How transparent fault tolerance and exactly-once delivery guarantee is achieved
Example - connection to a MQTT message broker
Summary

4. Apache Spark MLlib

Architecture
Classification with Naive Bayes
Clustering with K-Means
Artificial neural networks
Summary

5. Apache SparkML

What does the new API look like?
The concept of pipelines
Model evaluation
CrossValidation and hyperparameter tuning
Winning a Kaggle competition with Apache SparkML
Summary

6. Apache SystemML

Why do we need just another library?
A cost-based optimizer for machine learning algorithms
Performance measurements
Apache SystemML in action
Summary

7. Apache Spark GraphX

Overview
Graph analytics/processing with GraphX
Summary

8. Spark Tuning

Monitoring Spark jobs
Spark configuration
Common mistakes in Spark app development
Optimization techniques
Summary

9. Testing and Debugging Spark

Testing in a distributed environment
Testing Spark applications
Debugging Spark applications
Summary

10. Practical Machine Learning with Spark Using Scala

Introduction
Configuring IntelliJ to work with Spark and run Spark ML sample codes
Running a sample ML code from Spark
Identifying data sources for practical machine learning
Running your first program using Apache Spark 2.0 with the IntelliJ IDE
How to add graphics to your Spark program

11. Spark's Three Data Musketeers for Machine Learning - Perfect Together

Introduction
Creating RDDs with Spark 2.0 using internal data sources
Creating RDDs with Spark 2.0 using external data sources
Transforming RDDs with Spark 2.0 using the filter() API
Transforming RDDs with the super useful flatMap() API
Transforming RDDs with set operation APIs
RDD transformation/aggregation with groupBy() and reduceByKey()
Transforming RDDs with the zip() API
Join transformation with paired key-value RDDs
Reduce and grouping transformation with paired key-value RDDs
Creating DataFrames from Scala data structures
Operating on DataFrames programmatically without SQL
Loading DataFrames and setup from an external source
Using DataFrames with standard SQL language - SparkSQL
Working with the Dataset API using a Scala Sequence
Creating and using Datasets from RDDs and back again
Working with JSON using the Dataset API and SQL together
Functional programming with the Dataset API using domain objects

12. Common Recipes for Implementing a Robust Machine Learning System

Introduction
Spark's basic statistical API to help you build your own algorithms
ML pipelines for real-life machine learning applications
Normalizing data with Spark
Splitting data for training and testing
Common operations with the new Dataset API
Creating and using RDD versus DataFrame versus Dataset from a text file in Spark 2.0
LabeledPoint data structure for Spark ML
Getting access to Spark cluster in Spark 2.0
Getting access to Spark cluster pre-Spark 2.0
Getting access to SparkContext vis-a-vis SparkSession object in Spark 2.0
New model export and PMML markup in Spark 2.0
Regression model evaluation using Spark 2.0
Binary classification model evaluation using Spark 2.0
Multiclass classification model evaluation using Spark 2.0
Multilabel classification model evaluation using Spark 2.0
Using the Scala Breeze library to do graphics in Spark 2.0

13. Recommendation Engine that Scales with Spark

Introduction
Setting up the required data for a scalable recommendation engine in Spark 2.0
Exploring the movies data details for the recommendation system in Spark 2.0
Exploring the ratings data details for the recommendation system in Spark 2.0
Building a scalable recommendation engine using collaborative filtering in Spark 2.0

14. Unsupervised Clustering with Apache Spark 2.0

Introduction
Building a KMeans classifying system in Spark 2.0
Bisecting KMeans, the new kid on the block in Spark 2.0
Using Gaussian Mixture and Expectation Maximization (EM) in Spark to classify data
Classifying the vertices of a graph using Power Iteration Clustering (PIC) in Spark 2.0
Latent Dirichlet Allocation (LDA) to classify documents and text into topics
Streaming KMeans to classify data in near real-time

15. Implementing Text Analytics with Spark 2.0 ML Library

Introduction
Doing term frequency with Spark - everything that counts
Displaying similar words with Spark using Word2Vec
Downloading a complete dump of Wikipedia for a real-life Spark ML project
Using Latent Semantic Analysis for text analytics with Spark 2.0
Topic modeling with Latent Dirichlet allocation in Spark 2.0

16. Spark Streaming and Machine Learning Library

Introduction
Structured streaming for near real-time machine learning
Streaming DataFrames for real-time machine learning
Streaming Datasets for real-time machine learning
Streaming data and debugging with queueStream
Downloading and understanding the famous Iris data for unsupervised classification
Streaming KMeans for a real-time on-line classifier
Downloading wine quality data for streaming regression
Streaming linear regression for a real-time regression
Downloading Pima Diabetes data for supervised classification
Streaming logistic regression for an on-line classifier

1. Other Books You May Enjoy

Leave a review - let other readers know what you think

Index

Chapter 9. Testing and Debugging Spark

In an ideal world, we write perfect Spark codes and everything runs perfectly all the time, right? Just kidding; in practice, we know that working with large-scale datasets is hardly ever that easy, and there are inevitably some data points that will expose any corner cases with your code.

Considering the aforementioned challenges, therefore, in this chapter, we will see how difficult it can be to test an application if it is distributed; then, we will see some ways to tackle this. In a nutshell, the following topics will be cover throughout this chapter: