Search icon Close icon
Search icon CANCEL
Subscription
0
Cart icon
Cart
Close icon
You have no products in your basket yet
Save more on your purchases!
Savings automatically calculated. No voucher code required
Arrow left icon
All Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletters
Free Learning
Arrow right icon
Arrow up icon
GO TO TOP
Apache Spark 2.x Machine Learning Cookbook

You're reading from  Apache Spark 2.x Machine Learning Cookbook

Product type Book
Published in Sep 2017
Publisher Packt
ISBN-13 9781783551606
Pages 666 pages
Edition 1st Edition
Languages
Concepts
Authors (5):
Mohammed Guller Mohammed Guller
Profile icon Mohammed Guller
LinkedIn
Author of Big Data Analytics with Spark - http://www.apress.com/9781484209653
Read more
See other products by Mohammed Guller
Siamak Amirghodsi Siamak Amirghodsi
Profile icon Siamak Amirghodsi
LinkedIn
Siamak Amirghodsi (Sammy) is interested in building advanced technical teams, executive management, Spark, Hadoop, big data analytics, AI, deep learning nets, TensorFlow, cognitive models, swarm algorithms, real-time streaming systems, quantum computing, financial risk management, trading signal discovery, econometrics, long-term financial cycles, IoT, blockchain, probabilistic graphical models, cryptography, and NLP.
Read more
See other products by Siamak Amirghodsi
Shuen Mei Shuen Mei
Profile icon Shuen Mei
LinkedIn
Shuen Mei is a big data analytic platforms expert with 15+ years of experience in designing, building, and executing large-scale, enterprise-distributed financial systems with mission-critical low-latency requirements. He is certified in the Apache Spark, Cloudera Big Data platform, including Developer, Admin, and HBase. He is also a certified AWS solutions architect with emphasis on peta-byte range real-time data platform systems.
Read more
See other products by Shuen Mei
Meenakshi Rajendran Meenakshi Rajendran
Profile icon Meenakshi Rajendran
LinkedIn
Meenakshi Rajendran is experienced in the end-to-end delivery of data analytics and data science products for leading financial institutions. Meenakshi holds a master's degree in business administration and is a certified PMP with over 13 years of experience in global software delivery environments. Her areas of research and interest are Apache Spark, cloud, regulatory data governance, machine learning, Cassandra, and managing global data teams at scale.
Read more
See other products by Meenakshi Rajendran
Broderick Hall Broderick Hall
Profile icon Broderick Hall
LinkedIn
Broderick Hall is a hands-on big data analytics expert and holds a masters degree in computer science with 20 years of experience in designing and developing complex enterprise-wide software applications with real-time and regulatory requirements at a global scale. He is a deep learning early adopter and is currently working on a large-scale cloud-based data platform with deep learning net augmentation.
Read more
See other products by Broderick Hall
View More author details
Toc

Table of Contents (20) Chapters close

Title Page
Credits
About the Authors
About the Reviewer
www.PacktPub.com
Customer Feedback
Preface
1. Practical Machine Learning with Spark Using Scala 2. Just Enough Linear Algebra for Machine Learning with Spark 3. Spark's Three Data Musketeers for Machine Learning - Perfect Together 4. Common Recipes for Implementing a Robust Machine Learning System 5. Practical Machine Learning with Regression and Classification in Spark 2.0 - Part I 6. Practical Machine Learning with Regression and Classification in Spark 2.0 - Part II 7. Recommendation Engine that Scales with Spark 8. Unsupervised Clustering with Apache Spark 2.0 9. Optimization - Going Down the Hill with Gradient Descent 10. Building Machine Learning Systems with Decision Tree and Ensemble Models 11. Curse of High-Dimensionality in Big Data 12. Implementing Text Analytics with Spark 2.0 ML Library 13. Spark Streaming and Machine Learning Library

Downloading a complete dump of Wikipedia for a real-life Spark ML project

In this recipe, we will be downloading and exploring a dump of so we can have a real-life example. The dataset that we will be downloading in this recipe is a dump of Wikipedia articles. You will need the command-line tool curl, or a browser to retrieve a compressed file, which is about 13.6 GB at this time. Due to the size, we recommend the curl command-line tool.

How to do it...

  1. You can start with downloading the dataset using the following command:
curl -L -O http://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles-multistream.xml.bz2
  1. Now you want to decompress the ZIP file:
bunzip2 enwiki-latest-pages-articles-multistream.xml.bz2

This should create an uncompressed file which is named enwiki-latest-pages-articles-multistream.xml and is about 56 GB.

  1. Let us take a look at the Wikipedia XML file:
head -n50 enwiki-latest-pages-articles-multistream.xml
<mediawiki xmlns=http://www.mediawiki.org/xml/export-0...
lock icon The rest of the chapter is locked
arrow left Previous Section
Section 5 of 7
Next Section arrow right
Register for a free Packt account to unlock a world of extra content!
A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.
Sign up now
Unlock this book and the full library FREE for 7 days
Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of
Start free trial
Renews at $15.99/month. Cancel anytime}

Authors (5)

Left arrow icon
Profile icon Mohammed Guller
LinkedIn
Author of Big Data Analytics with Spark - http://www.apress.com/9781484209653
Read more
See other products by Mohammed Guller
Profile icon Siamak Amirghodsi
LinkedIn
Siamak Amirghodsi (Sammy) is interested in building advanced technical teams, executive management, Spark, Hadoop, big data analytics, AI, deep learning nets, TensorFlow, cognitive models, swarm algorithms, real-time streaming systems, quantum computing, financial risk management, trading signal discovery, econometrics, long-term financial cycles, IoT, blockchain, probabilistic graphical models, cryptography, and NLP.
Read more
See other products by Siamak Amirghodsi
Profile icon Shuen Mei
LinkedIn
Shuen Mei is a big data analytic platforms expert with 15+ years of experience in designing, building, and executing large-scale, enterprise-distributed financial systems with mission-critical low-latency requirements. He is certified in the Apache Spark, Cloudera Big Data platform, including Developer, Admin, and HBase. He is also a certified AWS solutions architect with emphasis on peta-byte range real-time data platform systems.
Read more
See other products by Shuen Mei
Profile icon Meenakshi Rajendran
LinkedIn
Meenakshi Rajendran is experienced in the end-to-end delivery of data analytics and data science products for leading financial institutions. Meenakshi holds a master's degree in business administration and is a certified PMP with over 13 years of experience in global software delivery environments. Her areas of research and interest are Apache Spark, cloud, regulatory data governance, machine learning, Cassandra, and managing global data teams at scale.
Read more
See other products by Meenakshi Rajendran
Profile icon Broderick Hall
LinkedIn
Broderick Hall is a hands-on big data analytics expert and holds a masters degree in computer science with 20 years of experience in designing and developing complex enterprise-wide software applications with real-time and regulatory requirements at a global scale. He is a deep learning early adopter and is currently working on a large-scale cloud-based data platform with deep learning net augmentation.
Read more
See other products by Broderick Hall
Right arrow icon

Other recommended products

Related to this chapter
Left arrow icon
Hands-On Big Data Analytics with PySpark
Hands-On Big Data Analytics with PySpark
Read more
In this book, you'll learn to implement some practical and proven techniques to improve aspects of programming and administration in Apache Spark. Techniques are demonstrated using practical examples and best practices. You will also learn how to use Spark and its Python API to create performant analytics with large-scale data.
Read more
Mar 2019 6 hours 4 minutes
Machine Learning with Spark
Machine Learning with Spark
Read more
Spark ML is the machine learning module of Spark. It uses in-memory RDDs to process machine learning models faster for clustering, classification, and regression.
Read more
Apr 2017 17 hours 44 minutes
Hands-On Data Analysis with Scala
Hands-On Data Analysis with Scala
Read more
This book will help you perform effective data analysis with Scala using practical examples. You will come across different challenges and their effective solutions for a variety of data processing tasks - be it data exploration, data manipulation, or real-time data analysis using Apache Spark.
Read more
May 2019 9 hours 56 minutes
Apache Spark Quick Start Guide
Apache Spark Quick Start Guide
Read more
Apache Spark is a ?exible in-memory framework that allows processing of both batch and real-time data. Its unified engine has made it quite popular for big data use cases. This book will help you to quickly get started with Apache Spark 2.0 and write efficient big data applications for a variety of use cases.
Read more
Jan 2019 5 hours 8 minutes
Modern Scala Projects
Modern Scala Projects
Read more
Scala is a multipurpose programming language, especially for analyzing large datasets without impacting the application performance. Its functional libraries can interact with databases and build scalable frameworks that create robust data pipelines. This book showcases how you can use Scala and its constructs to meet specific project demands.
Read more
Jul 2018 11 hours 8 minutes
PySpark Cookbook
PySpark Cookbook
Read more
This cookbook presents recipes on leveraging the power of Python and putting it to use in the Apache Spark ecosystem. By the end of this book, you will be able to solve any problem associated with building effective, data-intensive applications and performing machine learning and structured streaming using PySpark.
Read more
Jun 2018 11 hours 0 minutes
Learning Apache Spark 2
Learning Apache Spark 2
Read more
Apache Spark is one of the most popular Big Data processing frameworks today, delivering speed, accuracy and real-time results – all in one solution. With this book, you will delve into the world of Apache Spark and learn about the new features introduced in Spark 2, along with the architecture and the associated concepts. A comprehensive guide to Apache Spark 2 for beginners, this book covers everything you need to know to get up and running with Big Data processing, machine learning and stream processing with Apache Spark, and allows you to easily understand each of these concepts through real-world examples.
Read more
Mar 2017 11 hours 52 minutes
Machine Learning with Scala Quick Start Guide
Machine Learning with Scala Quick Start Guide
Read more
Scala as a programming language is a highly scalable integration of object-oriented and functional programming, which makes it easy to build scalable and complex big data applications. This book is a handy guide for machine learning developers and data scientists who want to train effective machine learning models using this popular language.
Read more
Apr 2019 7 hours 20 minutes
Scala and Spark for Big Data Analytics
Scala and Spark for Big Data Analytics
Read more
Over the last few years, Scala has been adopted increasingly, especially in the field of data science and analytics, along with Apache Spark, which is built on Scala and is widely used in the field of analytics. With this book, you’ll learn how to leverage the power of both Scala and Spark to make sense of big data.
Read more
Jul 2017 26 hours 32 minutes
Apache Spark 2.x for Java Developers
Apache Spark 2.x for Java Developers
Read more
Apache Spark is the buzzword in the big data industry right now, especially with the increasing need for real-time streaming and data processing. While Spark is built on Scala, the Spark Java API exposes all the Spark features available in the Scala version for Java developers. This book will show you how you can implement various functionalities of the Apache Spark framework in Java, without stepping out of your comfort zone.
Read more
Jul 2017 11 hours 40 minutes
Learning Spark SQL
Learning Spark SQL
Read more
In the past year, Apache Spark has been increasingly adopted for development of distributed applications. Spark SQL APIs provides an optimized interface that helps developers build such applications quickly and easily. However, designing web-scale production applications using Spark SQL APIs can be a complex task. Understanding the design and implementation best practices for Spark SQL API based applications before you start your project will help you avoid these problems and ensure that your project is a success. Learning Spark SQL gives an insight into the engineering practices used to design and build real-world Spark-based applications. The hands-on examples will give you the required confidence to work on any future projects you encounter in Spark SQL.
Read more
Sep 2017 15 hours 4 minutes
Apache Spark 2.x Cookbook
Apache Spark 2.x Cookbook
Read more
Apache Spark has become the hottest platform and sought after skill set when it comes to the fields of Big Data, Analytics and Data Science. Apache Spark 2.x comes with series of new improvements in the areas of performance, scalability, operational and production readiness for structured processing of massive datasets. This book brings in a systematic way of getting a practical hands on to using its improved programming APIs, expanded SQL functionalities and implement distributed machine learning applications with Spark ML. Through the course of chapters, you will have explored the power of Spark DataFrames/Datasets, harness MLLib for Data mining, analyze complex problems with iterative or multi-stage Spark scripts and other associated toolsets such as Spark SQL, Spark Streaming and GraphX .
Read more
May 2017 9 hours 48 minutes
Right arrow icon

Personalised recommendations for you

Based on your interests and search pattern
Left arrow icon
Et al.
Et al.
Read more
Ever wonder why speech recognition systems don't understand the Scottish accent, or what would happen if an astronaut only ate mac 'n' cheese, or other spurious reflections you'd have at a bar? We did, then collated those deliberations into absurd research articles with fake figures and methodologies inspired by even more fictionally absurd studies.
Read more
Aug 2023 7 hours 40 minutes
Generative AI with LangChain
Generative AI with LangChain
Read more
This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.
Read more
Dec 2023 12 hours 0 minutes
Generative AI with LangChain
Generative AI with LangChain
Read more
This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.
Read more
Dec 2023 12 hours 0 minutes
Generative AI with LangChain
Generative AI with LangChain
Read more
This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.
Read more
Dec 2023 12 hours 0 minutes
Generative AI with LangChain
Generative AI with LangChain
Read more
This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.
Read more
Dec 2023 12 hours 0 minutes
Mastering Tableau 2023
Mastering Tableau 2023
Read more
This book is a comprehensive resource to mastering your Tableau skills and becoming a BI expert. As you progress, you will learn how to build advanced dashboards and improve your storytelling to derive key business insight, as well as make you well-versed with advanced functionalities of Tableau in the business intelligence domain.
Read more
Aug 2023 22 hours 48 minutes
Building AI Applications with ChatGPT APIs
Building AI Applications with ChatGPT APIs
Read more
This guide covers all ChatGPT API features for effortless creation of robust AI powered apps. With its help, you’ll be able to leverage ChatGPT’s cutting-edge NLP models to take your app development skills to the next level. You’ll also work on ten exciting projects that will give you the practical know-how that you can apply to your existing applications.
Read more
Sep 2023 8 hours 36 minutes
Building AI Applications with ChatGPT APIs
Building AI Applications with ChatGPT APIs
Read more
This guide covers all ChatGPT API features for effortless creation of robust AI powered apps. With its help, you’ll be able to leverage ChatGPT’s cutting-edge NLP models to take your app development skills to the next level. You’ll also work on ten exciting projects that will give you the practical know-how that you can apply to your existing applications.
Read more
Sep 2023 8 hours 36 minutes
Data Engineering with AWS
Data Engineering with AWS
Read more
Embark on a journey to master data engineering pipelines on AWS! Our book offers a hands-on experience of AWS services for ingesting, transforming, and consuming data. Whether you're an absolute beginner or someone with basic data engineering experience, this guide is an indispensable resource.
Read more
Oct 2023 21 hours 12 minutes
Modern Data Architecture on AWS
Modern Data Architecture on AWS
Read more
Every organization wants an agile, performant, and cost-effective data platform that meets all their current and future business needs. Purpose-built AWS analytics services and their features play a big part in building such a modern data platform. This book brings to you all the design and architectural patterns that’ll help you achieve this goal.
Read more
Aug 2023 14 hours 0 minutes
Practical Guide to Applied Conformal Prediction in Python
Practical Guide to Applied Conformal Prediction in Python
Read more
Discover the power of Conformal Prediction with the "Practical Guide to Applied Conformal Prediction in Python." Master the latest techniques to quantify uncertainty in machine learning and computer vision models, and seamlessly apply them to your industry applications.
Read more
Dec 2023 8 hours 0 minutes
TinyML Cookbook
TinyML Cookbook
Read more
With over 70 project-based recipes, the TinyML Cookbook is a practical guide that will help you to get the most out of your microcontrollers. It provides a comprehensive understanding of the theoretical foundations while giving you hands-on experience training ML models for deployment on Arduino Nano 33 BLE Sense, Raspberry Pi Pico, and SparkFun RedBoard Artemis Nano microcontrollers.
Read more
Nov 2023 22 hours 8 minutes
Right arrow icon