Search icon CANCEL
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Conferences
Free Learning
Arrow right icon
Arrow up icon
GO TO TOP
Python High Performance, Second Edition

You're reading from   Python High Performance, Second Edition Build high-performing, concurrent, and distributed applications

Arrow left icon
Product type Paperback
Published in May 2017
Publisher Packt
ISBN-13 9781787282896
Length 270 pages
Edition 2nd Edition
Languages
Tools
Arrow right icon
Author (1):
Arrow left icon
Dr. Gabriele Lanaro Dr. Gabriele Lanaro
Author Profile Icon Dr. Gabriele Lanaro
Dr. Gabriele Lanaro
Arrow right icon
View More author details
Toc

Table of Contents (10) Chapters Close

Preface Benchmarking and Profiling FREE CHAPTER Pure Python Optimizations Fast Array Operations with NumPy and Pandas C Performance with Cython Exploring Compilers Implementing Concurrency Parallel Processing Distributed Processing Designing for High Performance

Using PySpark

Nowadays, Apache Spark is one of the most popular projects for distributed computing. Developed in Scala, Spark was released in 2014, and integrates with HDFS and provides several advantages and improvements over the Hadoop MapReduce framework.

Contrary to Hadoop MapReduce, Spark is designed to process data interactively and supports APIs for the Java, Scala, and Python programming languages. Given its different architecture, especially by the fact that Spark keep results in memory, Spark is generally much faster than Hadoop MapReduce.

Setting up Spark and PySpark

Setting up PySpark from scratch requires the installation of the Java and Scala runtimes, the compilation of the project from source, and the configuration of Python and Jupyter notebook...

lock icon The rest of the chapter is locked
Register for a free Packt account to unlock a world of extra content!
A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.
Unlock this book and the full library FREE for 7 days
Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of
Renews at £16.99/month. Cancel anytime