Search icon CANCEL
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Conferences
Free Learning
Arrow right icon
Arrow up icon
GO TO TOP
Distributed Data Systems with Azure Databricks

You're reading from   Distributed Data Systems with Azure Databricks Create, deploy, and manage enterprise data pipelines

Arrow left icon
Product type Paperback
Published in May 2021
Publisher Packt
ISBN-13 9781838647216
Length 414 pages
Edition 1st Edition
Languages
Tools
Arrow right icon
Author (1):
Arrow left icon
Alan Bernardo Palacio Alan Bernardo Palacio
Author Profile Icon Alan Bernardo Palacio
Alan Bernardo Palacio
Arrow right icon
View More author details
Toc

Table of Contents (17) Chapters Close

Preface 1. Section 1: Introducing Databricks
2. Chapter 1: Introduction to Azure Databricks FREE CHAPTER 3. Chapter 2: Creating an Azure Databricks Workspace 4. Section 2: Data Pipelines with Databricks
5. Chapter 3: Creating ETL Operations with Azure Databricks 6. Chapter 4: Delta Lake with Azure Databricks 7. Chapter 5: Introducing Delta Engine 8. Chapter 6: Introducing Structured Streaming 9. Section 3: Machine and Deep Learning with Databricks
10. Chapter 7: Using Python Libraries in Azure Databricks 11. Chapter 8: Databricks Runtime for Machine Learning 12. Chapter 9: Databricks Runtime for Deep Learning 13. Chapter 10: Model Tracking and Tuning in Azure Databricks 14. Chapter 11: Managing and Serving Models with MLflow and MLeap 15. Chapter 12: Distributed Deep Learning in Azure Databricks 16. Other Books You May Enjoy

Reading a dataset

Reading datasets using Petastorm can be very simple. In this section, we will demonstrate how we can easily load a Petastorm dataset into two frequently used deep learning frameworks, which are TensorFlow and PyTorch:

  1. To load our Petastorm datasets, we use the petastorm.reader.Reader class, which implements the iterator interface that allows us to use plain Python to go over the samples very efficiently. The petastorm.reader.Reader class can be created using the petastorm.make_reader factory method:
    from petastorm import make_reader
    with make_reader('dfs://some_dataset') as reader:
       for sample in reader:
           print(sample.id)
           plt.imshow(sample.image1)
  2. The following code example shows how we can stream a dataset into the TensorFlow Examples class, which as we have seen before is a named tuple with the keys being the ones specified in the Unischema of...
lock icon The rest of the chapter is locked
Register for a free Packt account to unlock a world of extra content!
A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.
Unlock this book and the full library FREE for 7 days
Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of
Renews at $19.99/month. Cancel anytime