Snowflake, founded in 2012, started its journey to the data cloud by completely re-engineering the world of data and rethinking how a reliable, secure, high-performance, and scalable data-processing system should be architected for the cloud. It started with offering cloud-based data warehousing through a managed Software as a Service (SaaS) platform to load, analyze, and process large volumes of data. The success of Snowflake lies in the fact that it is a cloud-native managed solution that is built on top of the major public cloud providers such as Amazon Web Services, Microsoft Azure, and Google Cloud Platform by automatically providing a reliable, secure, high-performance, and scalable data processing system for organizations without the need to deploy hardware or install or configure any software.

As with any cloud data warehousing, Snowflake supports American National Standards Institute (ANSI) SQL as the language of choice. Although SQL is a powerful declarative...

In June 2022, Snowflake made a significant announcement, revealing the much-anticipated Snowpark for Python. This new release has rapidly emerged as the preferred programming language for Snowpark, providing users with a more extensive range of options for programming data in Snowflake. Moreover, Snowpark has simplified managing data architectures, enabling users to operate more quickly and efficiently.

Snowpark for Python is a cutting-edge, enterprise-grade, open-source innovation integrated into the Snowflake data cloud. As a result, the platform delivers a seamless, unified experience for data scientists and developers. In addition, the Snowpark for Python package is built upon the Snowflake Python connector. The Python connector enables users to execute SQL commands and other essential functions in Snowflake and Snowpark for Python empowers users to undertake more advanced data applications.

For instance, the platform permits users to run user...

The release of Snowpark transformed Snowflake into a complete data platform designed to support various workloads. Snowpark supports multiple workloads, such as data science and ML, data engineering, and data applications.

Data science and ML

Python is the favorite language for data scientists. Snowpark for Python supports popular libraries and frameworks such as pandas, NumPy, and scikit-learn, making it the ideal framework for data scientists to perform ML development in Snowflake. In addition, data scientists can use the DataFrames API to interact with data inside Snowflake and perform batch training and inference inside Snowflake. Developers can also use Snowpark for feature engineering, ML model inference, and end-to-end ML pipelines. Snowpark also provides a SnowparkML library to support data science and ML in Snowpark.

Data engineering

Data cleansing and ELT workloads are complex, and building a data pipeline with just...

The traditional big data approach has been in the industry for a long time and is unsuitable for modern cloud-based scalable workloads. Traditional architecture has many challenges, such as the following:

• De-coupling the compute and data into separate systems
• Running separate processing clusters for different languages
• Complexity in managing the system
• Data silos and data duplication
• Lack of unified security and governance

Snowflake solves the traditional system’s challenges using Snowpark, providing tremendous value to the data ecosystem and Snowflake users. The following diagram shows the difference between a traditional approach and Snowflake’s streamlined approach:

Figure 1.1 – Traditional versus Snowflake approach

As you can see from the difference between both approaches, Snowpark’s streamlined approach benefits both the business and the developers by...

Snowflake’s Snowpark perfectly coalesces SQL and Python, running complex data processing jobs in the Snowflake data cloud and enabling data engineers, data scientists, and developers to take advantage of Snowflake. In this chapter, we have seen the benefits of Snowpark and why Python is the preferred development language. We also covered different workloads that Snowpark supports.

In the next chapter, we will examine configuring and operating with Snowpark in detail and learn how to use Snowpark for various workloads.