Managing Spark job configurations
Microsoft Fabric is based on the Software-as-a-Service (SaaS) model, which provides features such as auto-integration, auto-optimization, and auto-tuning for all the workloads, including the fully managed Spark runtime, which powers data engineering and data science experiences. While the default configuration will work fine in most cases, empowering you to do more with less, there are times when you’ll need more control. For these types of scenarios, you can control several aspects of the Fabric Spark runtime and Spark jobs to suit your business needs.
Open any Fabric workspace, click on the Workspace settings icon at the top, and then, under Data Engineering/Science, select Spark compute. You will find various options to make changes, as shown in Figure 8.4:
Figure 8.4 – Spark compute settings
In the next few sections, we will look at Spark configurations, including starter and custom Spark pools, configuring...
