Environment Setup
In this chapter, we will outline two different environments for developing data engineering pipelines.
The first environment will use cloud-based tooling and services and will require no local environment setup. This is beneficial for multiple reasons. First, this type of environment is highly portable because you will be able to access it from any machine and any location as long as you have an internet connection and a browser. Second, it requires the least amount of setup to get started. The downside to this type of environment is that there are costs associated with another organization maintaining the systems you will be using for development.
The second environment will utilize your local machine to develop your pipeline code. This moves all the work of setting up environments to you but avoids the cloud costs mentioned earlier.
We will be covering the following topics in this chapter:
- Setting up a cloud environment
- Local environment setup...
