What are the benefits of writing tasks rather than using simple scripts?
Scripts are great for simple and one-off jobs. If you have a repetitive task to do – or even more so if there is a set of tasks that depend on each other, and you need to ensure that they don't run without a dependency missing, or that they won't override (or append to) existing data—then ETL pipelines and tasks are for you. As a free bonus, frameworks such as Luigi have a lot of utility code that helps to build pipelines – you won't need to write a solution for writing to S3 or a database, or parse a command-line command.
What is the base element of Luigi jobs?
The base element of Luigi jobs (pipelines) is the Task class. All the business logic of a task needs to be wrapped in the run method. Its output and dependencies are defined within the output and requires...
