Scheduling execution
Data processing applications usually run frequently. Some run once a day and others run every few hours or minutes. The following is a list of some tools that can be used for job scheduling:
Cron: The time-based job scheduler in Unix-like operating systems. It is not a very sophisticated tool but suffices for scheduling few jobs.
Jenkins: The continuous integration tool that offers scheduling via a cron-like mechanism. It also preserves the history and messages, can send e-mail notifications, and use version control. It is capable of scheduling thousands of job executions per day.
Oozie: The official workflow scheduling system to manage Apache Hadoop jobs. It is a scalable, reliable, and extensible system, and it is built specifically to allow workflow scheduling. The downside is that it is based on XML configuration files that can easily grow in size and complexity.
Azkaban: The batch workflow job scheduler created at LinkedIn to run Hadoop jobs. It solves the problem of...
