Using Metadata Injection to re-use transformations

Earlier in this chapter, we covered how to re-use transformations and jobs that utilize the same data structures and allowed for smaller portions of transformations to be broken out and used by several different transformations. Another common pattern is repeating a given process, but having a very different dataset flow. For instance, suppose we wanted to pull data from multiple tables and perform the same kind of logic on the data stream without having to write the transformation over for the different data stream. While some of that functionality could be done with Mappings and parameterization of jobs or transformations, Metadata Injection will allow for a transformation to be reused against different data streams based on the metadata of the stream.

Getting ready

For this recipe, we will be reusing the book_news dataset used in earlier chapters. You can find the files used to create this dataset on the book's website.

How to do it...

This...