Using reload at each row to process real-time / near real-time data

Prior to attempting this recipe, you will need to ensure that you have an active MySQL database and have updated the context variables within the context MySQL to contain your database and login details. See the recipe Setting up a database connection in Chapter 7, Working with databases, for details on how to do this.

As we mentioned in the recipe Joining using tMap, tMap will load the join data into memory prior to processing the main input rows. This works fine for a batch processing model, because the overhead of loading large lookups in memory is offset against the efficiency in processing the joins against the data held in memory.

This paradigm does not however work in a real-time situation. In a real-time process, it would be unacceptable to wait for say 5 minutes to unload a large database table prior to processing a single record.

This recipe shows how the tMap 'reload at each row' feature can be used to process small...