Processing a large number of files in parallel

If you have worked through the two previous recipes, you should already have in place a framework for detecting the presence of a variable number of files in a designated location. You will also have developed a method for processing each of those files as they appear in a prepared list (that is, a table). But consider a situation where there are hundreds or even thousands of files to be processed. Managing all of those files serially would likely prove to be a bottleneck.

In this recipe, we will enhance the file-processing framework by introducing a way to execute the most critical components in parallel.

Getting ready

We will start from where we left off with the previous recipe and to do that, we will build on what we already have: a table that contains a list of files to be processed.

How to do it...

1. Create a new package called ProcessFiles.

2. Create a new variable called FILE_PARAM and set it as Alphanumeric. Drag-and-drop the variable in the...