Introduction

It isn't very efficient to process large batches of information via a web service, nor is it particularly desirable to pull data from an application database during peak hours. Thus, many organizations still maintain a file-based overnight batch processes using large extracts in file format.

In addition, many older, legacy applications rely solely on file-based data for communicating with the outside world.

It is therefore very important for the data integration developer to understand many file types and be able to manage them efficiently and effectively.

Note

This chapter deals with "flat" files, which, for our purposes means files that do not carry their metadata with them, such as XML or JSON, that are described in Chapter 9, Working with XML, Queues, and Web Services.

This does not mean that we will only deal with simple files. Some of the recipes in this chapter will deal with complex hierarchical file structures.