Working with flat files in Data Flow

In this recipe, we will demonstrate the use of the Flat File Source component that is often used in data integration projects. As explained in Chapter 2, Control Flow Tasks for security reasons, the Operational Systems (OS) owners usually prefer to push data to an external location in spite of providing direct access to the OS.

Data quality problems exist in conventional databases such as SQL Server, Oracle, DB2, and so on. It's possible to imagine the quality problems that could arise when dealing with flat files, the construction of these files could generate several problems because the external system that will read each record in this file (for example, the Extract step of the ETL process) needs to know how to split the record into columns and rows. The Row delimiter is required in order to split each row whereas the Column delimiter is required to split each column from each row. But in many cases, the Column delimiter can appear in the column content...