CSV File Input

    The CSV label for this transform is a misnomer because you can define whatever separator you want to use, such as pipes, tabs, and semicolons; you are not constrained to using commas. Internal processing allows this transform to process data quickly. Options for this transform are a subset of the Text File Input transform.

    This transform has fewer overall options than the general Text File Input transform, but it has a few advantages over it:

    • NIO — Native system calls for reading the file means faster performance, but it is limited to only local files currently. No VFS support.

    • Parallel running — If you configure this transform to run in multiple copies or in clustered mode, and you enable parallel running, each copy will read a separate block of a single file allowing you to distribute the file reading to several threads or even several slave nodes in a clustered pipeline.

    For information on valid date and numeric formats used in this transform, view the Date Formatting Table and Number Formatting Table.

    You can use the Metadata Injection supported fields with ETL Metadata Injection transform to pass metadata to your pipeline at runtime. The following Option and Value fields of the CSV File Input transform support metadata injection:

    • Values: Name, Length, Decimal, Type, Precision, Group, Format, Currency, and Trim Type