Metadata Injection

    This is typically used to automate repetitive ETL tasks.

    For example, you might have a simple pipeline to load transaction data values from a supplier, filter specific values, and output them to a file. If you have more than one supplier, you would need to run this simple pipeline for each supplier. Yet, with metadata injection, you can expand this simple repetitive pipeline by inserting metadata from another pipeline that contains the ETL Metadata Injection transform. This transform coordinates the data values from the various inputs through the metadata you define. This process reduces the need for you to adjust and run the repetitive pipeline for each specific input.

    The repetitive pipeline is known as the template pipeline. The template pipeline is called by the ETL Metadata Injection transform. You will create a pipeline to prepare what common values you want to use as metadata and inject these specific values through the ETL Metadata Injection transform.

    We recommend the following basic procedure for using this transform to inject metadata:

    1. Optimize your data for injection, such as preparing folder structures and inputs.

    2. Develop pipelines for the repetitive process (the template pipeline), for metadata injection through the ETL Metadata Injection transform, and for handling multiple inputs.

    The metadata is injected into the template pipeline through any transform that supports metadata injection.

    Options

    The ETL Metadata Injection transform features the two tabs with fields. Each tab is described below.

    To specify the source field as metadata to be injected, perform the following transforms:

    1. In the Target injection transform key column, double-click the field for which you want to specify a source field. The Source field dialog box opens.

    2. Optionally, select Use constant value to specify a constant value for the injected metadata through one of the following actions:

      • Manually entering a value.

      • Using an internal variable to set the value (${Internal.transform.Unique.Count} for example).

      • Using a combination of manually specified values and parameter values (${FILE_PREFIX}_${FILE_DATE}.txt for example).

    When specifying constant values for grouped lists of values like fields or filenames please note that there isn’t a good solution for that. Best practice is to use a transform to inject a complete set of constant values. You can map those in this metadata injection transform. It will do its best to accommodate you by allowing you to inject a single row in the group with the specified constant value.

    Injecting Metadata into the ETL Metadata Injection transform

    For injecting metadata into the ETL Metadata Injection transform itself, the following exceptions apply:

    • The target field for the ETL Metadata Injection transform inserting the metadata into the original injection is defined by [GROUP NAME].[FIELD NAME]. For example, if the GROUP NAME is ‘OUTPUT_FIELDS’ and the FIELD NAME is ‘OUTPUT_FIELDNAME’, you would set the target field to ‘OUTPUT_FIELDS.OUTPUT_FIELDNAME’.