Stream Lookup
The data coming from the Source transform is first read into memory and is then used to look up data from the main stream.
Option | Description |
---|---|
Transform name | Name of the transform this name has to be unique in a single pipeline |
Lookup Transform | The Transform name where the lookup data is coming from |
The keys to lookup… | |
Fields to retrieve | You can specify the names of the fields to retrieve here, as well as the default value in case the value was not found or a new field name in case you didn’t like the old one. |
Preserve memory | Encodes rows of data to preserve memory while sorting. (Technical background: Hop will store the lookup data as raw bytes in a custom storage object that uses a hashcode of the bytes as the key. More CPU cost related to calculating the hashcode, less memory needed.) |
Key and value are exactly one integer field | Preserves memory while executing a sort by . Note: Works only when “Preserve memory” is checked. Cannot be combined with the “Use sorted list” option. |
Enable to store values using a sorted list; this provides better memory usage when working with data sets containing wide row. Note: Works only when “Preserve memory” is checked. Cannot be combined with the “Key and value are exactly one integer field” option. | |
Get fields | Automatically fills in the names of all the available fields on the source side (A); you can then delete all the fields you don’t want to use for lookup. |
Get lookup fields | Automatically inserts the names of all the available fields on the lookup side (B). You can then delete the fields you don’t want to retrieve |
All fields of this transform support metadata injection. You can use this transform with ETL Metadata Injection to pass metadata to your pipeline at runtime.