Pipelines

A transform is each basic operation in your pipeline. Each transform performs an operation on the data: reads from a source, operates on data in the pipeline or writes to to a target.
A hop in a pipeline passes data between transforms.

There are two ways to create a pipeline.

click on the New option on the Horizontal toolbar and select the Pipeline option.

Your new pipeline is created. You’ll see the dialog below.

Now you are ready to add the first transform. Click anywhere in the pipeline canvas, the area where you’ll see the image below.

After clicking you will see the context dialog. This is the dialog you’ll use every time you need to select transforms to add to your pipeline.

For now, add a Generate Rows and a transform to your pipeline.

There are a number of ways to create a hop:

shift-drag: while holding down the shift key on your keyboard. Click on a transform, while holding down your primary mouse button, drag to the second transform. Release the primary mouse button and the shift key.
click on a transform in your pipeline to open the context dialog (the dialog you opened in the ‘click anywhere‘ step). Click the ‘Create hop’ button and select the transform you want to create the hop to.

Running a pipeline to see how it performs can be done in one of the following tasks:

You will see the Run Options dialog.

Make sure your configuration is selected and hit Launch.

You’ll see green check marks in the upper right corner of the transforms when a pipeline ran successfully.

After each run, the execution results are displayed in the panel on the bottom of your window. The Execution Results contains two tabs:

The Transform Metrics tab displays metrics per transform.

The logging tab displays the log of the pipeline according to the log level chosen on execution.