Index transforms

    For example, suppose that you have airline data that’s scattered across multiple fields and categories, and you want to view a summary of the data that’s organized by airline, quarter, and then price. You can use a transform job to create a new, summarized index that’s organized by those specific categories.

    You can use transform jobs in two ways:

    1. Use the OpenSearch Dashboards UI to specify the index you want to transform and any optional data filters you want to use to filter the original index. Then select the fields you want to transform and the aggregations to use in the transformation. Finally, define a schedule for your job to follow.
    2. Use the transforms API to specify all the details about your job: the index you want to transform, target groups you want the transformed index to have, any aggregations you want to use to group columns, and a schedule for your job to follow.

    OpenSearch Dashboards provides a detailed summary of the jobs you created and their relevant information, such as associated indexes and job statuses. You can review and edit your job’s details and selections before creation, and even preview a transformed index’s data as you’re choosing which fields to transform. However, you can also use the REST API to create transform jobs and preview transform job results, but you must know all of the necessary settings and parameters to submit them as part of the HTTP request body. Submitting your transform job configurations as JSON scripts offers you more portability, allowing you to share and replicate your transform jobs, which is harder to do using OpenSearch Dashboards.

    Your use cases will help you decide which method to use to create transform jobs.

    If you don’t have any data in your cluster, you can use the sample flight data within OpenSearch Dashboards to try out transform jobs. Otherwise, after launching OpenSearch Dashboards, choose Index Management. Select Transform Jobs, and choose Create Transform Job.

    1. In the Job name and description section, specify a name and an optional description for your job.
    2. In the Indices section, select the source and target index. You can either select an existing target index or create a new one by entering a name for your new index. If you want to transform just a subset of your source index, choose Edit data filter, and use the OpenSearch query DSL to specify a subset of your source index. For more information about the OpenSearch query DSL, see query DSL.
    3. Choose Next.

    You can use groupings to place your data into separate buckets in your transformed index. For example, if you want to group all of the airport destinations within the sample flight data, you can group the field into a target field of DestAirportID_terms field, and you can find the grouped airport IDs in your transformed index after the transform job finishes.

    On the other hand, aggregations let you perform simple calculations. For example, you can include an aggregation in your transform job to define a new field of sum_of_total_ticket_price that calculates the sum of all airplane tickets, and then analyze the newly summer data within your transformed index.

    1. Repeat step 1 for any other fields that you want to transform.

    2. After selecting the fields that you want to transform and verifying the transformation, choose Next.
    1. Choose whether the job should be continuous. Continuous jobs execute at each transform execution interval and incrementally transform newly modified buckets, which can include new data added to the source indexes. Non-continuous jobs execute only once.
    2. For transformation execution interval, specify a transform interval in minutes, hours, or days. This interval dicatates how often continuous jobs should execute, and non-continuous jobs execute once after the interval elapses.
    3. Under Advanced, specify an optional amount for Pages per execution. A larger number means more data is processed in each search request, but also uses more memory and causes higher latency. Exceeding allowed memory limits can cause exceptions and errors to occur.

    After confirming your transform job’s details are correct, choose Create Transform Job. If you want to edit any part of the job, choose Edit of the section you want to change, and make the necessary changes. You can’t change aggregations or groupings after creating a job.

    Once the transform job finishes, you can use the _search API operation to search the target index.

    For example, after running a transform job that transforms the flight data based on a DestAirportID field, you can run the following request that returns all of the fields that have a value of .

    Sample Request

    Sample Response