Tutorial: Roll-up

    This tutorial will demonstrate the effects of roll-up on an example dataset.

    For this tutorial, we’ll assume you’ve already downloaded Druid as described in the single-machine quickstart and have it running on your local machine.

    It will also be helpful to have finished and Tutorial: Querying data.

    For this tutorial, we’ll use a small sample of network flow event data, representing packet and byte counts for traffic from a source to a destination IP address that occurred within a particular second.

    A file containing this sample input data is located at .

    1. {
    2.   "type" : "index_parallel",
    3.   "spec" : {
    4.     "dataSchema" : {
    5.       "dataSource" : "rollup-tutorial",
    6.       "dimensionsSpec" : {
    7.         "dimensions" : [
    8.           "srcIP",
    9.           "dstIP"
    10.         ]
    11.       },
    12.       "timestampSpec": {
    13.         "column": "timestamp",
    14.         "format": "iso"
    15.       },
    16.       "metricsSpec" : [
    17.         { "type" : "count", "name" : "count" },
    18.         { "type" : "longSum", "name" : "packets", "fieldName" : "packets" },
    19.       ],
    20.       "granularitySpec" : {
    21.         "type" : "uniform",
    22.         "segmentGranularity" : "week",
    23.         "queryGranularity" : "minute",
    24.         "intervals" : ["2018-01-01/2018-01-03"],
    25.         "rollup" : true
    27.     },
    28.     "ioConfig" : {
    29.       "type" : "index_parallel",
    30.       "inputSource" : {
    31.         "type" : "local",
    32.         "baseDir" : "quickstart/tutorial",
    33.         "filter" : "rollup-data.json"
    34.       },
    35.       "inputFormat" : {
    36.         "type" : "json"
    37.       },
    38.       "appendToExisting" : false
    39.     },
    40.       "type" : "index_parallel",
    41.       "partitionsSpec": {
    42.         "type": "dynamic"
    43.       },
    44.       "maxRowsInMemory" : 25000
    45.     }
    46.   }
    47. }

    Roll-up has been enabled by setting "rollup" : true in the granularitySpec.

    Note that we have srcIP and dstIP defined as dimensions, a longSum metric is defined for the packets and columns, and the queryGranularity has been defined as minute.

    We will see how these definitions are used after we load this data.

    From the apache-druid-24.0.2 package root, run the following command:

    1. bin/post-index-task --file quickstart/tutorial/rollup-index.json --url http://localhost:8081

    After the script completes, we will query the data.

    Let’s look at the three events in the original input data that occurred during 2018-01-01T01:01:

    1. {"timestamp":"2018-01-01T01:01:35Z","srcIP":"1.1.1.1", "dstIP":"2.2.2.2","packets":20,"bytes":9024}
    2. {"timestamp":"2018-01-01T01:01:51Z","srcIP":"1.1.1.1", "dstIP":"2.2.2.2","packets":255,"bytes":21133}
    3. {"timestamp":"2018-01-01T01:01:59Z","srcIP":"1.1.1.1", "dstIP":"2.2.2.2","packets":11,"bytes":5780}

    These three rows have been “rolled up” into the following row:

    1. ┌──────────────────────────┬────────┬───────┬─────────┬─────────┬─────────┐
    2. │ __time                   │ bytes  │ count │ dstIP   │ packets │ srcIP   │
    3. ├──────────────────────────┼────────┼───────┼─────────┼─────────┼─────────┤
    4. │ 2018-01-01T01:01:00.000Z │  35937 │     3 │ 2.2.2.2 │     286 │ 1.1.1.1 │
    5. └──────────────────────────┴────────┴───────┴─────────┴─────────┴─────────┘

    The input rows have been grouped by the timestamp and dimension columns {timestamp, srcIP, dstIP} with sum aggregations on the metric columns packets and bytes.

    Before the grouping occurs, the timestamps of the original input data are bucketed/floored by minute, due to the "queryGranularity":"minute" setting in the ingestion spec.

    Likewise, these two events that occurred during 2018-01-01T01:02 have been rolled up:

    1. ┌──────────────────────────┬────────┬───────┬─────────┬─────────┬─────────┐
    2. │ __time                   │ bytes  │ count │ dstIP   │ packets │ srcIP   │
    3. ├──────────────────────────┼────────┼───────┼─────────┼─────────┼─────────┤
    4. │ 2018-01-01T01:02:00.000Z │ 366260 │     2 │ 2.2.2.2 │     415 │ 1.1.1.1 │
    1. {"timestamp":"2018-01-01T01:03:29Z","srcIP":"1.1.1.1", "dstIP":"2.2.2.2","packets":49,"bytes":10204}

    Note that the count metric shows how many rows in the original input data contributed to the final “rolled up” row.