Data frames

    The data frame structure is a concept that’s borrowed from data analysis tools like the R programming language, and .

    This document gives an overview of the data frame structure, and of how data is handled within Grafana.

    A data frame is a columnar-oriented table structure, which means it stores data by column and not by row. To understand what this means, let’s look at the TypeScript definition used by Grafana:

    In essence, a data frame is a collection of fields, where each field corresponds to a column. Each field, in turn, consists of a collection of values, along with meta information, such as the data type of those values.

    2.   name: string;
    3.   // Prometheus like Labels / Tags
    4.   labels?: Record<string, string>;
    6.   // For example string, number, time (or more specific primitives in the backend)
    7.   type: FieldType;
    8.   values: Vector<T>;
    10.   // Optional display data for the field (e.g. unit, name over-ride, etc)
    11.   config: FieldConfig;
    12. }

    Let’s look an example. The table below demonstrates a data frame with two fields, time and temperature.

    Each field has three values, and each value in a field must share the same type. In this case, all values in the time field are timestamps, and all values in the temperature field are numbers.

    One restriction on data frames is that all fields in the frame must be of the same length to be a valid data frame.

    Each field in a data frame contains optional information about the values in the field, such as units, scaling, and so on.

    Along with the type information, field configs enable data transformations within Grafana.

    A data transformation is any function that accepts a data frame as input, and returns another data frame as output. By using data frames in your plugin, you get a range of transformations for free.

    A data frame with at least one time field is considered a time series.

    For more information on time series, refer to our Introduction to time series.

    Wide format

    When a collection of time series shares the same time index—the time fields in each time series are identical—they can be stored together, in a wide format. By reusing the time field, we can reduce the amount of data being sent to the browser.

    In this example, the cpu usage from each host share the time index, so we can store them in the same data frame.

    However, if the two time series don’t share the same time values, they are represented as two distinct data frames.

    1. Name: cpu
    3. +---------------------+-----------------+
    4. | Name: time          | Name: cpu       |
    5. | Labels:             | Labels: host=a  |
    6. | Type: []time.Time   | Type: []float64 |
    7. +---------------------+-----------------+
    8. | 2020-01-02 03:04:00 | 3               |
    9. | 2020-01-02 03:05:00 | 6               |
    10. +---------------------+-----------------+
    12. Name: cpu
    13. +---------------------+-----------------+
    14. | Name: time          | Name: cpu       |
    15. | Labels:             | Labels: host=b  |
    17. +---------------------+-----------------+
    18. | 2020-01-02 03:04:01 | 4               |
    19. | 2020-01-02 03:05:01 | 7               |
    20. +---------------------+-----------------+

    The wide format can typically be used when multiple time series are collected by the same process. In this case, every measurement is made at the same interval and will therefore share the same time values.

    Some data sources return data in a long format (also called narrow format). This is common format returned by, for example, SQL databases.

    Grafana can detect and convert data frames in long format into wide format.

    For example, the following data frame in long format:

    can be converted into a data frame in wide format:

    1. Name: Wide
    2. Dimensions: 5 fields by 2 rows
    3. +---------------------+------------------+------------------+------------------+------------------+
    4. | Name: time          | Name: aMetric    | Name: bMetric    | Name: aMetric    | Name: bMetric    |
    5. | Labels:             | Labels: host=foo | Labels: host=foo | Labels: host=bar | Labels: host=bar |
    6. | Type: []time.Time   | Type: []float64  | Type: []float64  | Type: []float64  | Type: []float64  |
    7. +---------------------+------------------+------------------+------------------+------------------+
    8. | 2020-01-02 03:04:00 | 2                | 10               | 5                | 15               |
    9. | 2020-01-02 03:05:00 | 3                | 11               | 6                | 16               |

    This section contains links to technical reference and implementations of data frames.

    Apache Arrow

    The data frame structure is inspired by, and uses the . Javascript Data frames use Arrow Tables as the underlying structure, and the backend Go code serializes its Frames in Arrow Tables for transmission.

    The Javascript implementation of data frames is in the and /src/types/dataframe.ts of the .

    Go

    For documentation on the Go implementation of data frames, refer to the .