Optimize Flux queries

- Avoid processing filters inline
Use set() instead of map() when possible
Measure query performance with Flux profilers

Pushdowns are functions or function combinations that push data operations to the underlying data source rather than operating on data in memory. Start queries with pushdowns to improve query performance. Once a non-pushdown function runs, Flux pulls data into memory and runs all subsequent operations there.

Pushdown functions and function combinations

Most pushdowns are supported when querying an InfluxDB 2.2 or InfluxDB Cloud data source. As shown in the following table, a handful of pushdowns are not supported in InfluxDB 2.2.

* filter() only pushes down when all parameter values are static. See Avoid processing filters inline.

Use pushdown functions and function combinations at the beginning of your query. Once a non-pushdown function runs, Flux pulls data into memory and runs all subsequent operations there.

Pushdown functions in use

Avoid using mathematic operations or string manipulation inline to define data filters. Processing filter values inline prevents from pushing its operation down to the underlying data source, so data returned by the previous function loads into memory. This often results in a significant performance hit.

from(bucket: "example-bucket")
    |> range(start: -1h)                      
    |> filter(fn: (r) => r.region == v.provider + v.region)

To dynamically set filters and maintain the pushdown ability of the filter() function, use variables to define filter values outside of filter():

Avoid short window durations

Windowing (grouping data based on time intervals) is commonly used to aggregate and downsample data. Increase performance by avoiding short window durations. More windows require more compute power to evaluate which window each row should be assigned to. Reasonable window durations depend on the total time range queried.

The following functions use more memory or CPU than others. Consider their necessity in your data processing before using them:

We’re continually optimizing Flux and this list may not represent its current state.

Use set() instead of map() when possible

, experimental.set(), and can each set columns value in data, however set functions have performance advantages over map().

If setting a column value to a predefined, static value, use or experimental.set().

Set a column value to a static value

The following queries are functionally the same, but using set() is more performant than using map().

data
    |> map(fn: (r) => ({ r with foo: "bar" }))
// Recommended
    |> set(key: "foo", value: "bar")

Dynamically set a column value using existing row data

To ensure queries are performant, balance the time range and the precision of your data. For example, if you query data stored every second and request six months worth of data, results would include ≈15.5 million points per series. Depending on the number of series returned after filter()(cardinality), this can quickly become many billions of points. Flux must store these points in memory to generate a response. Use to optimize how many points are stored in memory.

To query data over large periods of time, create a task to downsample data, and then query the downsampled data instead.

Measure query performance with Flux profilers

Use the Flux Profiler package to measure query performance and append performance metrics to your query output. The following Flux profilers are available:

query: provides statistics about the execution of an entire Flux script.
operator: provides statistics about each operation in a query.

Import the profiler package and enable profilers with the profile.enabledProfilers option.

import "profiler"
option profiler.enabledProfilers = ["query", "operator"]

Optimize queries