InfluxDB shards and shard groups

A shard contains encoded and compressed time series data for a given time range defined by the shard group duration. All points in a within the specified shard group duration are stored in the same shard. A single shard contains multiple series, one or more TSM files on disk, and belongs to a .

A shard group belongs to an InfluxDB bucket and contains time series data for a specific time range defined by the .

In InfluxDB OSS, a shard group typically contains only a single shard. In an InfluxDB Enterprise 1.x cluster, shard groups contain multiple shards distributed across multiple data nodes.

The shard group duration specifies the time range for each shard group and determines how often to create a new shard group. By default, InfluxDB sets the shard group duration according to the retention period of the bucket:

Shard group duration configuration options

To configure a custom bucket shard group duration, use the flag with the influx bucket create and commands.

Shard group durations must be shorter than the bucket’s retention period.

To view your bucket’s shard group duration, use the influx bucket list command.

The following diagram represents a bucket with a 4d retention period and a 1d shard group duration:

0d

2d

3d

4d

Shard group

Shard

Shard group

Shard

Shard group

Shard

Shard

The InfluxDB shard precreation service pre-creates shards with future start and end times for each shard group based on the shard group duration.

The precreator service does not pre-create shards for past time ranges. When backfilling historical data, InfluxDB creates shards for past time ranges as needed, resulting in temporarily lower write throughput.

InfluxDB writes time series data to un-compacted or “hot” shards. When a shard is no longer actively written to, InfluxDB compacts shard data, resulting in a “cold” shard.

Typically, InfluxDB writes data to the most recent shard group, but when backfilling historical data, InfluxDB writes to older shards that must first be un-compacted. When the backfill is complete, InfluxDB re-compacts the older shards.

InfluxDB compacts shards at regular intervals to compress time series data and optimize disk usage. InfluxDB uses the following four compaction levels:

  • Level 1 (L1): InfluxDB flushes all newly written data held in an in-memory cache to disk.
  • Level 3 (L3): InfluxDB iterates over L2-compacted file blocks (over a certain size) and combines multiple blocks containing the same series into one block in a new file.
  • Level 4 (L4): Full compaction—InfluxDB iterates over L3-compacted file blocks and combines multiple blocks containing the same series into one block in a new file.

The InfluxDB retention enforcement service routinely checks for shard groups older than their bucket’s retention period. Once the start time of a shard group is beyond the bucket’s retention period, InfluxDB deletes the shard group and associated shards and TSM files.

In buckets with an infinite retention period, shards remain on disk indefinitely.

InfluxDB only deletes cold shards

InfluxDB only deletes cold shards. If backfilling data beyond a bucket’s retention period, the backfilled data will remain on disk until the following occurs:

  1. The shard returns to a cold state.
  2. The retention enforcement service deletes the shard group.