- Druid stores data in segments. If you’re using the mode, increasing the segment size might introduce further aggregation which reduces the dataSource size.
It would be best if you can optimize the segment size at ingestion time, but sometimes it’s not easy especially when it comes to stream ingestion because the amount of data ingested might vary over time. In this case, you can create segments with a sub-optimized size first and optimize them later using compaction.
- Number of rows per segment: it’s generally recommended for each segment to have around 5 million rows. This setting is usually more important than the below “segment byte size”. This is because Druid uses a single thread to process each segment, and thus this setting can directly control how many rows each thread processes, which in turn means how well the query execution is parallelized.
Please note that the query result might include overshadowed segments. In this case, you may want to see only rows of the max version per interval (pair of and ).
- Turning on the . The Coordinator periodically submits compaction tasks to re-index small segments. To enable the automatic compaction, you need to configure it for each dataSource via Coordinator’s dynamic configuration. See and Compaction Configuration for details.
For an overview of compaction and how to submit a manual compaction task, see .