Ingestion

For most ingestion methods, the Druid MiddleManager processes or the processes load your source data. One exception is Hadoop-based ingestion, which uses a Hadoop MapReduce job on YARN MiddleManager or Indexer processes to start and monitor Hadoop jobs.

During ingestion Druid creates segments and stores them in deep storage. Historical nodes load the segments into memory to respond to queries. For streaming ingestion, the Middle Managers and indexers can respond to queries in real-time with arriving data. See the section of the Druid design documentation for more information.

This topic introduces streaming and batch ingestion methods. The following topics describe ingestion concepts and information that apply to all ingestion methods:

• describes rollup as a concept and provides suggestions to maximize the benefits of rollup.
• Ingestion spec reference provides a reference for the configuration options in the ingestion spec.

The tables below list Druid’s most common data ingestion methods, along with comparisons to help you choose the best one for your situation. Each ingestion method supports its own set of source systems to pull from. For details about how each method works, as well as configuration properties specific to that method, check out its documentation page.

The most recommended, and most popular, method of streaming ingestion is the that reads directly from Kafka. Alternatively, the Kinesis indexing service works with Amazon Kinesis Data Streams.

Streaming ingestion uses an ongoing process called a supervisor that reads from the data stream to ingest data into Druid.

Batch

When doing batch loads from files, you should use one-time , and you have three options: index_parallel (native batch; parallel), index_hadoop (Hadoop-based), or index (native batch; single-task).

In general, we recommend native batch whenever it meets your needs, since the setup is simpler (it does not depend on an external Hadoop cluster). However, there are still scenarios where Hadoop-based batch ingestion might be a better choice, for example when you already have a running Hadoop cluster and want to use the cluster resource of the existing cluster for batch ingestion.

This table compares the three available options: