Ingestion

    For most ingestion methods, the Druid MiddleManager processes or the processes load your source data. The sole exception is Hadoop-based ingestion, which uses a Hadoop MapReduce job on YARN.

    This topic introduces streaming and batch ingestion methods. The following topics describe ingestion concepts and information that apply to all ingestion methods:

    • describes rollup as a concept and provides suggestions to maximize the benefits of rollup.
    • Ingestion spec reference provides a reference for the configuration options in the ingestion spec.

    The tables below list Druid’s most common data ingestion methods, along with comparisons to help you choose the best one for your situation. Each ingestion method supports its own set of source systems to pull from. For details about how each method works, as well as configuration properties specific to that method, check out its documentation page.

    Batch

    There are three available options for batch ingestion. Batch ingestion jobs are associated with a controller task that runs for the duration of the job.

    MethodNative batchHadoop-based
    Controller task typeindex_parallelindex_hadoop
    How you submit itSend an index_parallel spec to the .Send an INSERT or statement to the SQL task API.Send an spec to the .
    ParallelismUsing subtasks, if maxNumConcurrentSubTasks is greater than 1.Using query_worker subtasks.Using YARN.
    Fault toleranceWorkers automatically relaunched upon failure. Controller task failure leads to job failure.Controller or worker task failure leads to job failure.YARN containers automatically relaunched upon failure. Controller task failure leads to job failure.
    Can append?Yes.Yes (INSERT).No.
    Can overwrite?Yes.Yes (REPLACE).Yes.
    External dependenciesNone.None.Hadoop cluster.
    Input sourcesAny .Any inputSource (using ) or Druid datasource (using FROM).Any Hadoop FileSystem or Druid datasource.
    Input formatsAny inputFormat.Any .Any Hadoop InputFormat.
    Secondary partitioning optionsDynamic, hash-based, and range-based partitioning methods are available. See partitionsSpec for details.Range partitioning ().Hash-based or range-based partitioning via partitionsSpec.
    Perfect if forceGuaranteedRollup = true in the tuningConfig.Always perfect.Always perfect.