Metadata storage

    The metadata store includes the following:

    • Segments records
    • Rule records
    • Configuration records
    • Task-related tables

    Derby is the default metadata store for Druid, however, it is not suitable for production. MySQL and are more production suitable metadata stores. See Metadata storage configuration for the default configuration settings.

    Druid supports Derby, MySQL, and PostgreSQL for storing metadata.

    For production clusters, consider using MySQL or PostgreSQL instead of Derby.

    Configure metadata storage with Derby by setting the following properties in your Druid configuration.

    MySQL

    See mysql-metadata-storage extension documentation.

    You can add custom properties to customize the database connection pool (DBCP) for connecting to the metadata store. Define these properties with a prefix. For example:

    Certain properties cannot be set through druid.metadata.storage.connector.dbcp. and must be set with the prefix druid.metadata.storage.connector.:

    • username
    • connectURI
    • validationQuery
    • testOnBorrow

    See for a full list of configurable properties.

    This section describes the various tables in metadata storage.

    Segments table

    This is dictated by the property.

    This table stores metadata about the segments that should be available in the system. (This set of segments is called “used segments” elsewhere in the documentation and throughout the project.) The table is polled by the Coordinator to determine the set of segments that should be available for querying in the system. The table has two main functional columns, the other columns are for indexing purposes.

    Value 1 in the used column means that the segment should be “used” by the cluster (i.e., it should be loaded and available for requests). Value 0 means that the segment should not be loaded into the cluster. We do this as a means of unloading segments from the cluster without actually removing their metadata (which allows for simpler rolling back if that is ever an issue).

    The rule table stores the various rules about where segments should land. These rules are used by the Coordinator when making segment (re-)allocation decisions about the cluster.

    Config table

    The config table stores runtime configuration objects. We do not have many of these yet and we are not sure if we will keep this mechanism going forward, but it is the beginnings of a method of changing some configuration parameters across the cluster at runtime.

    Task-related tables are created and used by the and MiddleManager when managing tasks.

    Audit table

    The audit table stores the audit history for configuration changes such as rule changes done by Coordinator and other config changes.

    Only the following processes access the metadata storage:

    1. Indexing service processes (if any)
    2. Realtime processes (if any)
    3. Coordinator processes

    Thus you need to give permissions (e.g., in AWS security groups) for only these machines to access the metadata storage.

    See the following topics for more information: