Apache Druid vs SQL-on-Hadoop

    For a direct comparison between the technologies and when to only use one or the other, things basically comes down to your product requirements and what the systems were designed to do.

    Druid was designed to

    1. be an always on service
    2. handle slice-n-dice style ad-hoc queries

    SQL-on-Hadoop engines generally sidestep Map/Reduce, instead querying data directly from HDFS or, in some cases, other storage systems. Some of these engines (including Impala and Presto) can be co-located with HDFS data nodes and coordinate with them to achieve data locality for queries. What does this mean? We can talk about it in terms of three general areas

    1. Queries
    2. Query Flexibility

    Most SQL-on-Hadoop engines are responsible for query planning and execution for underlying storage layers and storage formats. They are processes that stay on even if there is no query running (eliminating the JVM startup costs from Hadoop MapReduce). Some (Impala/Presto) SQL-on-Hadoop engines have daemon processes that can be run where the data is stored, virtually eliminating network transfer costs. There is still some latency overhead (e.g. serialization/deserialization time) associated with pulling data from the underlying storage layer into the computation layer. We are unaware of exactly how much of a performance impact this makes.

    Druid is built to allow for real-time ingestion of data. You can ingest data and query it immediately upon ingestion, the latency between how quickly the event is reflected in the data is dominated by how long it takes to deliver the event to Druid.

    SQL-on-Hadoop, being based on data in HDFS or some other backing store, are limited in their data ingestion rates by the rate at which that backing store can make data available. Generally, the backing store is the biggest bottleneck for how quickly data can become available.

    SQL-on-Hadoop support SQL style queries with full joins.

    Parquet is a column storage format that is designed to work with SQL-on-Hadoop engines. Parquet doesn’t have a query execution engine, and instead relies on external sources to pull data out of it.

    Druid’s storage format is highly optimized for linear scans. Although Druid has support for nested data, Parquet’s storage format is much more hierarchical, and is more designed for binary chunking. In theory, this should lead to faster scans in Druid.