Exchange Materialization

As with other MPP databases, Presto leverages RPC shuffle to achieve efficient and low-latency query execution for join and aggregation. However, RPC shuffle also requires all the producers and consumers to be executed concurrently until the query is finished.

To illustrates this, consider the aggregation query:

With exchange materialization, the intermediate shuffle data is written to disk (currently, it is always a temporary Hive bucketed table). This opens the opportunity for flexible scheduling policies on the aggregation side, as only a subset of aggregation data needs to be held in memory at the same time – this execution strategy is called “grouped execution” in Presto.

Using Exchange Materialization

Exchange materialization can be enabled on per-query basis by setting the following 3 session properties: , partitioning_provider_catalog and hash_partition_count:

-- Set partitioning_provider_catalog to the Hive connector catalog
SET SESSION hash_partition_count = 4096;

To make it easy for user to use exchange materialization, the admin can leverage to set the session properties automatically based on client tags. The example in Session Property Managers demonstrates how to automatically enable exchange materialization for queries with high_mem_etl tag.