GROUPING SETS DESIGN

    The three GROUPING functions help you identify the group each row belongs to and enable sorting subtotal rows and filtering results.

    GROUPING SETS syntax lets you define multiple groupings in the same query. GROUP BY computes all the groupings specified and combines them with UNION ALL. For example, consider the following statement:

    This statement is equivalent to:

    1. SELECT k1, k2, SUM( k3 ) FROM t GROUP BY k1, k2
    2. UNION
    3. SELECT k1, null, SUM( k3 ) FROM t GROUP BY k1
    4. UNION
    5. SELECT null, k2, SUM( k3 ) FROM t GROUP BY k2
    6. UNION
    7. SELECT null, null, SUM( k3 ) FROM t

    This is an example of real query:

    1. mysql> SELECT * FROM t;
    2. +------+------+------+
    3. | k1   | k2   | k3   |
    4. +------+------+------+
    5. | a    | A    |    1 |
    6. | a    | A    |    2 |
    7. | a    | B    |    1 |
    8. | a    | B    |    3 |
    9. | b    | A    |    1 |
    10. | b    | A    |    4 |
    11. | b    | B    |    1 |
    12. | b    | B    |    5 |
    13. +------+------+------+
    14. 8 rows in set (0.01 sec)
    16. mysql> SELECT k1, k2, SUM(k3) FROM t GROUP BY GROUPING SETS ( (k1, k2), (k2), (k1), ( ) );
    17. +------+------+-----------+
    18. | k1   | k2   | sum(`k3`) |
    19. +------+------+-----------+
    20. | b    | B    |         6 |
    21. | a    | B    |         4 |
    22. | a    | A    |         3 |
    23. | b    | A    |         5 |
    24. | NULL | B    |        10 |
    25. | NULL | A    |         8 |
    26. | a    | NULL |         7 |
    27. | b    | NULL |        11 |
    28. | NULL | NULL |        18 |
    29. +------+------+-----------+
    30. 9 rows in set (0.06 sec)

    1.2 ROLLUP Syntax

    ROLLUP enables a SELECT statement to calculate multiple levels of subtotals across a specified group of dimensions. It also calculates a grand total. ROLLUP is a simple extension to the GROUP BY clause, so its syntax is extremely easy to use. The ROLLUP extension is highly efficient, adding minimal overhead to a query.

    ROLLUP appears in the GROUP BY clause in a SELECT statement. Its form is:

    1. SELECT a, b,c, SUM( d ) FROM tab1 GROUP BY ROLLUP(a,b,c)

    This statement is equivalent to GROUPING SETS as followed:

    1. GROUPING SETS (
    2. (a,b,c),
    3. ( a, b ),
    4. ( a),
    5. ( )
    6. )

    1.3 CUBE Syntax

    Like ROLLUP CUBE generates all the subtotals that could be calculated for a data cube with the specified dimensions.

    1. SELECT a, b,c, SUM( d ) FROM tab1 GROUP BY CUBE(a,b,c)

    e.g. CUBE ( a, b, c ) is equivalent to GROUPING SETS as followed:

    1. GROUPING SETS (
    2. ( a, b, c ),
    3. ( a, b ),
    4. ( a,    c ),
    5. ( a       ),
    7. (    b    ),
    8. (       c ),
    9. (         )
    10. )

    1.4 GROUPING and GROUPING_ID Function

    Indicates whether a specified column expression in a GROUP BY list is aggregated or not. GROUPINGreturns 1 for aggregated or 0 for not aggregated in the result set. GROUPING can be used only in the SELECT list, HAVING, and ORDER BY clauses when GROUP BY is specified.

    GROUPING_ID describes which of a list of expressions are grouped in a row produced by a GROUP BY query. The GROUPING_ID function simply returns the decimal equivalent of the binary value formed as a result of the concatenation of the values returned by the GROUPING functions.

    Each GROUPING_ID argument must be an element of the GROUP BY list. GROUPING_ID () returns an integer bitmap whose lowest N bits may be lit. A lit bit indicates the corresponding argument is not a grouping column for the given output row. The lowest-order bit corresponds to argument N, and the N-1th lowest-order bit corresponds to argument 1. If the column is a grouping column the bit is 0 else is 1.

    1. mysql> select * from t;
    2. +------+------+------+
    3. | k1   | k2   | k3   |
    4. +------+------+------+
    5. | a    | A    |    1 |
    6. | a    | A    |    2 |
    7. | a    | B    |    1 |
    8. | b    | A    |    1 |
    9. | b    | A    |    4 |
    10. | b    | B    |    1 |
    11. | b    | B    |    5 |
    12. +------+------+------+

    grouping sets result:

    First of all, a GROUP BY clause is essentially a special case of GROUPING SETS, for example:

    1.    GROUP BY a
    2. is equivalent to:
    3.    GROUP BY GROUPING SETS((a))
    4. also,
    5.    GROUP BY a,b,c
    6. is equivalent to:
    7.    GROUP BY GROUPING SETS((a,b,c))

    Similarly, CUBE and ROLLUP can be expanded into GROUPING SETS, so the various combinations and nesting of GROUP BY, CUBE, ROLLUP, GROUPING SETS are essentially the combination and nesting of GROUPING SETS.

    For GROUPING SETS nesting, it is semantically equivalent to writing the statements inside the nest directly outside. (ref:) mentions:

    1. The CUBE and ROLLUP constructs can be used either directly in the GROUP BY clause, or nested inside a GROUPING SETS clause. If one GROUPING SETS clause is nested inside another, the effect is the same as if all the elements of the inner clause had been written directly in the outer clause.

    For a combined list of multiple GROUPING SETS, many databases consider it a cross product relationship.

    for example:

    1. GROUP BY a, CUBE (b, c), GROUPING SETS ((d), (e))
    3. is equivalent to:
    5. GROUP BY GROUPING SETS (
    6. (a, b, c, d), (a, b, c, e),
    7. (a, b, d),    (a, b, e),
    8. (a, c, d),    (a, c, e),
    9. (a, d),       (a, e)
    10. )

    For the combination and nesting of GROUPING SETS, each database support is not the same. For example snowflake does not support any combination and nesting. （https://docs.snowflake.net/manuals/sql-reference/constructs/group-by.htmlGROUPING SETS DESIGN - 图2 (opens new window)）

    Oracle supports both composition and nesting. （）

    Presto supports composition, but not nesting. （https://prestodb.github.io/docs/current/sql/select.htmlGROUPING SETS DESIGN - 图4 (opens new window)）

    Support GROUPING SETS, ROLLUP and CUBE syntax, implements 1.1, 1.2, 1.3 1.4, 1.5, not support the combination and nesting of GROUPING SETS in current version.

    2.1 GROUPING SETS Syntax

    1. SELECT ...
    2. FROM ...
    3. [ ... ]
    4. GROUP BY GROUPING SETS ( groupSet [ , groupSet [ , ... ] ] )
    5. [ ... ]
    7. groupSet ::= { ( expr  [ , expr [ , ... ] ] )}
    9. <expr>
    10. Expression, column name.

    2.2 ROLLUP Syntax

    1. SELECT ...
    2. FROM ...
    3. [ ... ]
    4. GROUP BY ROLLUP ( expr  [ , expr [ , ... ] ] )
    5. [ ... ]
    7. <expr>
    8. Expression, column name.

    2.3 CUBE Syntax

    1. SELECT ...
    2. FROM ...
    3. [ ... ]
    4. GROUP BY CUBE ( expr  [ , expr [ , ... ] ] )
    5. [ ... ]
    7. <expr>
    8. Expression, column name.

    For GROUPING SET is equivalent to the UNION of GROUP BY . So we can expand input rows, and run an GROUP BY on these rows.

    For example:

    1. SELECT a, b FROM src GROUP BY a, b GROUPING SETS ((a, b), (a), (b), ());

    Base on GROUPING SETS , we can expend the input to:

    1. 1, 2       (GROUPING_ID: a, b -> 00 -> 0)
    2. 1, null    (GROUPING_ID: a, null -> 01 -> 1)
    3. null, 2    (GROUPING_ID: null, b -> 10 -> 2)
    4. null, null (GROUPING_ID: null, null -> 11 -> 3)
    6. 3, 4       (GROUPING_ID: a, b -> 00 -> 0)
    7. 3, null    (GROUPING_ID: a, null -> 01 -> 1)
    8. null, 4    (GROUPING_ID: null, b -> 10 -> 2)
    9. null, null (GROUPING_ID: null, null -> 11 -> 3)

    And then use those row as input, then GROUP BY a, b, GROUPING_ID

    3.2 Example

    Table t:

    1. mysql> select * from t;
    3. | k1   | k2   | k3   |
    4. +------+------+------+
    5. | a    | A    |    1 |
    6. | a    | A    |    2 |
    7. | a    | B    |    1 |
    8. | a    | B    |    3 |
    9. | b    | A    |    1 |
    10. | b    | A    |    4 |
    11. | b    | B    |    1 |
    12. | b    | B    |    5 |
    13. +------+------+------+
    14. 8 rows in set (0.01 sec)

    for the query:

    1. SELECT k1, k2, GROUPING_ID(k1,k2), SUM(k3) FROM t GROUP BY GROUPING SETS ((k1, k2), (k1), (k2), ());

    First, expand the input, every row expand into 4 rows ( the size of GROUPING SETS), and insert GROUPING_ID column

    e.g. a, A, 1 expanded to:

    1. +------+------+------+-------------------------+
    2. | k1   | k2   | k3   | GROUPING_ID(`k1`, `k2`) |
    3. +------+------+------+-------------------------+
    4. | a    | A    |    1 |                       0 |
    5. | a    | NULL |    1 |                       1 |
    6. | NULL | A    |    1 |                       2 |
    7. | NULL | NULL |    1 |                       3 |

    Finally, all rows expended as follows (32 rows):

    1. +------+------+------+-------------------------+
    2. | k1   | k2   | k3   | GROUPING_ID(`k1`, `k2`) |
    3. +------+------+------+-------------------------+
    4. | a    | A    |    1 |                       0 |
    5. | a    | A    |    2 |                       0 |
    6. | a    | B    |    1 |                       0 |
    7. | a    | B    |    3 |                       0 |
    8. | b    | A    |    1 |                       0 |
    9. | b    | A    |    4 |                       0 |
    10. | b    | B    |    1 |                       0 |
    11. | b    | B    |    5 |                       0 |
    12. | a    | NULL |    1 |                       1 |
    13. | a    | NULL |    1 |                       1 |
    14. | a    | NULL |    2 |                       1 |
    15. | a    | NULL |    3 |                       1 |
    16. | b    | NULL |    1 |                       1 |
    17. | b    | NULL |    1 |                       1 |
    18. | b    | NULL |    4 |                       1 |
    19. | b    | NULL |    5 |                       1 |
    20. | NULL | A    |    1 |                       2 |
    21. | NULL | A    |    1 |                       2 |
    22. | NULL | A    |    2 |                       2 |
    23. | NULL | A    |    4 |                       2 |
    24. | NULL | B    |    1 |                       2 |
    25. | NULL | B    |    1 |                       2 |
    26. | NULL | B    |    3 |                       2 |
    27. | NULL | B    |    5 |                       2 |
    28. | NULL | NULL |    1 |                       3 |
    29. | NULL | NULL |    1 |                       3 |
    30. | NULL | NULL |    1 |                       3 |
    31. | NULL | NULL |    1 |                       3 |
    32. | NULL | NULL |    2 |                       3 |
    33. | NULL | NULL |    3 |                       3 |
    34. | NULL | NULL |    4 |                       3 |
    35. | NULL | NULL |    5 |                       3 |
    36. +------+------+------+-------------------------+
    37. 32 rows in set.

    now GROUP BY k1, k2, GROUPING_ID(k1,k2):

    1. +------+------+-------------------------+-----------+
    2. | k1   | k2   | grouping_id(`k1`, `k2`) | sum(`k3`) |
    3. +------+------+-------------------------+-----------+
    4. | a    | A    |                       0 |         3 |
    5. | a    | B    |                       0 |         4 |
    6. | a    | NULL |                       1 |         7 |
    7. | b    | A    |                       0 |         5 |
    8. | b    | B    |                       0 |         6 |
    9. | b    | NULL |                       1 |        11 |
    10. | NULL | A    |                       2 |         8 |
    11. | NULL | B    |                       2 |        10 |
    12. | NULL | NULL |                       3 |        18 |
    13. +------+------+-------------------------+-----------+
    14. 9 rows in set (0.02 sec)

    The result is equivalent to the UNION ALL

    1. select k1, k2, sum(k3) from t group by k1, k2
    2. UNION ALL
    3. select NULL, k2, sum(k3) from t group by k2
    4. UNION ALL
    5. select k1, NULL, sum(k3) from t group by k1
    6. UNION ALL
    7. select NULL, NULL, sum(k3) from t;
    9. +------+------+-----------+
    10. | k1   | k2   | sum(`k3`) |
    11. +------+------+-----------+
    12. | b    | B    |         6 |
    13. | b    | A    |         5 |
    14. | a    | A    |         3 |
    15. | a    | B    |         4 |
    16. | a    | NULL |         7 |
    17. | b    | NULL |        11 |
    18. | NULL | B    |        10 |
    19. | NULL | A    |         8 |
    20. | NULL | NULL |        18 |
    22. 9 rows in set (0.06 sec)

    3.3 FE

    3.3.1 Tasks

    1. Add GroupByClause, replace groupingExprs.
    2. Add Grouping Sets, Cube and RollUp syntax.
    3. Add GroupByClause in SelectStmt.
    4. Add GroupingFunctionCallExpr, implements grouping grouping_id function call
    5. Add VirtualSlot, generate the map of virtual slots and real slots
    6. add virtual column GROUPING_ID and other virtual columns generated by grouping and grouping_id, insert into groupingExprs,
    7. Add a PlanNode, name as RepeatNode. For GroupingSets aggregation insert RepeatNode to the plan.

    3.3.2 Tuple

    In order to add GROUPING_ID to groupingExprs in GroupByClause, need to create virtual SlotRef, also, need tot create a tuple for this slot, named GROUPING__ID Tuple.

    For the plannode RepeatNode, its input are all the tuples of its children and its output tuple are the repeat data and GROUPING_ID.

    3.3.3 Expression and Function Substitution

    expr -> if(bitand(pos, grouping_id)=0, expr, null) for expr in extension grouping clause grouping_id() -> grouping_id(grouping_id) for grouping_id function

    3.4 BE

    3.4.1 Tasks

    1. Add RepeatNode executor, expend the input data and append GROUPING_ID to every row