Modules

    For example, users can define their own geo functions and plug them into Flink as built-in functions to be used in Flink SQL and Table APIs. Another example is users can load an out-of-shelf Hive module to use Hive built-in functions as Flink built-in functions.

    Furthermore, a module can provide built-in table source and sink factories which disable Flink’s default discovery mechanism based on Java’s Service Provider Interfaces (SPI), or influence how connectors of temporary tables should be created without a corresponding catalog.

    contains all of Flink’s system (built-in) functions and is loaded and enabled by default.

    The HiveModule provides Hive built-in functions as Flink’s system functions to SQL and Table API users. Flink’s provides full details on setting up the module.

    Users can develop custom modules by implementing the Module interface. To use custom modules in SQL CLI, users should develop both a module and its corresponding module factory by implementing the ModuleFactory interface.

    A module factory defines a set of properties for configuring the module when the SQL CLI bootstraps. Properties are passed to a discovery service where the service tries to match the properties to a ModuleFactory and instantiate a corresponding module instance.

    A module can be loaded, enabled, disabled and unloaded. When TableEnvironment loads a module initially, it enables the module by default. Flink supports multiple modules and keeps track of the loading order to resolve metadata. Besides, Flink only resolves the functions among enabled modules. E.g., when there are two functions of the same name residing in two modules, there will be three conditions.

    • If both of the modules are enabled, then Flink resolves the function according to the resolution order of the modules.
    • If one of them is disabled, then Flink resolves the function to the enabled module.
    • If both of the modules are disabled, then Flink cannot resolve the function.

    Besides, users can also disable modules by not declaring them. E.g., users can specify Flink to disable core module by USE MODULES hive (However, it is strongly not recommended disabling core module). Disable a module does not unload it, and users can enable it again by using it. E.g., users can bring back core module and place it in the first by USE MODULES core, hive. A module can be enabled only when it is loaded already. Using an unloaded module will throw an Exception. Eventually, users can unload a module.

    The difference between disabling and unloading a module is that TableEnvironment still keeps the disabled modules, and users can list all loaded modules to view the disabled modules.

    Objects provided by modules are considered part of Flink’s system (built-in) objects; thus, they don’t have any namespaces.

    Users can use SQL to load/unload/use/list modules in both Table API and SQL CLI.

    Java

    Scala

    1. val settings = EnvironmentSettings.inStreamingMode()
    2. val tableEnv = TableEnvironment.create(setting)
    4. // Show initially loaded and enabled modules
    5. tableEnv.executeSql("SHOW MODULES").print()
    6. // +-------------+
    7. // | module name |
    8. // +-------------+
    9. // |        core |
    10. // +-------------+
    11. tableEnv.executeSql("SHOW FULL MODULES").print()
    12. // +-------------+------+
    13. // | module name | used |
    14. // +-------------+------+
    15. // |        core | true |
    16. // +-------------+------+
    18. // Load a hive module
    19. tableEnv.executeSql("LOAD MODULE hive WITH ('hive-version' = '...')")
    21. // Show all enabled modules
    22. tableEnv.executeSql("SHOW MODULES").print()
    23. // +-------------+
    24. // | module name |
    25. // +-------------+
    26. // |        core |
    27. // |        hive |
    28. // +-------------+
    30. // Show all loaded modules with both name and use status
    31. tableEnv.executeSql("SHOW FULL MODULES")
    32. // +-------------+------+
    33. // | module name | used |
    34. // +-------------+------+
    35. // |        core | true |
    36. // |        hive | true |
    37. // +-------------+------+
    39. // Change resolution order
    40. tableEnv.executeSql("USE MODULES hive, core")
    41. tableEnv.executeSql("SHOW MODULES").print()
    42. // +-------------+
    43. // | module name |
    44. // +-------------+
    45. // |        hive |
    46. // |        core |
    47. // +-------------+
    48. tableEnv.executeSql("SHOW FULL MODULES").print()
    49. // +-------------+------+
    50. // | module name | used |
    51. // +-------------+------+
    52. // |        hive | true |
    53. // |        core | true |
    54. // +-------------+------+
    56. // Disable core module
    57. tableEnv.executeSql("USE MODULES hive")
    58. tableEnv.executeSql("SHOW MODULES").print()
    59. // +-------------+
    60. // | module name |
    61. // +-------------+
    62. // |        hive |
    63. // +-------------+
    64. tableEnv.executeSql("SHOW FULL MODULES").print()
    65. // +-------------+-------+
    66. // | module name |  used |
    67. // +-------------+-------+
    68. // |        hive |  true |
    69. // |        core | false |
    70. // +-------------+-------+
    72. // Unload hive module
    73. tableEnv.executeSql("UNLOAD MODULE hive")
    74. tableEnv.executeSql("SHOW MODULES").print()
    75. // Empty set
    76. tableEnv.executeSql("SHOW FULL MODULES").print()
    77. // +-------------+-------+
    78. // | module name |  used |
    79. // +-------------+-------+
    80. // |        hive | false |
    81. // +-------------+-------+

    Python

    1. -- Show initially loaded and enabled modules
    2. Flink SQL> SHOW MODULES;
    3. +-------------+
    4. | module name |
    5. +-------------+
    6. |        core |
    7. +-------------+
    8. 1 row in set
    9. Flink SQL> SHOW FULL MODULES;
    10. +-------------+------+
    11. | module name | used |
    12. +-------------+------+
    13. |        core | true |
    14. +-------------+------+
    15. 1 row in set
    16. -- Load a hive module
    17. Flink SQL> LOAD MODULE hive WITH ('hive-version' = '...');
    20. Flink SQL> SHOW MODULES;
    21. +-------------+
    22. | module name |
    23. +-------------+
    24. |        core |
    25. |        hive |
    26. +-------------+
    27. 2 rows in set
    29. -- Show all loaded modules with both name and use status
    30. Flink SQL> SHOW FULL MODULES;
    31. +-------------+------+
    32. | module name | used |
    33. +-------------+------+
    34. |        core | true |
    35. |        hive | true |
    36. +-------------+------+
    37. 2 rows in set
    39. -- Change resolution order
    40. Flink SQL> USE MODULES hive, core ;
    41. Flink SQL> SHOW MODULES;
    42. +-------------+
    43. | module name |
    44. +-------------+
    45. |        hive |
    46. |        core |
    47. +-------------+
    48. 2 rows in set
    49. Flink SQL> SHOW FULL MODULES;
    50. +-------------+------+
    51. | module name | used |
    52. +-------------+------+
    53. |        hive | true |
    54. |        core | true |
    55. +-------------+------+
    56. 2 rows in set
    58. -- Unload hive module
    59. Flink SQL> UNLOAD MODULE hive;
    60. Flink SQL> SHOW MODULES;
    61. Empty set
    62. Flink SQL> SHOW FULL MODULES;
    63. +-------------+-------+
    64. | module name |  used |
    65. +-------------+-------+
    66. |        hive | false |
    67. +-------------+-------+
    68. 1 row in set

    YAML

    All modules defined using YAML must provide a type property that specifies the type. The following types are supported out of the box.

    Users can use Java, Scala or Python to load/unload/use/list modules programmatically.

    Java

    1. EnvironmentSettings settings = EnvironmentSettings.inStreamingMode();
    2. TableEnvironment tableEnv = TableEnvironment.create(settings);
    4. // Show initially loaded and enabled modules
    5. tableEnv.listModules();
    6. // +-------------+
    7. // | module name |
    8. // +-------------+
    9. // |        core |
    10. // +-------------+
    11. tableEnv.listFullModules();
    12. // +-------------+------+
    13. // | module name | used |
    14. // +-------------+------+
    15. // |        core | true |
    16. // +-------------+------+
    18. // Load a hive module
    19. tableEnv.loadModule("hive", new HiveModule());
    21. // Show all enabled modules
    22. tableEnv.listModules();
    23. // +-------------+
    24. // | module name |
    25. // +-------------+
    26. // |        core |
    27. // |        hive |
    28. // +-------------+
    30. // Show all loaded modules with both name and use status
    31. tableEnv.listFullModules();
    32. // +-------------+------+
    33. // | module name | used |
    34. // +-------------+------+
    35. // |        core | true |
    36. // |        hive | true |
    37. // +-------------+------+
    39. // Change resolution order
    40. tableEnv.useModules("hive", "core");
    41. tableEnv.listModules();
    42. // +-------------+
    43. // | module name |
    44. // +-------------+
    45. // |        hive |
    46. // |        core |
    47. // +-------------+
    48. tableEnv.listFullModules();
    49. // +-------------+------+
    50. // | module name | used |
    51. // +-------------+------+
    52. // |        hive | true |
    53. // |        core | true |
    54. // +-------------+------+
    56. tableEnv.useModules("hive");
    58. // +-------------+
    59. // | module name |
    60. // +-------------+
    61. // |        hive |
    62. // +-------------+
    63. tableEnv.listFullModules();
    64. // +-------------+-------+
    65. // | module name |  used |
    66. // +-------------+-------+
    67. // |        hive |  true |
    68. // |        core | false |
    69. // +-------------+-------+
    71. // Unload hive module
    72. tableEnv.unloadModule("hive");
    73. tableEnv.listModules();
    74. // Empty set
    75. tableEnv.listFullModules();
    76. // +-------------+-------+
    77. // | module name |  used |
    78. // +-------------+-------+
    79. // |        hive | false |
    80. // +-------------+-------+

    Scala

    Python

    1. from pyflink.table import *
    3. # environment configuration
    4. settings = EnvironmentSettings.inStreamingMode()
    5. t_env = TableEnvironment.create(settings)
    7. # Show initially loaded and enabled modules
    8. t_env.list_modules()
    9. # +-------------+
    10. # | module name |
    11. # +-------------+
    12. # |        core |
    13. # +-------------+
    14. t_env.list_full_modules()
    15. # +-------------+------+
    16. # | module name | used |
    17. # +-------------+------+
    18. # |        core | true |
    19. # +-------------+------+
    21. # Load a hive module
    22. t_env.load_module("hive", HiveModule())
    24. # Show all enabled modules
    25. t_env.list_modules()
    26. # +-------------+
    27. # | module name |
    28. # +-------------+
    29. # |        core |
    30. # |        hive |
    31. # +-------------+
    33. # Show all loaded modules with both name and use status
    34. t_env.list_full_modules()
    35. # +-------------+------+
    36. # | module name | used |
    37. # +-------------+------+
    38. # |        core | true |
    39. # |        hive | true |
    40. # +-------------+------+
    42. # Change resolution order
    43. t_env.use_modules("hive", "core")
    44. t_env.list_modules()
    45. # +-------------+
    46. # | module name |
    47. # +-------------+
    48. # |        hive |
    49. # |        core |
    50. # +-------------+
    51. t_env.list_full_modules()
    52. # +-------------+------+
    53. # | module name | used |
    54. # +-------------+------+
    55. # |        hive | true |
    56. # |        core | true |
    57. # +-------------+------+
    59. # Disable core module
    60. t_env.use_modules("hive")
    61. t_env.list_modules()
    62. # +-------------+
    63. # | module name |
    64. # +-------------+
    65. # |        hive |
    66. # +-------------+
    67. t_env.list_full_modules()
    68. # +-------------+-------+
    69. # | module name |  used |
    70. # +-------------+-------+
    71. # |        hive |  true |
    72. # |        core | false |
    73. # +-------------+-------+
    75. # Unload hive module
    76. t_env.unload_module("hive")
    77. t_env.list_modules()
    78. # Empty set
    79. t_env.list_full_modules()
    80. # +-------------+-------+
    81. # | module name |  used |
    82. # +-------------+-------+
    83. # |        hive | false |