Doris Plugin Framework

    For example, the audit plugin worked after a request execution, it can obtain information related to a request (access user, request IP, SQL, etc…) and write the information into the specified table.

    Differences from UDF:

    • UDF is a function used for data calculation when SQL is executed. Plugin is additional function that is used to extend Doris with customized function, such as support different storage engines and different import ways, and plugin doesn’t participate in data calculation when executing SQL.
    • The execution cycle of UDF is limited to a SQL execution. The execution cycle of plugin may be the same as the Doris process.
    • The usage scene is different. If you need to support special data algorithms when executing SQL, then UDF is recommended, if you need to run custom functions on Doris, or start a background thread to do tasks, then the use of plugin is recommended.

    Currently the plugin framework only supports audit plugins.

    A FE Plugin can be a .zip package or a directory, which contains at least two parts: the plugin.properties and .jar files. The plugin.properties file is used to describe the plugin information.

    The file structure of a Plugin looks like this:

    plugin.properties example:

    1. ### required:
    2. #
    3. # the plugin name
    4. name = audit_plugin_demo
    5. #
    6. # the plugin type
    7. type = AUDIT
    8. #
    9. # simple summary of the plugin
    10. description = just for test
    11. #
    12. # Doris's version, like: 0.11.0
    13. version = 0.11.0
    15. ### FE-Plugin optional:
    16. #
    17. # version of java the code is built against
    18. # use the command "java -version" value, like 1.8.0, 9.0.1, 13.0.4
    19. java.version = 1.8.31
    20. #
    21. # the name of the class to load, fully-qualified.
    22. classname = AuditPluginDemo
    24. ### BE-Plugin optional:
    25. # the name of the so to load
    26. soName = example.so

    fe_plugins is the parent module of the fe plugins. It can uniformly manage the third-party library information that the plugin depends on. Adding a plugin can add a submodule implementation under fe_plugins.

    We can add a submodule in the fe_plugins directory to implement Plugin and create a project:

    1. mvn archetype: generate -DarchetypeCatalog = internal -DgroupId = org.apache -DartifactId = doris-fe-test -DinteractiveMode = false

    The command produces a new mvn project, and a new submodule is automatically added to fe_plugins/pom.xml:

    The new plugin project file structure is as follows:

    1. -doris-fe-test/
    2. -pom.xml
    3. -src/
    4.     ---- main/java/org/apache/
    5.     ------- App.java # mvn auto generate, ignore
    6.     ---- test/java/org/apache

    We will add an assembly folder under main to store plugin.properties and zip.xml. After completion, the file structure is as follows:

    1. -doris-fe-test/
    2. -src/
    3. ---- main/
    4. ------ assembly/
    5. -------- plugin.properties
    6. -------- zip.xml
    7. ------ java/org/apache/
    8. --------App.java # mvn auto generate, ignore

    Add zip.xml

    zip.xml, used to describe the content of the final package of the plugin (.jar file, plugin.properties):

    Then we need to update pom.xml, add doris-fe dependency, and modify maven packaging way:

    1. <?xml version="1.0" encoding="UTF-8"?>
    2. <project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    3.          xmlns="http://maven.apache.org/POM/4.0.0"
    4.          xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    5.     <parent>
    6.         <groupId>org.apache</groupId>
    7.         <artifactId>doris-fe-plugins</artifactId>
    8.         <version>1.0-SNAPSHOT</version>
    9.     </parent>
    10.     <modelVersion>4.0.0</modelVersion>
    12.     <artifactId>auditloader</artifactId>
    13.     <packaging>jar</packaging>
    15.     <dependencies>
    16.         <!-- doris-fe dependencies -->
    17.         <dependency>
    18.             <groupId>org.apache</groupId>
    19.             <artifactId>doris-fe</artifactId>
    20.         </dependency>
    22.         <!-- other dependencies -->
    23.         <dependency>
    24.             ...
    25.         </dependency>
    26.     </dependencies>
    29.     <build>
    30.         <finalName>auditloader</finalName>
    31.         <plugins>
    32.             <plugin>
    33.                 <artifactId>maven-assembly-plugin</artifactId>
    34.                 <version>2.4.1</version>
    35.                 <configuration>
    36.                     <appendAssemblyId>false</appendAssemblyId>
    37.                     <descriptors>
    38.                         <descriptor>src/main/assembly/zip.xml</descriptor>
    39.                     </descriptors>
    40.                 </configuration>
    41.                     <execution>
    42.                         <id>make-assembly</id>
    43.                         <phase>package</phase>
    44.                         <goals>
    45.                             <goal>single</goal>
    46.                         </goals>
    47.                     </execution>
    48.                 </executions>
    49.             </plugin>
    51.     </build>
    53. </project>

    Implement plugin

    Before compiling the plugin, you must first execute sh build.sh --fe of Doris to complete the compilation of Doris FE.

    Finally, execute sh build_plugin.sh in the ${DORIS_HOME} path and you will find the your_plugin_name.zip file in fe_plugins/output

    Or you can execute sh build_plugin.sh --plugin your_plugin_name to only build your plugin.

    Other way

    The easiest way, you can implement your plugin by modifying the example auditdemo

    Doris’s plugin can be deployed in three ways:

    • Http or Https .zip, like http://xxx.xxxxxx.com/data/plugin.zip, Doris will download this .zip file. At the same time, the value of md5sum needs to be set in properties, or an md5 file with the same name as the .zip file needs to be placed, such as http://xxx.xxxxxx.com/data/my_plugin.zip.md5. The content is the MD5 value of the .zip file.
    • Local .zip, like /home/work/data/plugin.zip. If the plug-in is only used for FE, it needs to be deployed in the same directory of all FE nodes. Otherwise, it needs to be deployed on all FE and BE nodes.
    • Local directory, like /home/work/data/plugin, .zip decompressed folder. If the plug-in is only used for FE, it needs to be deployed in the same directory of all FE nodes. Otherwise, it needs to be deployed on all FE and BE nodes.

    Note: Need to ensure that the plugin .zip file is available in the life cycle of doris!

    Install and uninstall the plugin through the install/uninstall statements. More details, see HELP INSTALL PLUGIN; HELP IUNNSTALL PLUGIN; HELP SHOW PLUGINS;

    1. mysql> install plugin from "/home/users/doris/auditloader.zip";
    2. Query OK, 0 rows affected (0.09 sec)
    4. mysql> show plugins\G
    5. *************************** 1. row ***************************
    6.        Name: auditloader
    7.        Type: AUDIT
    8. Description: load audit log to olap load, and user can view the statistic of queries
    9.     Version: 0.12.0
    10. JavaVersion: 1.8.31
    11.   ClassName: AuditLoaderPlugin
    12.      SoName: NULL
    13.     Sources: /home/users/doris/auditloader.zip
    14.      Status: INSTALLED
    15.  Properties: {}
    16. *************************** 2. row ***************************
    17.        Name: AuditLogBuilder
    18.        Type: AUDIT
    19. Description: builtin audit logger
    20.     Version: 0.12.0
    21. JavaVersion: 1.8.31
    22.   ClassName: org.apache.doris.qe.AuditLogBuilder
    23.      SoName: NULL
    24.     Sources: Builtin
    25.      Status: INSTALLED
    26.  Properties: {}   
    27. 2 rows in set (0.00 sec)
    29. mysql> uninstall plugin auditloader;
    30. Query OK, 0 rows affected (0.05 sec)
    32. mysql> show plugins;