Pipeline Engine

    Note: before compiling the engine, you need to compile the linkis project in full Currently, the pipeline engine needs to be installed and deployed by itself

    This engine plug-in is not included in the published installation and deployment package by default, You can follow this guide to deploy the installation https://linkis.apache.org/zh-CN/blog/2022/04/15/how-to-download-engineconn-plugin Or manually compile the deployment according to the following process

    Compile separatelypipeline

    1.2 Material deployment and loading

    将 1.1 The engine package compiled in step, located in

      Upload to the engine directory of the server

      1. ${LINKIS_HOME}/lib/linkis-engineplugins

      And restart the linkis engineplugin to refresh the engine

      Or refresh through the engine interface. After the engine is placed in the corresponding directory, send a refresh request to the linkis CG engineconplugin service through the HTTP interface.

      • Interfacehttp://${engineconn-plugin-server-IP}:${port}/api/rest_j/v1/rpc/receiveAndReply

      1. {
      2.   "method": "/enginePlugin/engineConn/refreshAll"

      Check whether the engine is refreshed successfully: if you encounter problems during the refresh process and need to confirm whether the refresh is successful, you can view thelinkis_engine_conn_plugin_bml_resourcesOf this tablelast_update_timeWhether it is the time when the refresh is triggered.

      1. #Log in to the database of linkis
      2. select *  from linkis_cg_engine_conn_plugin_bml_resources

      Linkis1.XIt is carried out through labels, so it is necessary to insert data into our database. The insertion method is shown below.

      2 Use of engine

      2.1 Task submission via linkis cli

      Link 1.0 provides cli to submit tasks. We only need to specify the corresponding enginecon and codetype tag types. The use of pipeline is as follows:

      • Note that the enginetype pipeline-1 engine version setting is prefixed. If the pipeline version is V1 , it is set to pipeline-1

      from hdfs:///000/000/000/A.dolphin to file:///000/000/000/B.csv 3.3 Explained

      For specific use, please refer to： Linkis CLI Manual.

      becausepipelineThe engine is mainly used to import and export files. Now let’s assume that importing files from a to B is the most introduced case

      Right click the workspace module and select Create a new workspace of typestorageScript for

      2.3 Script

      Syntax is：from path to path

      The syntax is file copy rule:dolphinSuffix type files are result set files that can be converted to.csvType and.xlsxType file, other types can only be copied from address a to address B, referred to as handling

      1. #dolphin type
      2. from hdfs:///000/000/000/A.dolphin to file:///000/000/000/B.xlsx
      5. #Other types
      6. from hdfs:///000/000/000/A.txt to file:///000/000/000/B.txt

      A file importing script to B folder

      1. from hdfs:///000/000/000/A.csv to file:///000/000/B/
      • from grammar，to：grammar
      • hdfs:///000/000/000/A.csv：Input file path
      • file:///000/000/B/： Output path

      file B is exported as file A

      • ： Input file path

      Note: no semicolon is allowed at the end of the syntax; Otherwise, the syntax is incorrect.

      speed of progress

      Pipeline Engine - 图3

      historical information