Introduction to Linkis Configuration Parameters

    1. Linkis server parameters, mainly including the parameters of Linkis itself and the parameters of Spring
    2. Parameters submitted by client calls such as Linkis SDK and Restful
    3. Linkis console parameters
    1. Parameters of Linkis itself The parameters of linkis itself can be set in the configuration file, or through environment variables and system properties. It is recommended to use the configuration file to set. The Linkis configuration file format is as follows:
    2. │ ├── application-eureka.yml
    3. │ ├── application-linkis.yml
    4. │ ├── linkis-cg-engineconnmanager-io.properties
    5. │ ├── linkis-cg-engineconnmanager.properties
    6. │ ├── linkis-cg-engineplugin.properties
    7. │ ├── linkis-cg-entrance.properties
    8. │ ├── linkis-cg-linkismanager.properties
    9. │ ├── linkis.properties ── linkis global properties
    10. │ ├── linkis-ps-bml.properties
    11. │ ├── linkis-ps-cs.properties
    12. │ ├── linkis-ps-datasource.properties
    13. │ ├── linkis-ps-publicservice.properties
    14. │ ├── log4j2.xml
    1. Spring parameters Linkis service is based on SpringBoot application, Spring related parameters can be set in application-linkis.yml or in linkis configuration file. The configuration in the linkis configuration file needs to be prefixed with spring. as follows:
    1. server.port=9102
    2. # in linkis conf need spring prefix
    3. spring.server.port=9102
    1. How restful sets parameters:
    1. {
    2.     "executionContent": {"code": "show tables", "runType": "sql"},
    4.                         "variable":{ //Custom variables needed in the code
    5.                                 "k1":"v1"
    6.                         },
    7.                         "configuration":{
    8.                                 "special":{ //Special configuration parameters such as log path, result set path, etc.
    9.                                         "k2":"v2"
    10.                                 },
    11.                                 "runtime":{ //Runtime parameters, execution configuration parameters, such as database connection parameters of JDBC engine, data source parameters of presto engine
    12.                                         "k3":"v3"
    13.                                 },
    14.                                 "startup":{ //Startup parameters, such as memory parameters for starting EC, spark engine parameters, hive engine parameters, etc.
    15.                                         "k4":"v4" For example: spark.executor.memory:5G Set the Spark executor memory, the underlying Spark, hive and other engine parameters keyName are consistent with the native parameters
    16.                                 }
    17.                         }
    18.                 },
    19.     "labels": { //Label parameters, support setting engine version, user and application
    20.         "userCreator": "hadoop-IDE"
    21.     }
    1. How to set parameters in SDK:
    1. JobSubmitAction jobSubmitAction = JobSubmitAction.builder()
    2.                 .addExecuteCode(code)
    3.                 .setStartupParams(startupMap) //Startup parameters, such as memory parameters for starting EC, spark engine parameters, hive engine parameters, etc., such as: spark.executor.memory:5G Set the Spark executor memory, the underlying Spark, hive and other engine parameters keyName is the same as the original parameter
    4.                 .setRuntimeParams(runTimeMap) //Engine, execute configuration parameters, such as database connection parameters of JDBC engine, data source parameters of presto engine
    5.                 .setVariableMap(varMap) //Custom variables needed in the code
    6.                 .setLabels(labels) //Label parameters, support setting engine version, user and application, etc.
    7.                 .setUser(user) //submit user
    8.                 .addExecuteUser(user) // execute user
    9.                 .build();
    1. How linkis-cli sets parameters
    1. Common label parameters:
    1.     Map<String, Object> labels = new HashMap<String, Object>();
    2.      labels.put(LabelKeyConstant.ENGINE_TYPE_KEY, "spark-2.4.3"); // Specify engine type and version
    3.      labels.put(LabelKeyConstant.USER_CREATOR_TYPE_KEY, user + "-IDE");// Specify the running user and your APPName
    4.      labels.put(LabelKeyConstant.CODE_TYPE_KEY, "sql"); // Specify the type of script to run: spark supports: sql, scala, py; Hive: hql; shell: sh; python: python; presto: psql
    5.      labels.put(LabelKeyConstant.JOB_RUNNING_TIMEOUT_KEY, "10000");//The job runs for 10s and automatically initiates Kill, the unit is s
    6.      labels.put(LabelKeyConstant.JOB_QUEUING_TIMEOUT_KEY, "10000");//The job is queued for more than 10s and automatically initiates Kill, the unit is s
    7.      labels.put(LabelKeyConstant.RETRY_TIMEOUT_KEY, "10000");//The waiting time for the job to retry due to resources and other reasons, the unit is ms. If it fails due to insufficient queue resources, it will initiate 10 retries at intervals by default
    8.      labels.put(LabelKeyConstant.TENANT_KEY,"hduser02");//Tenant label, if the tenant parameter is specified for the task, the task will be routed to a separate ECM machine
    1.     Linkis management console parameters are convenient for users to specify resource limit parameters and default task parameters. The web interface provided is as follows: Global configuration parameters: ![](/projects/Linkis-1.2.0-en/34101f2e00d2cbbcb1cb76e055a13b31.png) It mainly includes the global queue parameter \[wds.linkis.rm.yarnqueue\], the Yarn queue used by the task by default, which can be specified in the client StartUPMap. Resource limit parameters, these parameters do not support task settings, but can be adjusted by the management console.