Overall Tasks Storage Structure

The following shows the ‘t_ds_process_definition’ table structure:

The ‘process_definition_json’ field is the core field, which defines the task information in the DAG diagram, and it is stored in JSON format.

The following table describes the common data structure.

No.fieldtypedescription
1globalParamsArrayglobal parameters
2tasksArraytask collections in the process [for the structure of each type, please refer to the following sections]
3tenantIdinttenant ID
4timeoutinttimeout

Data example:

The Detailed Explanation of The Storage Structure of Each Task Type

The node data structure is as follows:

No.parameter nametypedescriptionnotes
1idStringtask Id
2typeStringtask typeSHELL
3nameStringtask name
4paramsObjectcustomized parametersJson format
5rawScriptStringShell script
6localParamsArraycustomized local parameters
7resourceListArrayresource files
8descriptionStringdescription
9runFlagStringexecution flag
10conditionResultObjectcondition branch
11successNodeArrayjump to node if success
12failedNodeArrayjump to node if failure
13dependenceObjecttask dependencymutual exclusion with params
14maxRetryTimesStringmax retry times
15retryIntervalStringretry interval
16timeoutObjecttimeout
17taskInstancePriorityStringtask priority
18workerGroupStringWorker group
19preTasksArraypreposition tasks

Node data example:

  2.     "type":"SHELL",
  3.     "id":"tasks-80760",
  4.     "name":"Shell Task",
  5.     "params":{
  6.         "resourceList":[
  7.             {
  8.                 "id":3,
  9.                 "name":"run.sh",
  10.                 "res":"run.sh"
  11.             }
  12.         ],
  13.         "localParams":[
  15.         ],
  16.         "rawScript":"echo "This is a shell script""
  17.     },
  18.     "description":"",
  19.     "runFlag":"NORMAL",
  20.     "conditionResult":{
  21.         "successNode":[
  22.             ""
  23.         ],
  24.         "failedNode":[
  25.             ""
  26.         ]
  27.     },
  28.     "dependence":{
  30.     },
  31.     "maxRetryTimes":"0",
  32.     "retryInterval":"1",
  33.     "timeout":{
  34.         "strategy":"",
  35.         "interval":null,
  36.         "enable":false
  37.     },
  38.     "taskInstancePriority":"MEDIUM",
  39.     "workerGroup":"default",
  40.     "preTasks":[
  42.     ]
  43. }

SQL Node

Perform data query and update operations on the specified datasource through SQL.

The node data structure is as follows:

No.parameter nametypedescriptionnote
1idStringtask id
2typeStringtask typeSQL
3nameStringtask name
4paramsObjectcustomized parametersJson format
5typeStringdatabase type
6datasourceIntdatasource id
7sqlStringquery SQL statement
8udfsStringudf functionsspecify UDF function ids, separate by comma
9sqlTypeStringSQL node type0 for query and 1 for none-query SQL
10titleStringmail title
11receiversStringreceivers
12receiversCcStringCC receivers
13showTypeStringdisplay type of mailoptionals: TABLE or ATTACHMENT
14connParamsStringconnect parameters
15preStatementsArraypreposition SQL statements
16postStatementsArraypostposition SQL statements
17localParamsArraycustomized parameters
18descriptionStringdescription
19runFlagStringexecution flag
20conditionResultObjectcondition branch
21successNodeArrayjump to node if success
22failedNodeArrayjump to node if failure
23dependenceObjecttask dependencymutual exclusion with params
24maxRetryTimesStringmax retry times
25retryIntervalStringretry interval
26timeoutObjecttimeout
27taskInstancePriorityStringtask priority
28workerGroupStringWorker group
29preTasksArraypreposition tasks

Node data example:

  1. {
  2.     "type":"SQL",
  3.     "id":"tasks-95648",
  4.     "name":"SqlTask-Query",
  5.     "params":{
  6.         "type":"MYSQL",
  7.         "datasource":1,
  8.         "sql":"select id , namge , age from emp where id =  ${id}",
  9.         "udfs":"",
  10.         "sqlType":"0",
  11.         "title":"xxxx@xxx.com",
  12.         "receivers":"xxxx@xxx.com",
  13.         "receiversCc":"",
  14.         "showType":"TABLE",
  15.         "localParams":[
  16.             {
  17.                 "prop":"id",
  18.                 "direct":"IN",
  19.                 "type":"INTEGER",
  20.                 "value":"1"
  21.             }
  22.         ],
  23.         "connParams":"",
  24.         "preStatements":[
  25.             "insert into emp ( id,name ) value (1,'Li' )"
  26.         ],
  27.         "postStatements":[
  29.         ]
  30.     },
  31.     "description":"",
  32.     "runFlag":"NORMAL",
  33.     "conditionResult":{
  34.         "successNode":[
  35.             ""
  36.         ],
  37.         "failedNode":[
  38.             ""
  39.         ]
  40.     },
  41.     "dependence":{
  43.     },
  44.     "maxRetryTimes":"0",
  45.     "retryInterval":"1",
  46.     "timeout":{
  47.         "strategy":"",
  48.         "interval":null,
  49.         "enable":false
  50.     },
  51.     "taskInstancePriority":"MEDIUM",
  52.     "workerGroup":"default",
  53.     "preTasks":[
  55.     ]
  56. }

PROCEDURE [stored procedures] Node

SPARK Node

The node data structure is as follows:

Node data example:

  1. {
  2.     "type":"SPARK",
  3.     "id":"tasks-87430",
  4.     "name":"SparkTask",
  5.     "params":{
  6.         "mainClass":"org.apache.spark.examples.SparkPi",
  7.         "mainJar":{
  8.             "id":4
  9.         },
  10.         "deployMode":"cluster",
  11.         "resourceList":[
  12.             {
  13.                 "id":3,
  14.                 "name":"run.sh",
  15.                 "res":"run.sh"
  16.             }
  17.         ],
  18.         "localParams":[
  20.         ],
  21.         "driverCores":1,
  22.         "driverMemory":"512M",
  23.         "numExecutors":2,
  24.         "executorMemory":"2G",
  25.         "executorCores":2,
  26.         "mainArgs":"10",
  27.         "others":"",
  28.         "programType":"SCALA",
  29.     },
  30.     "description":"",
  31.     "runFlag":"NORMAL",
  32.     "conditionResult":{
  33.         "successNode":[
  34.             ""
  35.         ],
  36.         "failedNode":[
  37.             ""
  39.     },
  40.     "dependence":{
  42.     },
  43.     "maxRetryTimes":"0",
  44.     "retryInterval":"1",
  45.     "timeout":{
  46.         "strategy":"",
  47.         "interval":null,
  48.         "enable":false
  49.     },
  50.     "taskInstancePriority":"MEDIUM",
  51.     "workerGroup":"default",
  52.     "preTasks":[
  54.     ]
  55. }

The node data structure is as follows:

No.parameter nametypedescriptionnotes
1idStringtask Id
2typeStringtask typeMR
3nameStringtask name
4paramsObjectcustomized parametersJson format
5mainClassStringmain class
6mainArgsStringexecution arguments
7othersStringother arguments
8mainJarObjectapplication jar package
9programTypeStringprogram typeJAVA,PYTHON
10localParamsArraycustomized local parameters
11resourceListArrayresource files
12descriptionStringdescription
13runFlagStringexecution flag
14conditionResultObjectcondition branch
15successNodeArrayjump to node if success
16failedNodeArrayjump to node if failure
17dependenceObjecttask dependencymutual exclusion with params
18maxRetryTimesStringmax retry times
19retryIntervalStringretry interval
20timeoutObjecttimeout
21taskInstancePriorityStringtask priority
22workerGroupStringWorker group
23preTasksArraypreposition tasks

Node data example:

Python Node

The node data structure is as follows:

No.parameter nametypedescriptionnotes
1idStringtask Id
2typeStringtask typePYTHON
3nameStringtask name
4paramsObjectcustomized parametersJson format
5rawScriptStringPython script
6localParamsArraycustomized local parameters
7resourceListArrayresource files
8descriptionStringdescription
9runFlagStringexecution flag
10conditionResultObjectcondition branch
11successNodeArrayjump to node if success
12failedNodeArrayjump to node if failure
13dependenceObjecttask dependencymutual exclusion with params
14maxRetryTimesStringmax retry times
15retryIntervalStringretry interval
16timeoutObjecttimeout
17taskInstancePriorityStringtask priority
18workerGroupStringWorker group
19preTasksArraypreposition tasks

Node data example:

  1. {
  2.     "type":"PYTHON",
  3.     "id":"tasks-5463",
  4.     "name":"Python Task",
  5.     "params":{
  6.         "resourceList":[
  7.             {
  8.                 "id":3,
  9.                 "name":"run.sh",
  10.                 "res":"run.sh"
  11.             }
  12.         ],
  13.         "localParams":[
  15.         ],
  16.         "rawScript":"print("This is a python script")"
  17.     },
  18.     "description":"",
  19.     "runFlag":"NORMAL",
  20.     "conditionResult":{
  21.         "successNode":[
  22.             ""
  23.         ],
  24.         "failedNode":[
  25.             ""
  26.         ]
  27.     },
  28.     "dependence":{
  30.     },
  31.     "maxRetryTimes":"0",
  32.     "retryInterval":"1",
  33.     "timeout":{
  34.         "strategy":"",
  35.         "interval":null,
  36.         "enable":false
  37.     },
  38.     "taskInstancePriority":"MEDIUM",
  39.     "workerGroup":"default",
  40.     "preTasks":[
  42.     ]
  43. }

The node data structure is as follows:

No.parameter nametypedescriptionnotes
1idStringtask Id
2typeStringtask typeFLINK
3nameStringtask name
4paramsObjectcustomized parametersJson format
5mainClassStringmain class
6mainArgsStringexecution arguments
7othersStringother arguments
8mainJarObjectapplication jar package
9deployModeStringdeployment modelocal,client,cluster
10slotStringslot count
11taskManagerStringtaskManager count
12taskManagerMemoryStringtaskManager memory size
13jobManagerMemoryStringjobManager memory size
14programTypeStringprogram typeJAVA,SCALA,PYTHON
15localParamsArraylocal parameters
16resourceListArrayresource files
17descriptionStringdescription
18runFlagStringexecution flag
19conditionResultObjectcondition branch
20successNodeArrayjump node if success
21failedNodeArrayjump node if failure
22dependenceObjecttask dependencymutual exclusion with params
23maxRetryTimesStringmax retry times
24retryIntervalStringretry interval
25timeoutObjecttimeout
26taskInstancePriorityStringtask priority
27workerGroupStringWorker group
38preTasksArraypreposition tasks

Node data example:

  1. {
  2.     "type":"FLINK",
  3.     "id":"tasks-17135",
  4.     "name":"FlinkTask",
  5.     "params":{
  6.         "mainClass":"com.flink.demo",
  7.         "mainJar":{
  8.             "id":6
  9.         },
  10.         "deployMode":"cluster",
  11.         "resourceList":[
  12.             {
  13.                 "id":3,
  14.                 "name":"run.sh",
  15.                 "res":"run.sh"
  16.             }
  17.         ],
  18.         "localParams":[
  20.         ],
  21.         "slot":1,
  22.         "taskManager":"2",
  23.         "jobManagerMemory":"1G",
  24.         "taskManagerMemory":"2G",
  25.         "executorCores":2,
  26.         "mainArgs":"100",
  27.         "others":"",
  28.         "programType":"SCALA"
  29.     },
  30.     "description":"",
  31.     "runFlag":"NORMAL",
  32.     "conditionResult":{
  33.         "successNode":[
  34.             ""
  35.         ],
  36.         "failedNode":[
  37.             ""
  38.         ]
  39.     },
  40.     "dependence":{
  42.     },
  43.     "maxRetryTimes":"0",
  44.     "retryInterval":"1",
  45.     "timeout":{
  46.         "strategy":"",
  47.         "interval":null,
  48.         "enable":false
  49.     },
  50.     "taskInstancePriority":"MEDIUM",
  51.     "workerGroup":"default",
  52.     "preTasks":[
  54.     ]
  55. }

HTTP Node

The node data structure is as follows:

  1. {
  2.     "type":"HTTP",
  3.     "id":"tasks-60499",
  4.     "name":"HttpTask",
  5.     "params":{
  6.         "localParams":[
  8.         ],
  9.         "httpParams":[
  10.             {
  11.                 "prop":"id",
  12.                 "httpParametersType":"PARAMETER",
  13.                 "value":"1"
  14.             },
  15.             {
  16.                 "prop":"name",
  17.                 "value":"Bo"
  18.             }
  19.         ],
  20.         "url":"https://www.xxxxx.com:9012",
  22.         "httpCheckCondition":"STATUS_CODE_DEFAULT",
  23.         "condition":""
  24.     },
  25.     "description":"",
  26.     "runFlag":"NORMAL",
  27.     "conditionResult":{
  28.         "successNode":[
  29.             ""
  30.         ],
  31.         "failedNode":[
  32.             ""
  33.         ]
  34.     },
  35.     "dependence":{
  37.     },
  38.     "maxRetryTimes":"0",
  39.     "retryInterval":"1",
  40.     "timeout":{
  41.         "strategy":"",
  42.         "interval":null,
  43.         "enable":false
  44.     },
  45.     "taskInstancePriority":"MEDIUM",
  46.     "workerGroup":"default",
  47.     "preTasks":[
  49.     ]
  50. }

The node data structure is as follows:

No.parameter nametypedescriptionnotes
1idStringtask Id
2typeStringtask typeDATAX
3nameStringtask name
4paramsObjectcustomized parametersJson format
5customConfigIntspecify whether use customized config0 none customized, 1 customized
6dsTypeStringdatasource type
7dataSourceIntdatasource ID
8dtTypeStringtarget database type
9dataTargetInttarget database ID
10sqlStringSQL statements
11targetTableStringtarget table
12jobSpeedByteIntjob speed limiting(bytes)
13jobSpeedRecordIntjob speed limiting(records)
14preStatementsArraypreposition SQL
15postStatementsArraypostposition SQL
16jsonStringcustomized configsvalid if customConfig=1
17localParamsArraycustomized parametersvalid if customConfig=1
18descriptionStringdescription
19runFlagStringexecution flag
20conditionResultObjectcondition branch
21successNodeArrayjump node if success
22failedNodeArrayjump node if failure
23dependenceObjecttask dependencymutual exclusion with params
24maxRetryTimesStringmax retry times
25retryIntervalStringretry interval
26timeoutObjecttimeout
27taskInstancePriorityStringtask priority
28workerGroupStringWorker group
29preTasksArraypreposition tasks

Node data example:

Sqoop Node

The node data structure is as follows:

No.parameter nametypedescriptionnotes
1idStringtask ID
2typeStringtask typeSQOOP
3nameStringtask name
4paramsObjectcustomized parametersJson format
5concurrencyIntconcurrency rate
6modelTypeStringflow directionimport,export
7sourceTypeStringdatasource type
8sourceParamsStringdatasource parametersJSON format
9targetTypeStringtarget datasource type
10targetParamsStringtarget datasource parametersJSON format
11localParamsArraycustomized local parameters
12descriptionStringdescription
13runFlagStringexecution flag
14conditionResultObjectcondition branch
15successNodeArrayjump node if success
16failedNodeArrayjump node if failure
17dependenceObjecttask dependencymutual exclusion with params
18maxRetryTimesStringmax retry times
19retryIntervalStringretry interval
20timeoutObjecttimeout
21taskInstancePriorityStringtask priority
22workerGroupStringWorker group
23preTasksArraypreposition tasks

Node data example:

  1. {
  2.             "type":"SQOOP",
  3.             "id":"tasks-82041",
  4.             "name":"Sqoop Task",
  5.             "params":{
  6.                 "concurrency":1,
  7.                 "modelType":"import",
  8.                 "sourceType":"MYSQL",
  9.                 "targetType":"HDFS",
  10.                 "sourceParams":"{"srcType":"MYSQL","srcDatasource":1,"srcTable":"","srcQueryType":"1","srcQuerySql":"selec id , name from user","srcColumnType":"0","srcColumns":"","srcConditionList":[],"mapColumnHive":[{"prop":"hivetype-key","direct":"IN","type":"VARCHAR","value":"hivetype-value"}],"mapColumnJava":[{"prop":"javatype-key","direct":"IN","type":"VARCHAR","value":"javatype-value"}]}",
  11.                 "targetParams":"{"targetPath":"/user/hive/warehouse/ods.db/user","deleteTargetDir":false,"fileType":"--as-avrodatafile","compressionCodec":"snappy","fieldsTerminated":",","linesTerminated":"@"}",
  12.                 "localParams":[
  14.                 ]
  15.             },
  16.             "description":"",
  17.             "runFlag":"NORMAL",
  18.             "conditionResult":{
  19.                 "successNode":[
  20.                     ""
  21.                 ],
  22.                 "failedNode":[
  23.                     ""
  24.                 ]
  25.             },
  26.             "dependence":{
  28.             },
  29.             "maxRetryTimes":"0",
  30.             "retryInterval":"1",
  31.             "timeout":{
  32.                 "strategy":"",
  33.                 "interval":null,
  34.                 "enable":false
  35.             },
  36.             "taskInstancePriority":"MEDIUM",
  37.             "workerGroup":"default",
  38.             "preTasks":[
  40.             ]
  41.         }

Condition Branch Node

The node data structure is as follows:

No.parameter nametypedescriptionnotes
1idStringtask ID
2typeStringtask typeSHELL
3nameStringtask name
4paramsObjectcustomized parametersnull
5descriptionStringdescription
6runFlagStringexecution flag
7conditionResultObjectcondition branch
8successNodeArrayjump to node if success
9failedNodeArrayjump to node if failure
10dependenceObjecttask dependencymutual exclusion with params
11maxRetryTimesStringmax retry times
12retryIntervalStringretry interval
13timeoutObjecttimeout
14taskInstancePriorityStringtask priority
15workerGroupStringWorker group
16preTasksArraypreposition tasks

Node data example:

  1. {
  2.     "type":"CONDITIONS",
  3.     "id":"tasks-96189",
  4.     "name":"条件",
  5.     "params":{
  7.     },
  8.     "description":"",
  9.     "runFlag":"NORMAL",
  10.     "conditionResult":{
  11.         "successNode":[
  12.             "test04"
  13.         ],
  14.         "failedNode":[
  15.             "test05"
  16.         ]
  17.     },
  18.     "dependence":{
  19.         "relation":"AND",
  20.         "dependTaskList":[
  22.         ]
  23.     },
  24.     "maxRetryTimes":"0",
  25.     "retryInterval":"1",
  26.     "timeout":{
  27.         "strategy":"",
  28.         "interval":null,
  29.         "enable":false
  30.     },
  31.     "taskInstancePriority":"MEDIUM",
  32.     "workerGroup":"default",
  33.     "preTasks":[
  34.         "test01",
  35.         "test02"
  36.     ]
  37. }

Subprocess Node

The node data structure is as follows:

Node data example:

  1. {
  2.             "type":"SUB_PROCESS",
  3.             "id":"tasks-14806",
  4.             "name":"SubProcessTask",
  5.             "params":{
  6.                 "processDefinitionId":2
  7.             },
  8.             "description":"",
  9.             "runFlag":"NORMAL",
  10.             "conditionResult":{
  11.                 "successNode":[
  12.                     ""
  13.                 ],
  14.                 "failedNode":[
  15.                     ""
  16.                 ]
  17.             },
  18.             "dependence":{
  20.             },
  21.             "timeout":{
  22.                 "strategy":"",
  23.                 "interval":null,
  24.                 "enable":false
  25.             },
  26.             "taskInstancePriority":"MEDIUM",
  27.             "workerGroup":"default",
  28.             "preTasks":[
  30.         }

The node data structure is as follows:

No.parameter nametypedescriptionnotes
1idStringtask ID
2typeStringtask typeDEPENDENT
3nameStringtask name
4paramsObjectcustomized parametersJson format
5rawScriptStringShell script
6localParamsArraycustomized local parameters
7resourceListArrayresource files
8descriptionStringdescription
9runFlagStringexecution flag
10conditionResultObjectcondition branch
11successNodeArrayjump to node if success
12failedNodeArrayjump to node if failure
13dependenceObjecttask dependencymutual exclusion with params
14relationStringrelationAND,OR
15dependTaskListArraydependent task list
16maxRetryTimesStringmax retry times
17retryIntervalStringretry interval
18timeoutObjecttimeout
19taskInstancePriorityStringtask priority
20workerGroupStringWorker group
21preTasksArraypreposition tasks