我的书签
添加书签
移除书签Hive Example
Prepare to modify configuration file
- conf/SortClusterConfig.conf, sink configuration of all sort tasks.
- conf/sid_hive_inlong6th_v3.conf, data source configuration example of a sort task, the file name is same with sort task name in SortClusterConfig.conf.
Example: conf/SortClusterConfig.conf
Example: conf/sid_hive_inlong6th_v3.conf
Modify configuration file: SortClusterConfig.conf
- Get cluster configuration data from the file:SortClusterConfig.conf in classpath, it can not support online updating.
- Get cluster configuration data from InlongManager URL, it can support online updating.
| Parameter | Required | Type | DefaultValue | Remark |
|---|---|---|---|---|
| clusterName | Y | String | NA | inlong-sort-standalone cluster id |
| sortTasks | Y | JsonArray<SortTaskConfig> | NA | Sort task list |
Modify configuration: idParams of Hive sort task
| Parameter | Required | DefaultValue | Remark |
|---|---|---|---|
| inlongGroupId | Y | NA | inlongGroupId |
| inlongStreamId | Y | NA | inlongStreamId |
| separator | Y | NA | separator of Inlong datastream in data source |
| partitionIntervalMs | N | 3600000 | partition interval(millisecond) |
| idRootPath | Y | NA | HDFS root path of Inlong DataStream |
| partitionSubPath | Y | NA | partition sub path of Inlong DataStream |
| hiveTableName | Y | NA | Hive table name of Inlong DataStream |
| partitionFieldName | N | dt | partition field name of Inlong DataStream |
| partitionFieldPattern | Y | NA | Date format of partition field value, the type have {yyyyMMdd},{yyyyMMddHH},{yyyyMMddHHmm} |
| msgTimeFieldPattern | Y | NA | Date format of message generation time, it support Java date format |
| maxPartitionOpenDelayHour | N | 8 | Max delay hour of partition(hour) |
Modify configuration: sinkParams of Hive sort task
- The file name include sort task name plus the postfix “.conf”.
- Get the configuration data from the file in classpath, it can not support online updating.
- Get the configuration data from InlongManager URL, it can support online updating.
| Parameter | Required | Type | DefaultValue | Remark |
|---|---|---|---|---|
| sortClusterName | Y | String | NA | inlong-sort-standalone cluster id |
| sortTaskId | Y | String | NA | Sort task name |
| cacheZones | Y | JsonObject<String, JsonObject> | NA | Cache cluster list, Map<cacheClusterName, CacheCluster> |
Modify configuration: CacheCluster
Modify configuration: Topic
| Parameter | Required | Type | DefaultValue | Remark |
|---|---|---|---|---|
| topic | Y | String | NA | cache topic name |
| partitionCnt | Y | Integer | NA | cache topic partition count |
| topicProperties | N | Map<String,String> | NA | Cache topic configuration |
