Configurations in Details

Preface

Directory Structure

Currently, all the configuration files are under [conf ] directory. Please check the following simplified DolphinScheduler installation directories to have a direct view about the position [conf] directory in and configuration files inside. This document only describes DolphinScheduler configurations and other modules are not going into.

[Note: the DolphinScheduler (hereinafter called the ‘DS’) .]

Configurations in Details

dolphinscheduler-daemon.sh is responsible for DS startup & shutdown. Essentially, startup/shutdown the cluster via dolphinscheduler-daemon.sh. Currently, DS just makes a basic config, please config further JVＭ options based on your practical situation of resources.

2.datasource.properties [datasource config properties]

DS uses Druid to manage database connections and default simplified configs are:

Parameters	Default value	Description
spring.datasource.driver-class-name		datasource driver
spring.datasource.url		datasource connection url
spring.datasource.username		datasource username
spring.datasource.password		datasource password
spring.datasource.initialSize	5	initail connection pool size number
spring.datasource.minIdle	5	minimum connection pool size number
spring.datasource.maxActive	5	maximum connection pool size number
spring.datasource.maxWait	60000	max wait mili-seconds
spring.datasource.timeBetweenEvictionRunsMillis	60000	idle connection check interval
spring.datasource.timeBetweenConnectErrorMillis	60000	retry interval
spring.datasource.minEvictableIdleTimeMillis	300000	connections over minEvictableIdleTimeMillis will be collect when idle check
spring.datasource.validationQuery	SELECT 1	validate connection by running the SQL
spring.datasource.validationQueryTimeout	3	validate connection timeout[seconds]
spring.datasource.testWhileIdle	true	set whether the pool validates the allocated connection when a new connection request comes
spring.datasource.testOnBorrow	true	validity check when the program requests a new connection
spring.datasource.testOnReturn	false	validity check when the program recalls a connection
spring.datasource.defaultAutoCommit	true	whether auto commit
spring.datasource.keepAlive	true	runs validationQuery SQL to avoid the connection closed by pool when the connection idles over minEvictableIdleTimeMillis
spring.datasource.poolPreparedStatements	true	Open PSCache
spring.datasource.maxPoolPreparedStatementPerConnectionSize	20	specify the size of PSCache on each connection

3.zookeeper.properties [zookeeper config properties]

Parameters	Default value	Description
zookeeper.quorum	localhost:2181	zookeeper cluster connection info
zookeeper.dolphinscheduler.root	/dolphinscheduler	DS is stored under zookeeper root directory
zookeeper.session.timeout	60000	session timeout
zookeeper.connection.timeout	30000	connection timeout
zookeeper.retry.base.sleep	100	time to wait between subsequent retries
zookeeper.retry.max.sleep	30000	maximum time to wait between subsequent retries
zookeeper.retry.maxtime	10	maximum retry times

4.common.properties [hadoop、s3、yarn config properties]

Currently, common.properties mainly configures hadoop/s3a related configurations.

Parameters	Default value	Description
server.port	12345	api service communication port
server.servlet.session.timeout	7200	session timeout
server.servlet.context-path	/dolphinscheduler	request path
spring.servlet.multipart.max-file-size	1024MB	maximum file size
spring.servlet.multipart.max-request-size	1024MB	maximum request size
server.jetty.max-http-post-size	5000000	jetty maximum post size
spring.messages.encoding	UTF-8	message encoding
spring.jackson.time-zone	GMT+8	time zone
spring.messages.basename	i18n/messages	i18n config
security.authentication.type	PASSWORD	authentication type

6.master.properties [master-service log config]

Parameters	Default value	Description
master.listen.port	5678	master listen port
master.exec.threads	100	master execute thread number to limit process instances in parallel
master.exec.task.num	20	master execute task number in parallel per process instance
master.dispatch.task.num	3	master dispatch task number per batch
master.host.selector	LowerWeight	master host selector to select a suitable worker, default value: LowerWeight. Optional values include Random, RoundRobin, LowerWeight
master.heartbeat.interval	10	master heartbeat interval, the unit is second
master.task.commit.retryTimes	5	master commit task retry times
master.task.commit.interval	1000	master commit task interval, the unit is millisecond
master.max.cpuload.avg	-1	master max CPU load avg, only higher than the system CPU load average, master server can schedule. default value -1: the number of CPU cores * 2
master.reserved.memory	0.3	master reserved memory, only lower than system available memory, master server can schedule. default value 0.3, the unit is G

7.worker.properties [worker-service log config]

8.alert.properties [alert-service log config]

Parameters	Default value	Description
alert.type	EMAIL	alter type
mail.protocol	SMTP	mail server protocol
mail.server.host		mail server host
mail.server.port	25	mail server port
mail.sender	xxx@xxx.com	mail sender email
mail.user		mail sender email name
mail.passwd	111111	mail sender email password
mail.smtp.starttls.enable	true	specify mail whether open tls
mail.smtp.ssl.enable	false	specify mail whether open ssl
mail.smtp.ssl.trust	xxx.xxx.com	specify mail ssl trust list
xls.file.path	/tmp/xls	mail attachment temp storage directory
	following configure WeCom[optional]
enterprise.wechat.enable	false	specify whether enable WeCom
	xxxxxxx	WeCom corp id
enterprise.wechat.secret	xxxxxxx	WeCom secret
enterprise.wechat.agent.id	xxxxxxx	WeCom agent id
enterprise.wechat.users	xxxxxxx	WeCom users
enterprise.wechat.token.url	? corpid=corpId&corpsecret=secret	WeCom token url
enterprise.wechat.push.url	https://qyapi.weixin.qq.com/cgi-bin/message/send? access_token=$token	WeCom push url
enterprise.wechat.user.send.msg		send message format
enterprise.wechat.team.send.msg		group message format
plugin.dir	/Users/xx/your/path/to/plugin/dir	plugin directory

Parameters	Default value	Description
org.quartz.jobStore.driverDelegateClass	org.quartz.impl.jdbcjobstore.StdJDBCDelegate
org.quartz.jobStore.driverDelegateClass	org.quartz.impl.jdbcjobstore.PostgreSQLDelegate
org.quartz.scheduler.instanceName	DolphinScheduler
org.quartz.scheduler.instanceId	AUTO
org.quartz.scheduler.makeSchedulerThreadDaemon	true
org.quartz.jobStore.useProperties	false
org.quartz.threadPool.class	org.quartz.simpl.SimpleThreadPool
org.quartz.threadPool.makeThreadsDaemons	true
org.quartz.threadPool.threadCount	25
org.quartz.threadPool.threadPriority	5
org.quartz.jobStore.class	org.quartz.impl.jdbcjobstore.JobStoreTX
org.quartz.jobStore.tablePrefix	QRTZ_
org.quartz.jobStore.isClustered	true
org.quartz.jobStore.misfireThreshold	60000
org.quartz.jobStore.clusterCheckinInterval	5000
org.quartz.jobStore.acquireTriggersWithinLock	true
org.quartz.jobStore.dataSource	myDs
org.quartz.dataSource.myDs.connectionProvider.class	org.apache.dolphinscheduler.service.quartz.DruidConnectionProvider

10.install_config.conf [DS environment variables configuration script[install/start DS]]

install_config.conf is a bit complicated and is mainly used in the following two places.

2.Startup/shutdown DS cluster

11.dolphinscheduler_env.sh [load environment variables configs]

When using shell to commit tasks, DS will load environment variables inside dolphinscheduler_env.sh into the host. Types of tasks involved are: Shell task、Python task、Spark task、Flink task、Datax task and etc.

Configuration File