调优手册

    1. wds.linkis.rpc.receiver.asyn.consumer.thread.max=400
    2. wds.linkis.rpc.receiver.asyn.queue.size.max=5000
    3. wds.linkis.rpc.sender.asyn.consumer.thread.max=100
    4. wds.linkis.rpc.sender.asyn.queue.size.max=2000
    1. private val groupNameToGroups = new JMap[String, Group]
    2. override def getOrCreateGroup(groupName: String): Group = {
    3. if (!groupNameToGroups.containsKey(groupName)) synchronized {
    4. val initCapacity = 100
    5. val maxCapacity = 100
    6. // 其它代码...

    在linkis-engineconn-plugins/engineconn-plugins里的Spark的部分配置,可以调整该配置改变提交到Yarn上的任务运行时环境,由于篇幅有限,如更多有关Hive,Yarn配置需要用户参考源码和参数文档:

    1. "spark.driver.memory" = 2 //单位为G
    2. "wds.linkis.driver.cores" = 1
    3. "spark.executor.memory" = 4 //单位为G
    4. "spark.executor.cores" = 2
    5. "spark.executor.instances" = 3