SPARK

• Through the SPARK node, you can directly execute the SPARK program. For the spark node, the worker will use the method to submit tasks

• Program type: supports JAVA, Scala and Python three languages
• The class of the main function: is the full path of the Spark program’s entry Main Class
• Deployment mode: support three modes of yarn-cluster, yarn-client and local
• Driver core number: You can set the number of Driver cores and the number of memory
• Number of Executors: You can set the number of Executors, the number of Executor memory, and the number of Executor cores
• Other parameters: support —jars, —files, —archives, —conf format
• Resource: If the resource file is referenced in other parameters, you need to select and specify in the resource
• User-defined parameter: It is a user-defined parameter of the MR part, which will replace the content with ${variable} in the script