Resource Manager

2. The role of RM in Linkis

1. Engine Manager, ECM for short: Processes the microservices that start the engine connector request. As a resource provider, ECM is responsible for registering and unregistering resources with RM. At the same time, as the manager of the engine, ECM is responsible for applying for resources from RM instead of the new engine connector that is about to start. For each ECM instance, there is a corresponding resource record in the RM, which contains information such as the total resources and protection resources it provides, and dynamically updates the used resources.
   
2. The engine connector, referred to as EC, is the actual execution unit of user operations. At the same time, as the actual user of the resource, the EC is responsible for reporting the actual use of the resource to the RM. Each EC has a corresponding resource record in the RM: during the startup process, it is reflected as a locked resource; during the running process, it is reflected as a used resource; after being terminated, the resource record is subsequently deleted.
   

As shown in the figure above, all resource classes implement a top-level Resource interface, which defines the calculation and comparison methods that all resource classes need to support, and overloads the corresponding mathematical operators to enable resources to be Directly calculated and compared like numbers.

4. Available resource management

ECM resource report:

1. When the ECM is started, it will broadcast the ECM registration message. After receiving the message, the RM will register the resource according to the content contained in the message. The resource-related content includes:

   1. Total resources: the total number of resources that the ECM can provide.

   2. Protect resources: When the remaining resources are less than this resource, no further resources are allowed to be allocated.

   3. Resource type: such as LoadResource, DriverAndYarnResource and other type names.

   4. Instance information: machine name plus port name.

2. After RM receives the resource registration request, it adds a record in the resource table, the content is consistent with the parameter information of the interface, and finds the label representing the ECM through the instance information, and adds an association in the resource and label association table recording.

Configuration module tag resource configuration In the Configuration module, users can configure the number of resources based on different tag combinations, such as limiting the maximum available resources of the User/Creator/EngineType combination.

Receive user’s resource application:

1. When LinkisManager receives a request to start EngineConn, it will call RM’s resource application interface to apply for resources. The resource application interface accepts an optional time parameter. When the waiting time for applying for a resource exceeds the limit of the time parameter, the resource application will be automatically processed as a failure.
   Judging whether there are enough resources:
   That is, to determine whether the remaining available resources are greater than the requested resources, if greater than or equal to, the resources are sufficient; otherwise, the resources are insufficient.

2. RM preprocesses the label information attached to the resource application, and filters, combines and converts the original labels according to the rules (such as combining the User/Creator label and EngineType label), which makes the subsequent resource judgment more granular flexible.

4. According to each label:

   1. Query the corresponding resource record from the database through the Persistence module. If the record contains the remaining available resources, it is directly used for comparison.

   2. If there is no direct remaining available resource record, it will be calculated by the formula of [Remaining Available Resource=Maximum Available Resource-Used Resource-Locked Resource-Protected Resource].

   3. If there is no maximum available resource record, request the Configuration module to see if there is configured resource information, if so, use the formula for calculation, if not, skip the resource judgment for this tag.

   4. If there is no resource record, skip the resource judgment for this tag.

5. As long as one tag is judged to be insufficient in resources, the resource application will fail, and each tag will be unlocked one by one.

6. Only when all tags are judged to be sufficient resources, can the resource application be successfully passed and proceed to the next step.

lock by application of resources:

1. The number of resource request by generating a new record in the resource table, and associated with each tag.

2. If there is a tag corresponding to the remaining available resource record, the corresponding number of the abatement.

3. Generate a timed task, the lock checks whether these resources are actually used after a certain time, if the timeout is not used, it is mandatory recycling.

4. unlock each tag.

report the actual use of resources:

1. If you have locked resource, the EngineConn all labels associated lock.

2. For each tag, the resource record corresponding lock record for the conversion of used resources.

3. Unlock all labels.

Release actual used resources:

1. EngineConn after the end of the life cycle, recycling broadcast messages. RM after receiving the message, check whether the EngineConn corresponding label resources have been recorded.

2. If so, all the labels associated EngineConn be locked.

3, minus the amount used in the corresponding resource record for each label.

1. If there is a tag corresponding to the remaining available resource record, the corresponding increase in number.

2. The unlocking each tag

6. External resource management

1. Unified management of resources through tags. After the resource is registered, it is associated with the tag, so that the attributes of the resource can be expanded infinitely. At the same time, resource applications are also tagged to achieve flexible matching.

2. Abstract the cluster into one or more tags, and maintain the environmental information corresponding to each cluster tag in the external resource management module to achieve dynamic docking.

3. Abstract a general external resource management module. If you need to access new external resource types, you can convert different types of resource information into Resource entities in the RM as long as you implement a fixed interface to achieve unified management.
   

4. The type, label, configuration and other attributes of all external resources (such as cluster name, Yarn web url, Hadoop version and other information) are maintained in the linkis_external_resource_provider table.