Resource Quotas

    Resource quotas are a tool for administrators to address this concern.

    A resource quota, defined by a object, provides constraints that limit aggregate resource consumption per namespace. It can limit the quantity of objects that can be created in a namespace by type, as well as the total amount of compute resources that may be consumed by resources in that namespace.

    Resource quotas work like this:

    • Different teams work in different namespaces. Currently this is voluntary, but support for making this mandatory via ACLs is planned.

    • The administrator creates one ResourceQuota for each namespace.

    • Users create resources (pods, services, etc.) in the namespace, and the quota system tracks usage to ensure it does not exceed hard resource limits defined in a ResourceQuota.

    • If creating or updating a resource violates a quota constraint, the request will fail with HTTP status code 403 FORBIDDEN with a message explaining the constraint that would have been violated.

    • If quota is enabled in a namespace for compute resources like cpu and memory, users must specify requests or limits for those values; otherwise, the quota system may reject pod creation. Hint: Use the LimitRanger admission controller to force defaults for pods that make no compute resource requirements.

      See the walkthrough for an example of how to avoid this problem.

    The name of a ResourceQuota object must be a valid .

    Examples of policies that could be created using namespaces and quotas are:

    • In a cluster with a capacity of 32 GiB RAM, and 16 cores, let team A use 20 GiB and 10 cores, let B use 10GiB and 4 cores, and hold 2GiB and 2 cores in reserve for future allocation.
    • Limit the “testing” namespace to using 1 core and 1GiB RAM. Let the “production” namespace use any amount.

    In the case where the total capacity of the cluster is less than the sum of the quotas of the namespaces, there may be contention for resources. This is handled on a first-come-first-served basis.

    Neither contention nor changes to quota will affect already created resources.

    Resource Quota support is enabled by default for many Kubernetes distributions. It is enabled when the API server --enable-admission-plugins= flag has ResourceQuota as one of its arguments.

    A resource quota is enforced in a particular namespace when there is a ResourceQuota in that namespace.

    Compute Resource Quota

    You can limit the total sum of compute resources that can be requested in a given namespace.

    The following resource types are supported:

    In addition to the resources mentioned above, in release 1.10, quota support for is added.

    As overcommit is not allowed for extended resources, it makes no sense to specify both requests and limits for the same extended resource in a quota. So for extended resources, only quota items with prefix requests. is allowed for now.

    Take the GPU resource as an example, if the resource name is nvidia.com/gpu, and you want to limit the total number of GPUs requested in a namespace to 4, you can define a quota as follows:

    • requests.nvidia.com/gpu: 4

    See Viewing and Setting Quotas for more detail information.

    Storage Resource Quota

    You can limit the total sum of storage resources that can be requested in a given namespace.

    In addition, you can limit consumption of storage resources based on associated storage-class.

    For example, if an operator wants to quota storage with gold storage class separate from bronze storage class, the operator can define a quota as follows:

    • gold.storageclass.storage.k8s.io/requests.storage: 500Gi
    • bronze.storageclass.storage.k8s.io/requests.storage: 100Gi

    Note: When using a CRI container runtime, container logs will count against the ephemeral storage quota. This can result in the unexpected eviction of pods that have exhausted their storage quotas. Refer to for details.

    You can set quota for the total number of certain resources of all standard, namespaced resource types using the following syntax:

    • count/<resource>.<group> for resources from non-core groups
    • count/<resource> for resources from the core group

    Here is an example set of resources users may want to put under object count quota:

    • count/persistentvolumeclaims
    • count/services
    • count/secrets
    • count/configmaps
    • count/replicationcontrollers
    • count/deployments.apps
    • count/replicasets.apps
    • count/statefulsets.apps
    • count/jobs.batch
    • count/cronjobs.batch

    The same syntax can be used for custom resources. For example, to create a quota on a widgets custom resource in the example.com API group, use count/widgets.example.com.

    When using count/* resource quota, an object is charged against the quota if it exists in server storage. These types of quotas are useful to protect against exhaustion of storage resources. For example, you may want to limit the number of Secrets in a server given their large size. Too many Secrets in a cluster can actually prevent servers and controllers from starting. You can set a quota for Jobs to protect against a poorly configured CronJob. CronJobs that create too many Jobs in a namespace can lead to a denial of service.

    It is also possible to do generic object count quota on a limited set of resources. The following types are supported:

    For example, pods quota counts and enforces a maximum on the number of pods created in a single namespace that are not terminal. You might want to set a pods quota on a namespace to avoid the case where a user creates many small pods and exhausts the cluster’s supply of Pod IPs.

    Quota Scopes

    Each quota can have an associated set of scopes. A quota will only measure usage for a resource if it matches the intersection of enumerated scopes.

    When a scope is added to the quota, it limits the number of resources it supports to those that pertain to the scope. Resources specified on the quota outside of the allowed set results in a validation error.

    The BestEffort scope restricts a quota to tracking the following resource:

    • pods

    The Terminating, NotTerminating, NotBestEffort and PriorityClass scopes restrict a quota to tracking the following resources:

    • pods
    • cpu
    • memory
    • requests.cpu
    • requests.memory
    • limits.cpu
    • limits.memory

    Note that you cannot specify both the Terminating and the NotTerminating scopes in the same quota, and you cannot specify both the BestEffort and NotBestEffort scopes in the same quota either.

    The scopeSelector supports the following values in the operator field:

    • In
    • NotIn
    • Exists
    • DoesNotExist

    When using one of the following values as the scopeName when defining the scopeSelector, the operator must be Exists.

    • Terminating
    • NotTerminating
    • BestEffort
    • NotBestEffort

    If the operator is In or NotIn, the values field must have at least one value. For example:

    If the operator is Exists or DoesNotExist, the values field must NOT be specified.

    FEATURE STATE: Kubernetes v1.17 [stable]

    Pods can be created at a specific . You can control a pod’s consumption of system resources based on a pod’s priority, by using the scopeSelector field in the quota spec.

    A quota is matched and consumed only if scopeSelector in the quota spec selects the pod.

    When quota is scoped for priority class using scopeSelector field, quota object is restricted to track only following resources:

    • pods
    • cpu
    • memory
    • ephemeral-storage
    • limits.cpu
    • limits.memory
    • limits.ephemeral-storage
    • requests.cpu
    • requests.memory
    • requests.ephemeral-storage

    This example creates a quota object and matches it with pods at specific priorities. The example works as follows:

    • Pods in the cluster have one of the three priority classes, “low”, “medium”, “high”.
    • One quota object is created for each priority.

    Save the following YAML to a file .

    1. apiVersion: v1
    2. kind: List
    3. items:
    4. - apiVersion: v1
    5. kind: ResourceQuota
    6. metadata:
    7. name: pods-high
    8. spec:
    9. hard:
    10. cpu: "1000"
    11. memory: 200Gi
    12. pods: "10"
    13. scopeSelector:
    14. matchExpressions:
    15. - operator : In
    16. scopeName: PriorityClass
    17. values: ["high"]
    18. - apiVersion: v1
    19. kind: ResourceQuota
    20. metadata:
    21. name: pods-medium
    22. spec:
    23. hard:
    24. cpu: "10"
    25. memory: 20Gi
    26. pods: "10"
    27. scopeSelector:
    28. matchExpressions:
    29. - operator : In
    30. scopeName: PriorityClass
    31. values: ["medium"]
    32. - apiVersion: v1
    33. metadata:
    34. name: pods-low
    35. spec:
    36. hard:
    37. cpu: "5"
    38. memory: 10Gi
    39. pods: "10"
    40. scopeSelector:
    41. matchExpressions:
    42. - operator : In
    43. scopeName: PriorityClass
    44. values: ["low"]

    Apply the YAML using kubectl create.

    1. kubectl create -f ./quota.yml
    1. resourcequota/pods-high created
    2. resourcequota/pods-medium created
    3. resourcequota/pods-low created

    Verify that Used quota is 0 using kubectl describe quota.

    1. kubectl describe quota
    1. Name: pods-high
    2. Namespace: default
    3. Resource Used Hard
    4. -------- ---- ----
    5. cpu 0 1k
    6. memory 0 200Gi
    7. pods 0 10
    8. Name: pods-low
    9. Namespace: default
    10. Resource Used Hard
    11. -------- ---- ----
    12. cpu 0 5
    13. memory 0 10Gi
    14. pods 0 10
    15. Name: pods-medium
    16. Namespace: default
    17. Resource Used Hard
    18. -------- ---- ----
    19. cpu 0 10
    20. memory 0 20Gi
    21. pods 0 10
    1. apiVersion: v1
    2. kind: Pod
    3. metadata:
    4. name: high-priority
    5. spec:
    6. containers:
    7. - name: high-priority
    8. image: ubuntu
    9. command: ["/bin/sh"]
    10. args: ["-c", "while true; do echo hello; sleep 10;done"]
    11. resources:
    12. requests:
    13. memory: "10Gi"
    14. cpu: "500m"
    15. limits:
    16. memory: "10Gi"
    17. cpu: "500m"
    18. priorityClassName: high

    Apply it with kubectl create.

    1. kubectl create -f ./high-priority-pod.yml

    Verify that “Used” stats for “high” priority quota, pods-high, has changed and that the other two quotas are unchanged.

    1. kubectl describe quota
    1. Name: pods-high
    2. Namespace: default
    3. Resource Used Hard
    4. -------- ---- ----
    5. cpu 500m 1k
    6. memory 10Gi 200Gi
    7. pods 1 10
    8. Name: pods-low
    9. Namespace: default
    10. Resource Used Hard
    11. -------- ---- ----
    12. cpu 0 5
    13. memory 0 10Gi
    14. pods 0 10
    15. Name: pods-medium
    16. Namespace: default
    17. Resource Used Hard
    18. -------- ---- ----
    19. cpu 0 10
    20. memory 0 20Gi

    FEATURE STATE: Kubernetes v1.22 [beta]

    Operators can use CrossNamespacePodAffinity quota scope to limit which namespaces are allowed to have pods with affinity terms that cross namespaces. Specifically, it controls which pods are allowed to set namespaces or namespaceSelector fields in pod affinity terms.

    Preventing users from using cross-namespace affinity terms might be desired since a pod with anti-affinity constraints can block pods from all other namespaces from getting scheduled in a failure domain.

    Using this scope operators can prevent certain namespaces (foo-ns in the example below) from having pods that use cross-namespace pod affinity by creating a resource quota object in that namespace with CrossNamespaceAffinity scope and hard limit of 0:

    If operators want to disallow using namespaces and namespaceSelector by default, and only allow it for specific namespaces, they could configure CrossNamespaceAffinity as a limited resource by setting the kube-apiserver flag —admission-control-config-file to the path of the following configuration file:

    1. apiVersion: apiserver.config.k8s.io/v1
    2. kind: AdmissionConfiguration
    3. plugins:
    4. - name: "ResourceQuota"
    5. configuration:
    6. apiVersion: apiserver.config.k8s.io/v1
    7. kind: ResourceQuotaConfiguration
    8. limitedResources:
    9. matchScopes:
    10. - scopeName: CrossNamespaceAffinity

    With the above configuration, pods can use namespaces and namespaceSelector in pod affinity only if the namespace where they are created have a resource quota object with CrossNamespaceAffinity scope and a hard limit greater than or equal to the number of pods using those fields.

    This feature is beta and enabled by default. You can disable it using the feature gate PodAffinityNamespaceSelector in both kube-apiserver and kube-scheduler.

    Requests compared to Limits

    When allocating compute resources, each container may specify a request and a limit value for either CPU or memory. The quota can be configured to quota either value.

    If the quota has a value specified for requests.cpu or requests.memory, then it requires that every incoming container makes an explicit request for those resources. If the quota has a value specified for limits.cpu or limits.memory, then it requires that every incoming container specifies an explicit limit for those resources.

    Kubectl supports creating, updating, and viewing quotas:

    1. kubectl create namespace myspace
    1. cat <<EOF > compute-resources.yaml
    2. apiVersion: v1
    3. kind: ResourceQuota
    4. metadata:
    5. name: compute-resources
    6. spec:
    7. hard:
    8. requests.cpu: "1"
    9. requests.memory: 1Gi
    10. limits.cpu: "2"
    11. limits.memory: 2Gi
    12. requests.nvidia.com/gpu: 4
    13. EOF
    1. kubectl create -f ./compute-resources.yaml --namespace=myspace
    1. cat <<EOF > object-counts.yaml
    2. apiVersion: v1
    3. kind: ResourceQuota
    4. metadata:
    5. name: object-counts
    6. spec:
    7. hard:
    8. configmaps: "10"
    9. persistentvolumeclaims: "4"
    10. pods: "4"
    11. replicationcontrollers: "20"
    12. secrets: "10"
    13. services: "10"
    14. services.loadbalancers: "2"
    15. EOF
    1. kubectl create -f ./object-counts.yaml --namespace=myspace
    1. kubectl get quota --namespace=myspace
    1. NAME AGE
    2. compute-resources 30s
    3. object-counts 32s
    1. kubectl describe quota compute-resources --namespace=myspace
    1. kubectl describe quota object-counts --namespace=myspace
    1. Name: object-counts
    2. Namespace: myspace
    3. Resource Used Hard
    4. -------- ---- ----
    5. configmaps 0 10
    6. persistentvolumeclaims 0 4
    7. pods 0 4
    8. replicationcontrollers 0 20
    9. secrets 1 10
    10. services 0 10
    11. services.loadbalancers 0 2

    Kubectl also supports object count quota for all standard namespaced resources using the syntax count/<resource>.<group>:

    1. kubectl create namespace myspace
    1. kubectl create quota test --hard=count/deployments.apps=2,count/replicasets.apps=4,count/pods=3,count/secrets=4 --namespace=myspace
    1. kubectl create deployment nginx --image=nginx --namespace=myspace --replicas=2
    1. kubectl describe quota --namespace=myspace
    1. Name: test
    2. Namespace: myspace
    3. Resource Used Hard
    4. -------- ---- ----
    5. count/deployments.apps 1 2
    6. count/pods 2 3
    7. count/replicasets.apps 1 4
    8. count/secrets 1 4

    Quota and Cluster Capacity

    ResourceQuotas are independent of the cluster capacity. They are expressed in absolute units. So, if you add nodes to your cluster, this does not automatically give each namespace the ability to consume more resources.

    Sometimes more complex policies may be desired, such as:

    • Proportionally divide total cluster resources among several teams.
    • Allow each tenant to grow resource usage as needed, but have a generous limit to prevent accidental resource exhaustion.
    • Detect demand from one namespace, add nodes, and increase quota.

    Such policies could be implemented using ResourceQuotas as building blocks, by writing a “controller” that watches the quota usage and adjusts the quota hard limits of each namespace according to other signals.

    Note that resource quota divides up aggregate cluster resources, but it creates no restrictions around nodes: pods from several namespaces may run on the same node.

    Limit Priority Class consumption by default

    It may be desired that pods at a particular priority, eg. “cluster-services”, should be allowed in a namespace, if and only if, a matching quota object exists.

    With this mechanism, operators are able to restrict usage of certain high priority classes to a limited number of namespaces and not every namespace will be able to consume these priority classes by default.

    To enforce this, kube-apiserver flag --admission-control-config-file should be used to pass path to the following configuration file:

    1. apiVersion: apiserver.config.k8s.io/v1
    2. kind: AdmissionConfiguration
    3. plugins:
    4. - name: "ResourceQuota"
    5. configuration:
    6. apiVersion: apiserver.config.k8s.io/v1
    7. kind: ResourceQuotaConfiguration
    8. limitedResources:
    9. - resource: pods
    10. matchScopes:
    11. - scopeName: PriorityClass
    12. operator: In
    13. values: ["cluster-services"]

    Then, create a resource quota object in the kube-system namespace:

    policy/priority-class-resourcequota.yaml

    1. apiVersion: v1
    2. kind: ResourceQuota
    3. metadata:
    4. name: pods-cluster-services
    5. spec:
    6. scopeSelector:
    7. matchExpressions:
    8. - operator : In
    9. scopeName: PriorityClass
    10. values: ["cluster-services"]
    1. resourcequota/pods-cluster-services created

    In this case, a pod creation will be allowed if:

    1. the Pod’s priorityClassName is not specified.
    2. the Pod’s priorityClassName is specified to a value other than cluster-services.
    3. the Pod’s priorityClassName is set to cluster-services, it is to be created in the kube-system namespace, and it has passed the resource quota check.

    A Pod creation request is rejected if its priorityClassName is set to cluster-services and it is to be created in a namespace other than kube-system.