Considerations for large clusters

    • No more than 110 pods per node
    • No more than 5,000 nodes
    • No more than 150,000 total pods

    You can scale your cluster by adding or removing nodes. The way you do this depends on how your cluster is deployed.

    To avoid running into cloud provider quota issues, when creating a cluster with many nodes, consider:

    • Requesting a quota increase for cloud resources such as:
      • Computer instances
      • CPUs
      • Storage volumes
      • In-use IP addresses
      • Packet filtering rule sets
      • Number of load balancers
      • Network subnets
    • Gating the cluster scaling actions to bring up new nodes in batches, with a pause between batches, because some cloud providers rate limit the creation of new instances.

    For a large cluster, you need a control plane with sufficient compute and other resources.

    Typically you would run one or two control plane instances per failure zone, scaling those instances vertically first and then scaling horizontally after reaching the point of falling returns to (vertical) scale.

    For example, using a managed load balancer, you configure the load balancer to send traffic that originates from the kubelet and Pods in failure zone A, and direct that traffic only to the control plane hosts that are also in zone A. If a single control-plane host or endpoint failure zone A goes offline, that means that all the control-plane traffic for nodes in zone A is now being sent between zones. Running multiple control plane hosts in each zone makes that outcome less likely.

    To improve performance of large clusters, you can store Event objects in a separate dedicated etcd instance.

    When creating a cluster, you can (using custom tooling):

    • start and configure additional etcd instance
    • configure the to use it for storing events

    See Operating etcd clusters for Kubernetes and for details on configuring and managing etcd for a large cluster.

    For example, you can set CPU and memory limits for a logging component:

    Addons’ default limits are typically based on data collected from experience running each addon on small or medium Kubernetes clusters. When running on large clusters, addons often consume more of some resources than their default limits. If a large cluster is deployed without adjusting these values, the addon(s) may continuously get killed because they keep hitting the memory limit. Alternatively, the addon may run but with poor performance due to CPU time slice restrictions.

    To avoid running into cluster addon resource issues, when creating a cluster with many nodes, consider the following:

    • Some addons scale vertically - there is one replica of the addon for the cluster or serving a whole failure zone. For these addons, increase requests and limits as you scale out your cluster.
    • Many addons scale horizontally - you add capacity by running more pods - but with a very large cluster you may also need to raise CPU or memory limits slightly. The VerticalPodAutoscaler can run in recommender mode to provide suggested figures for requests and limits.
    • Some addons run as one copy per node, controlled by a : for example, a node-level log aggregator. Similar to the case with horizontally-scaled addons, you may also need to raise CPU or memory limits slightly.

    • is a custom resource that you can deploy into your cluster to help you manage resource requests and limits for pods.
      Learn more about and how you can use it to scale cluster components, including cluster-critical addons.