Using A Manifest to Manage kOps Clusters

    Background

    Because of the above statement kops includes an API which provides a feature for users to utilize YAML or JSON manifests for managing their kops created Kubernetes installations. In the same way that you can use a YAML manifest to deploy a Job, you can deploy and manage a kops Kubernetes instance with a manifest. All of these values are also usable via the interactive editor with kops edit.

    You can see all the options that are currently supported in kOps or more prettily here

    The following is a list of the benefits of using a file to manage instances.

    • Capability to access API values that are not accessible via the command line such as setting the max price for spot instances.
    • Create, replace, update, and delete clusters without entering an interactive editor. This feature is helpful when automating cluster creation.
    • Ability to check-in files to source control that represents an installation.
    • Run commands such as kops delete -f mycluster.yaml.

    At this time you must run kops create cluster and then export the YAML from the state store. We plan in the future to have the capability to generate kOps YAML via the command line. The following is an example of creating a cluster and exporting the YAML.

    The above command exports a YAML document which contains the definition of the cluster, kind: Cluster, and the definitions of the instance groups, kind: InstanceGroup.

    NOTE: If you run kops get cluster $NAME -o yaml > $NAME.yaml, you will only get a cluster spec. Use the command above (kops get $NAME ...)for both the cluster spec and all instance groups.

    1. apiVersion: kops.k8s.io/v1alpha2
    2. kind: Cluster
    3. metadata:
    4.   name: k8s.example.com
    5. spec:
    6.   api:
    7.     loadBalancer:
    8.       type: Public
    9.   authorization:
    10.     alwaysAllow: {}
    11.   channel: stable
    12.   cloudProvider: aws
    13.   configBase: s3://example-state-store/k8s.example.com
    14.   etcdClusters:
    15.   - etcdMembers:
    16.     - instanceGroup: master-us-east-2d
    17.       name: a
    18.     - instanceGroup: master-us-east-2b
    19.       name: b
    20.     - instanceGroup: master-us-east-2c
    21.       name: c
    22.     name: main
    23.   - etcdMembers:
    24.     - instanceGroup: master-us-east-2d
    25.       name: a
    26.     - instanceGroup: master-us-east-2b
    27.       name: b
    28.     - instanceGroup: master-us-east-2c
    29.       name: c
    30.     name: events
    31.   kubernetesApiAccess:
    32.   - 0.0.0.0/0
    33.   kubernetesVersion: 1.6.6
    34.   masterPublicName: api.k8s.example.com
    35.   networkCIDR: 172.20.0.0/16
    36.   networkID: vpc-6335dd1a
    37.   networking:
    38.     weave: {}
    39.   nonMasqueradeCIDR: 100.64.0.0/10
    40.   sshAccess:
    41.   - 0.0.0.0/0
    42.   subnets:
    43.   - cidr: 172.20.32.0/19
    44.     name: us-east-2d
    45.     type: Private
    46.     zone: us-east-2d
    47.   - cidr: 172.20.64.0/19
    48.     name: us-east-2b
    49.     type: Private
    50.   - cidr: 172.20.96.0/19
    51.     name: us-east-2c
    52.     type: Private
    53.     zone: us-east-2c
    54.   - cidr: 172.20.0.0/22
    56.     type: Utility
    57.     zone: us-east-2d
    58.   - cidr: 172.20.4.0/22
    59.     name: utility-us-east-2b
    60.     type: Utility
    61.     zone: us-east-2b
    62.   - cidr: 172.20.8.0/22
    63.     name: utility-us-east-2c
    64.     type: Utility
    65.     zone: us-east-2c
    66.   topology:
    67.     bastion:
    68.       bastionPublicName: bastion.k8s.example.com
    69.     dns:
    70.       type: Public
    71.     masters: private
    72.     nodes: private
    74. ---
    76. apiVersion: kops.k8s.io/v1alpha2
    77. kind: InstanceGroup
    78. metadata:
    79.   labels:
    80.     kops.k8s.io/cluster: k8s.example.com
    81.   name: bastions
    82. spec:
    83.   image: kope.io/k8s-1.6-debian-jessie-amd64-hvm-ebs-2017-05-02
    84.   machineType: t2.micro
    85.   maxSize: 1
    86.   minSize: 1
    87.   role: Bastion
    88.   subnets:
    89.   - utility-us-east-2d
    90.   - utility-us-east-2b
    91.   - utility-us-east-2c
    94. ---
    96. apiVersion: kops.k8s.io/v1alpha2
    97. kind: InstanceGroup
    98. metadata:
    99.   labels:
    100.     kops.k8s.io/cluster: k8s.example.com
    101.   name: master-us-east-2d
    102. spec:
    103.   image: kope.io/k8s-1.6-debian-jessie-amd64-hvm-ebs-2017-05-02
    104.   machineType: m4.large
    105.   maxSize: 1
    106.   minSize: 1
    107.   role: Master
    108.   subnets:
    109.   - us-east-2d
    112. ---
    114. apiVersion: kops.k8s.io/v1alpha2
    115. kind: InstanceGroup
    116. metadata:
    117.   labels:
    118.     kops.k8s.io/cluster: k8s.example.com
    119. spec:
    120.   image: kope.io/k8s-1.6-debian-jessie-amd64-hvm-ebs-2017-05-02
    122.   maxSize: 1
    123.   minSize: 1
    124.   role: Master
    125.   subnets:
    126.   - us-east-2b
    129. ---
    131. apiVersion: kops.k8s.io/v1alpha2
    132. kind: InstanceGroup
    133. metadata:
    134.   labels:
    135.     kops.k8s.io/cluster: k8s.example.com
    136.   name: master-us-east-2c
    137. spec:
    138.   image: kope.io/k8s-1.6-debian-jessie-amd64-hvm-ebs-2017-05-02
    139.   machineType: m4.large
    140.   maxSize: 1
    141.   minSize: 1
    142.   role: Master
    143.   subnets:
    144.   - us-east-2c
    147. ---
    149. apiVersion: kops.k8s.io/v1alpha2
    150. kind: InstanceGroup
    151. metadata:
    152.   labels:
    153.     kops.k8s.io/cluster: k8s.example.com
    154.   name: nodes
    155. spec:
    156.   image: kope.io/k8s-1.6-debian-jessie-amd64-hvm-ebs-2017-05-02
    157.   machineType: m4.xlarge
    158.   maxSize: 3
    159.   minSize: 3
    160.   role: Node
    161.   subnets:
    162.   - us-east-2d
    163.   - us-east-2b
    164.   - us-east-2c

    YAML Examples

    With the above YAML file, a user can add configurations that are not available via the command line. For instance, you can add a maxPrice value to a new instance group and use spot instances. Also add node and cloud labels for the new instance group.

    This configuration will create an autoscale group that will include 42 m4.10xlarge nodes running as spot instances with custom labels.

    To create the cluster execute:

    1. kops create -f $NAME.yaml
    2. kops create secret --name $NAME sshpublickey admin -i ~/.ssh/id_rsa.pub
    3. kops update cluster $NAME --yes
    4. kops rolling-update cluster $NAME --yes

    Please refer to the rolling-update documentation.

    Update the cluster spec YAML file, and to update the cluster run:

    Please refer to the rolling-update .

    kops implements a full API that defines the various elements in the YAML file exported above. Two top level components exist; ClusterSpec and InstanceGroup.

    1. apiVersion: kops.k8s.io/v1alpha2
    2. kind: Cluster
    3. metadata:
    4.   name: k8s.example.com
    5. spec:
    6.   api:

    The ClusterSpec allows a user to set configurations for such values as Docker log driver, Kubernetes API server log level, VPC for reusing a VPC (NetworkID), and the Kubernetes version.

    More information about some of the elements in the ClusterSpec is available in the following:

    • Cluster Spec document which outlines some of the values in the Cluster Specification.
    • GPU setup
    • - adding additional instance IAM roles.
    • Labels

    To access the full configuration that a kops installation is running execute:

    This command prints the entire YAML configuration. But do not use the full document, you may experience strange and unique unwanted behaviors.

    Instance Groups

    1. apiVersion: kops.k8s.io/v1alpha2
    2. kind: InstanceGroup
    3. metadata:
    4.   name: foo
    5. spec:

    Full documentation is accessible via .

    Instance Groups map to Auto Scaling Groups in AWS, and Instance Groups in GCE. They are an API level description of a group of compute instances used as Masters or Nodes.

    More documentation is available in the Instance Group document.

    Closing Thoughts

    • If you do not need to define or customize a value, let kOps set that value. Setting too many values prevents kOps from doing its job in setting up the cluster and you may end up with strange bugs.
    • If you end up with strange bugs, try letting kOps do more.

    If you need to run a custom version of Kubernetes Controller Manager, set and update your cluster. This is the beauty of using a manifest for your cluster!