Development Tips

    Since we are interested in using Ansible for the lifecycle management of our application on Kubernetes, it is beneficial for a developer to get a good grasp of the Kubernetes Collection for Ansible. This Ansible collection allows a developer to either leverage their existing Kubernetes resource files (written in YAML) or express the lifecycle management in native Ansible. One of the biggest benefits of using Ansible in conjunction with existing Kubernetes resource files is the ability to use Jinja templating so that you can customize deployments with the simplicity of a few variables in Ansible.

    The easiest way to get started is to install the collection on your local machine and test it using a playbook.

    To install the Kubernetes Collection, one must first install Ansible 2.9+. For example, on Fedora/Centos:

    In addition to Ansible, a user must install the package:

    Finally, install the Kubernetes Collection from ansible-galaxy:

    1. $ ansible-galaxy collection install community.kubernetes

    Alternatively, if you’ve already initialized your operator, you may have a requirements.yml file at the top level of your project. This file specifies Ansible dependencies that need to be installed for your operator to function. By default it will install the community.kubernetes collection as well as the operator_sdk.util collection, which provides modules and plugins for operator-specific operations.

    To install the dependent modules from this file, run:

    1. $ ansible-galaxy collection install -r requirements.yml

    Testing the Kubernetes Collection locally

    Sometimes it is beneficial for a developer to run the Ansible code from their local machine as opposed to running/rebuilding the operator each time. To do this, initialize a new project:

    1. $ mkdir foo-operator && cd foo-operator
    2. $ operator-sdk init --plugins=ansible --domain=example.com --group=foo --version=v1alpha1 --kind=Foo --generate-role
    3. $ ansible-galaxy collection install -r requirements.yml

    Modify roles/foo/tasks/main.yml with desired Ansible logic. For this example we will create and delete a ConfigMap based on the value of a variable named state:

    1. ---
    2. - name: set ConfigMap example-config to {{ state }}
    3.   community.kubernetes.k8s:
    4.     api_version: v1
    5.     kind: ConfigMap
    6.     name: example-config
    7.     namespace: default
    8.     state: "{{ state }}"
    9.   ignore_errors: true

    Note

    Setting ignore_errors: true is done so that deleting a nonexistent ConfigMap doesn’t error out.

    Modify roles/foo/defaults/main.yml to set state to present as default.

    1. ---
    2. state: present

    Create an Ansible playbook playbook.yml in the top-level directory which includes role foo:

    1. ---
    2. - hosts: localhost
    3.   roles:
    4.     - foo

    Run the playbook:

    1. $ ansible-playbook playbook.yml
    2.  [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'
    5. PLAY [localhost] ***************************************************************************
    7. TASK [Gathering Facts] *********************************************************************
    8. ok: [localhost]
    10. Task [foo : set example-coonfig configmap to present]
    11. changed: [localhost]
    13. PLAY RECAP *********************************************************************************
    14. localhost                  : ok=2    changed=1    unreachable=0    failed=0

    Check that the ConfigMap was created:

    1. $ kubectl get configmaps
    2. NAME                    STATUS    AGE
    3. example-config          Active    3s

    Rerun the playbook setting state to absent:

    1. $ ansible-playbook playbook.yml --extra-vars state=absent
    2.  [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'
    5. PLAY [localhost] ***************************************************************************
    7. TASK [Gathering Facts] *********************************************************************
    8. ok: [localhost]
    9. Task [foo : set example-config configmap to absent]
    10. changed: [localhost]
    12. PLAY RECAP *********************************************************************************
    13. localhost                  : ok=2    changed=1    unreachable=0    failed=0

    Check that the ConfigMap was deleted:

    Now that we have demonstrated using the Kubernetes Collection, we want to

    The Custom Resource (CR) file format is Kubernetes resource file. The object has some mandatory fields:

    • apiVersion: The version of the Custom Resource that will be created.
    • kind: The kind of the Custom Resource that will be created
    • metadata: Kubernetes specific metadata to be created
    • spec: This is the key-value list of variables which are passed to Ansible. This field is optional and empty by default.
    • annotations: Kubernetes specific annotations to be appended to the CR. See the below section for Ansible Operator specific annotations. This field is optional.

    Annotations for Custom Resource

    This is the list of CR annotations which will modify the behavior of the operator:

    • ansible.operator-sdk/reconcile-period: Specifies the maximum time before a reconciliation is triggered. Note that at scale, this can reduce performance, see reference for more information. This value is parsed using the standard Golang package time. Specifically is used which will apply the default suffix of s giving the value in seconds.

      Example:

      1. apiVersion: foo.example.com/v1alpha1
      2. kind: Foo
      4.   name: example
      5.   annotations:
      6.     ansible.operator-sdk/reconcile-period: "30s"

    Note that a lower period will correct entropy more quickly, but reduce responsiveness to change if there are many watched resources. Typically, this option should only be used in advanced use cases where watchDependentResources is set to False and when is not possible to use the watch feature. E.g To managing external resources that don’t raise Kubernetes events.

    Testing an Ansible Operator locally

    Prerequisites: Ensure that and Ansible Runner HTTP Plugin is installed or else you will see unexpected errors from Ansible Runner when a Custom Resource is created.

    Once a developer is comfortable working with the above workflow, it will be beneficial to test the logic inside an operator. To accomplish this, you can use make run from the top-level directory of our project. The make run Makefile target runs the ansible-operator binary locally, which reads from ./watches.yaml and uses ~/.kube/config to communicate with a Kubernetes cluster just as the k8s modules do. This section assumes the developer has read the and has the proper dependencies installed.

    Note

    You can customize the roles path by setting the environment variable ANSIBLE_ROLES_PATH or using the flag ansible-roles-path. Note that if the role is not found in ANSIBLE_ROLES_PATH, then the operator will look for it in {{current directory}}/roles.

    Create a Custom Resource Definition (CRD) and proper Role-Based Access Control (RBAC) definitions for resource Foo.

    1. $ make install

    Run the make run command:

    1. $ make run
    2. /home/user/go/bin/ansible-operator
    3. {"level":"info","ts":1595899073.9861593,"logger":"cmd","msg":"Version","Go Version":"go1.13.12","GOOS":"linux","GOARCH":"amd64","ansible-operator":"v0.19.0+git"}
    4. {"level":"info","ts":1595899073.987384,"logger":"cmd","msg":"WATCH_NAMESPACE environment variable not set. Watching all namespaces.","Namespace":""}
    5. {"level":"info","ts":1595899074.9504397,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
    6. {"level":"info","ts":1595899074.9522583,"logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_MEMCACHED_CACHE_EXAMPLE_COM","default":2}
    7. {"level":"info","ts":1595899074.9524004,"logger":"cmd","msg":"Environment variable not set; using default value","Namespace":"","envVar":"ANSIBLE_DEBUG_LOGS","ANSIBLE_DEBUG_LOGS":false}
    8. {"level":"info","ts":1595899074.9524298,"logger":"ansible-controller","msg":"Watching resource","Options.Group":"cache.example.com","Options.Version":"v1","Options.Kind":"Memcached"}

    Now that the operator is watching resource Foo for events, the creation of a Custom Resource will trigger our Ansible Role to be executed. Take a look at config/samples/foo_v1alpha1_foo.yaml:

    1. apiVersion: foo.example.com/v1alpha1
    2. kind: Foo
    3. metadata:
    4.   name: "foo-sample"

    Since spec is not set, Ansible is invoked with no extra variables. The next section covers how extra variables are passed from a Custom Resource to Ansible. This is why it is important to set sane defaults for the operator.

    Create a Custom Resource instance of Foo with variable state default to present:

    1. $ kubectl create -f config/samples/foo_v1alpha1_foo.yaml

    Check that ConfigMap example-config was created:

    1. $ kubectl get configmaps
    2. NAME                    STATUS    AGE
    3. example-config          Active    3s

    Modify deploy/crds/foo.example.com_v1alpha1_foo_cr.yaml to set state to absent:

    1. apiVersion: foo.example.com/v1alpha1
    2. kind: Foo
    3. metadata:
    4.   name: foo-sample
    5. spec:
    6.   state: absent

    Apply the changes to Kubernetes and confirm that the ConfiMap is deleted:

    1. $ kubectl apply -f config/samples/foo_v1alpha1_foo.yaml
    2. $ kubectl get configmaps

    Now that a developer is confident in the operator logic, testing the operator inside of a pod on a Kubernetes cluster is desired. Running as a pod inside a Kubernetes cluster is preferred for production use.

    1. $ make docker-build docker-push IMG=quay.io/example/foo-operator:v0.0.1

    Deploy the foo-operator:

    1. $ make install
    2. $ make deploy IMG=quay.io/example/foo-operator:v0.0.1

    Verify that the foo-operator is up and running:

    Viewing the Ansible logs

    In order to see the logs from a particular operator you can run:

    1. kubectl logs deployment/foo-operator-controller-manager -n foo-operator-system

    The logs contain the information about the Ansible run and are useful for debugging your Ansible tasks. Note that the logs may contain much more detailed information about the Ansible Operator’s internals and its interactions with Kubernetes as well.

    Also, you can set the environment variable ANSIBLE_DEBUG_LOGS to True to check the full Ansible result in the logs in order to be able to debug it.

    Example

    In config/manager/manager.yaml and config/default/manager_auth_proxy_patch.yaml:

    1. ...
    2.       containers:
    3.       - name: manager
    4.         env:
    5.         - name: ANSIBLE_DEBUG_LOGS
    6.           value: "True"
    7. ...

    Occasionally while developing additional debug in the Operator logs is nice to have. Using the memcached operator as an example, we can simply add the "ansible.sdk.operatorframework.io/verbosity" annotation to the Custom Resource with the desired verbosity.

    1. apiVersion: "cache.example.com/v1alpha1"
    2. kind: "Memcached"
    3. metadata:
    4.   name: "example-memcached"
    5.   annotations:
    6.     "ansible.sdk.operatorframework.io/verbosity": "4"
    7. spec:

    By default, an Ansible Operator will include the generic output from previous Ansible run as the status subresource of a CR. This includes the number of successful and failed tasks and relevant error messages as shown below:

    1. status:
    2.   conditions:
    3.   - ansibleResult:
    4.       changed: 3
    5.       completion: 2018-12-03T13:45:57.13329
    6.       failures: 1
    7.       ok: 6
    8.       skipped: 0
    9.     lastTransitionTime: 2018-12-03T13:45:57Z
    10.     message: 'Status code was -1 and not [200]: Request failed: <urlopen error [Errno
    11.       113] No route to host>'
    12.     reason: Failed
    13.     status: "True"
    14.     type: Failure
    15.   - lastTransitionTime: 2018-12-03T13:46:13Z
    16.     message: Running reconciliation
    17.     reason: Running
    18.     status: "True"
    19.     type: Running

    An Ansible Operator also allows you to supply custom status values with the k8s_status Ansible module, which is included in operator_sdk.util collection. You can update the status from within Ansible with any key/value pairs as desired. If you do not want the operator to update the status with Ansible output, and you want to track the CR status manually from your application, you can update the watches.yaml file with manageStatus, as shown below:

    1. - version: v1
    2.   group: api.example.com
    3.   kind: Foo
    4.   role: foo
    5.   manageStatus: false

    The simplest way to invoke the k8s_status module is to use its fully qualified collection name (fqcn), i.e. operator_sdk.util.k8s_status. The following example updates the status subresource with key foo and value bar:

    1. - operator_sdk.util.k8s_status:
    2.     api_version: app.example.com/v1
    3.     kind: Foo
    4.     name: "{{ ansible_operator_meta.name }}"
    5.     namespace: "{{ ansible_operator_meta.namespace }}"
    6.     status:
    7.       foo: bar

    Collections can also be declared in the role’s meta/main.yml, which is included for newly scaffolded Ansible operators.

    1. collections:
    2.   - operator_sdk.util

    Declaring collections in the role meta allows you to invoke the k8s_status module directly.

    1. - k8s_status:
    2.     <snip>
    3.     status:
    4.       foo: bar

    An Ansible Operator has a set of conditions that are used during reconciliation. There are only a few main conditions:

    • Running - the Ansible Operator is currently running the Ansible for reconciliation.
    • Successful - if the run has finished and there were no errors, the Ansible Operator will be marked as Successful. It will then wait for the next reconciliation action, either the reconcile period, dependent watches triggers or the resource is updated.
    • Failed - if there is any error during the reconciliation run, the Ansible Operator will be marked as Failed with the error message from the error that caused this condition. The error message is the raw output from the Ansible run for reconciliation. If the Failure is intermittent, often times the situation can be resolved when the Operator reruns the reconciliation loop.

    The extra vars that are sent to Ansible are managed by the operator. The spec section will pass along the key-value pairs as extra vars. This is equivalent to how above extra vars are passed in to ansible-playbook. The operator also passes along additional variables under the ansible_operator_meta field for the name of the CR and the namespace of the CR.

    For the CR example:

    1. apiVersion: "foo.example.com/v1alpha1"
    2. kind: "Foo"
    3. metadata:
    4.   name: "foo-sample"
    5. spec:
    6.   message: "Hello world 2"
    7.   newParameter: "newParam"

    The structure passed to Ansible as extra vars is:

    1. { "ansible_operator_meta": {
    2.         "name": "<cr-name>",
    3.         "namespace": "<cr-namespace>",
    4.   },
    5.   "message": "Hello world 2",
    6.   "new_parameter": "newParam",
    7.   "_app_example_com_database": {
    8.      <Full CR>
    9.    },
    10.   "_app_example_com_database_spec": {
    11.      <Full CR .spec>
    12.    },
    13. }