Updating OpenShift Logging

To upgrade from cluster logging in OKD version 4.6 and earlier to OpenShift Logging 5.x, you update the OKD cluster to version 4.7 or 4.8. Then, you update the following operators:

From Elasticsearch Operator 4.x to OpenShift Elasticsearch Operator 5.x
From Cluster Logging Operator 4.x to Red Hat OpenShift Logging Operator 5.x

To upgrade from a previous version of OpenShift Logging to the current version, you update OpenShift Elasticsearch Operator and Red Hat OpenShift Logging Operator to their current versions.

OKD 4.7 made the following name changes:

The cluster logging feature became the Red Hat OpenShift Logging 5.x product.
The Cluster Logging Operator became the Red Hat OpenShift Logging Operator.
The Elasticsearch Operator became OpenShift Elasticsearch Operator.

To upgrade from cluster logging in OKD version 4.6 and earlier to OpenShift Logging 5.x, you update the OKD cluster to version 4.7 or 4.8. Then, you update the following operators:

From Elasticsearch Operator 4.x to OpenShift Elasticsearch Operator 5.x
From Cluster Logging Operator 4.x to Red Hat OpenShift Logging Operator 5.x

If you update the operators in the wrong order, Kibana does not update and the Kibana custom resource (CR) is not created. To work around this problem, you delete the Red Hat OpenShift Logging Operator pod. When the Red Hat OpenShift Logging Operator pod redeploys, it creates the Kibana CR and Kibana becomes available again.

Prerequisites

The OKD version is 4.7 or later.
The OpenShift Logging status is healthy:
- All pods are .
- The Elasticsearch cluster is healthy.
Your Elasticsearch and Kibana data is backed up.

Procedure

Update the OpenShift Elasticsearch Operator:
1. From the web console, click Operators → Installed Operators.
2. Select the openshift-operators-redhat project.
3. Click the OpenShift Elasticsearch Operator.
4. Click Subscription → Channel.
5. In the Change Subscription Update Channel window, select 5.0 or stable-5.x and click Save.
6. Wait for a few seconds, then click Operators → Installed Operators.
  
  Verify that the OpenShift Elasticsearch Operator version is 5.x.x.
  
  Wait for the Status field to report Succeeded.
Update the Cluster Logging Operator:
2. Select the openshift-logging project.
3. Click the Cluster Logging Operator.
4. Click Subscription → Channel.
5. In the Change Subscription Update Channel window, select 5.0 or stable-5.x and click Save.
6. Wait for a few seconds, then click Operators → Installed Operators.
  
  Verify that the Red Hat OpenShift Logging Operator version is 5.0.x or 5.x.x.
  
  Wait for the Status field to report Succeeded.

Check the logging components:

Ensure that all Elasticsearch pods are in the Ready status:

Example output

NAME                                            READY   STATUS    RESTARTS   AGE
elasticsearch-cdm-1pbrl44l-1-55b7546f4c-mshhk   2/2     Running   0          31m
elasticsearch-cdm-1pbrl44l-2-5c6d87589f-gx5hk   2/2     Running   0          30m
elasticsearch-cdm-1pbrl44l-3-88df5d47-m45jc     2/2     Running   0          29m

Ensure that the Elasticsearch cluster is healthy:

$ oc exec -n openshift-logging -c elasticsearch elasticsearch-cdm-1pbrl44l-1-55b7546f4c-mshhk -- health

{
  "cluster_name" : "elasticsearch",
  "status" : "green",
}

Ensure that the Elasticsearch cron jobs are created:

$ oc project openshift-logging

$ oc get cronjob

NAME                     SCHEDULE       SUSPEND   ACTIVE   LAST SCHEDULE   AGE
elasticsearch-im-app     */15 * * * *   False     0        <none>          56s
elasticsearch-im-audit   */15 * * * *   False     0        <none>          56s
elasticsearch-im-infra   */15 * * * *   False     0        <none>          56s

Verify that the log store is updated to 5.0 or 5.x and the indices are green:
```
$ oc exec -c elasticsearch <any_es_pod_in_the_cluster> -- indices
```
Verify that the output includes the app-00000x, infra-00000x, audit-00000x, .security indices.

Sample output with indices in a green status
Verify that the log collector is updated to 5.0 or 5.x:
```
$ oc get ds fluentd -o json | grep fluentd-init
```
Verify that the output includes a fluentd-init container:
```
"containerName": "fluentd-init"
```

Verify that the log visualizer is updated to 5.0 or 5.x using the Kibana CRD:

$ oc get kibana kibana -o json

Verify that the output includes a Kibana pod with the ready status:

Sample output with a ready Kibana pod

[
{
"clusterCondition": {
"kibana-5fdd766ffd-nb2jj": [
{
"reason": "ContainerCreating",
"status": "True",
"type": ""
},
{
"lastTransitionTime": "2020-06-30T14:11:07Z",
"reason": "ContainerCreating",
"status": "True",
"type": ""
}
]
"deployment": "kibana",
"pods": {
"failed": [],
"notReady": []
"ready": []
},
"replicaSets": [
"kibana-5fdd766ffd"
],
"replicas": 1
}
]