Viewing OpenShift Logging status

    You can view the status of your Red Hat OpenShift Logging Operator.

    Prerequisites

    • The Red Hat OpenShift Logging and Elasticsearch Operators must be installed.

    Procedure

    1. Change to the project.

    2. To view the OpenShift Logging status:

      1. Get the OpenShift Logging status:

        1. $ oc get clusterlogging instance -o yaml

        Example output

        1. apiVersion: logging.openshift.io/v1
        2. kind: ClusterLogging
        4. ....
        6. status:  (1)
        7.   collection:
        8.     logs:
        9.       fluentdStatus:
        10.         daemonSet: fluentd  (2)
        11.         nodes:
        12.           fluentd-2rhqp: ip-10-0-169-13.ec2.internal
        13.           fluentd-6fgjh: ip-10-0-165-244.ec2.internal
        14.           fluentd-6l2ff: ip-10-0-128-218.ec2.internal
        15.           fluentd-54nx5: ip-10-0-139-30.ec2.internal
        16.           fluentd-flpnn: ip-10-0-147-228.ec2.internal
        17.           fluentd-n2frh: ip-10-0-157-45.ec2.internal
        18.         pods:
        19.           failed: []
        20.           notReady: []
        21.           ready:
        22.           - fluentd-2rhqp
        23.           - fluentd-54nx5
        24.           - fluentd-6fgjh
        25.           - fluentd-6l2ff
        26.           - fluentd-flpnn
        27.           - fluentd-n2frh
        28.   logstore: (3)
        29.     elasticsearchStatus:
        30.     - ShardAllocationEnabled:  all
        31.       cluster:
        32.         activePrimaryShards:    5
        33.         activeShards:           5
        34.         initializingShards:     0
        35.         numDataNodes:           1
        36.         numNodes:               1
        37.         pendingTasks:           0
        38.         relocatingShards:       0
        39.         status:                 green
        40.       clusterName:             elasticsearch
        41.       nodeConditions:
        42.         elasticsearch-cdm-mkkdys93-1:
        43.       nodeCount:  1
        45.         client:
        46.           failed:
        47.           notReady:
        48.           ready:
        49.           - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7c
        50.         data:
        51.           failed:
        52.           notReady:
        53.           ready:
        54.           - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7c
        55.         master:
        56.           failed:
        57.           notReady:
        58.           ready:
        59.           - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7c
        60. visualization:  (4)
        61.     kibanaStatus:
        62.     - deployment: kibana
        63.       pods:
        64.         failed: []
        65.         notReady: []
        66.         ready:
        67.         - kibana-7fb4fd4cc9-f2nls
        68.       replicaSets:
        69.       - kibana-7fb4fd4cc9
        70.       replicas: 1

    The following are examples of some condition messages from the Status.Nodes section of the OpenShift Logging instance.

    A status message similar to the following indicates a node has exceeded the configured low watermark and no shard will be allocated to this node:

    1.   nodes:
    2.   - conditions:
    3.     - lastTransitionTime: 2019-03-15T15:57:22Z
    4.       message: Disk storage usage for node is 27.5gb (36.74%). Shards will be not
    5.         be allocated on this node.
    6.       reason: Disk Watermark Low
    7.       status: "True"
    8.       type: NodeStorage
    9.     deploymentName: example-elasticsearch-clientdatamaster-0-1
    10.     upgradeStatus: {}

    A status message similar to the following indicates a node has exceeded the configured high watermark and shards will be relocated to other nodes:

    Example output

    1.   nodes:
    2.   - conditions:
    3.     - lastTransitionTime: 2019-03-15T16:04:45Z
    4.       message: Disk storage usage for node is 27.5gb (36.74%). Shards will be relocated
    6.       reason: Disk Watermark High
    7.       type: NodeStorage
    8.     deploymentName: cluster-logging-operator
    9.     upgradeStatus: {}

    A status message similar to the following indicates the Elasticsearch node selector in the CR does not match any nodes in the cluster:

    Example output

    A status message similar to the following indicates that the requested PVC could not bind to PV:

    Example output

    1.       Node Conditions:
    2.         elasticsearch-cdm-mkkdys93-1:
    3.           Last Transition Time:  2019-06-26T03:37:32Z
    4.           Message:               pod has unbound immediate PersistentVolumeClaims (repeated 5 times)
    5.           Reason:                Unschedulable
    6.           Status:                True
    7.           Type:                  Unschedulable

    A status message similar to the following indicates that the Fluentd pods cannot be scheduled because the node selector did not match any nodes:

    Example output

    1. Status:
    2.   Collection:
    3.     Logs:
    4.       Fluentd Status:
    5.         Daemon Set:  fluentd
    6.         Nodes:
    7.         Pods:
    8.           Failed:
    9.           Not Ready:
    10.           Ready:

    Viewing the status of logging subsystem components

    You can view the status for a number of logging subsystem components.

    • The Red Hat OpenShift Logging and Elasticsearch Operators must be installed.

    Procedure

    1. Change to the openshift-logging project.

      1. $ oc project openshift-logging

    2. View the status of the logging subsystem for Red Hat OpenShift environment:

      1. $ oc describe deployment cluster-logging-operator

      Example output

    3. View the status of the logging subsystem replica set:

      1. Get the name of a replica set:

        Example output

        1. $ oc get replicaset

        Example output

        1. NAME                                      DESIRED   CURRENT   READY   AGE
        2. cluster-logging-operator-574b8987df       1         1         1       159m
        3. elasticsearch-cdm-uhr537yu-1-6869694fb    1         1         1       157m
        4. elasticsearch-cdm-uhr537yu-2-857b6d676f   1         1         1       156m
        5. elasticsearch-cdm-uhr537yu-3-5b6fdd8cfd   1         1         1       155m
        6. kibana-5bd5544f87                         1         1         1       157m

      2. Get the status of the replica set:

        1. $ oc describe replicaset cluster-logging-operator-574b8987df
        1. Name:           cluster-logging-operator-574b8987df
        3. ....
        5. Replicas:       1 current / 1 desired
        6. Pods Status:    1 Running / 0 Waiting / 0 Succeeded / 0 Failed
        8. ....
        10. Events:
        11.   Type    Reason            Age   From                   Message