Configuring your logging deployment

    Red Hat Openshift Logging Operator:

    • (CL) - Deploys the collector and forwarder which currently are both implemented by a daemonset running on each node.

    • ClusterLogForwarder (CLF) - Generates collector configuration to forward logs per user configuration.

    Loki Operator:

    • LokiStack - Controls the Loki cluster as log store and the web proxy with OpenShift Container Platform authentication integration to enforce multi-tenancy.

    OpenShift Elasticsearch Operator:

    • ElasticSearch - Configure and deploy an Elasticsearch instance as the default log store.

    • Kibana - Configure and deploy Kibana instance to search, query and view logs.

    The supported way of configuring the logging subsystem for Red Hat OpenShift is by configuring it using the options described in this documentation. Do not use other configurations, as they are unsupported. Configuration paradigms might change across OpenShift Container Platform releases, and such cases can only be handled gracefully if all configuration possibilities are controlled. If you use configurations other than those described in this documentation, your changes will disappear because the Operators reconcile any differences. The Operators reverse everything to the defined state by default and by design.

    If you must perform configurations not described in the OpenShift Container Platform documentation, you must set your Red Hat OpenShift Logging Operator to Unmanaged. An unmanaged OpenShift Logging environment is not supported and does not receive updates until you return OpenShift Logging to Managed.

    With Logging version 5.6 and higher, you can configure retention policies based on log streams. Rules for these may be set globally, per tenant, or both. If you configure both, tenant rules apply before global rules.

    1. To enable stream-based retention, create or edit the LokiStack custom resource (CR):
    1. You can refer to the examples below to configure your LokiStack CR.

    Example global stream-based retention

    1. apiVersion: loki.grafana.com/v1
    2. kind: LokiStack
    3. metadata:
    4.   name: logging-loki
    5.   namespace: openshift-logging
    6. spec:
    7.   limits:
    8.     global: (1)
    9.       retention: (2)
    10.         days: 20
    11.         - days: 4
    12.           priority: 1
    13.           selector: '{kubernetes_namespace_name=~"test.+"}' (3)
    15.           priority: 1
    16.           selector: '{log_type="infrastructure"}'
    17.   managementState: Managed
    18.   replicationFactor: 1
    19.   size: 1x.small
    20.   storage:
    21.     schemas:
    22.     - effectiveDate: "2020-10-11"
    23.       version: v11
    24.     secret:
    25.       name: logging-loki-s3
    26.       type: aws
    27.   storageClassName: standard
    28.   tenants:
    29.     mode: openshift-logging

    Example per-tenant stream-based retention

    1Sets retention policy by tenant. Valid tenant types are application, audit, and infrastructure.
    2Contains the used to define the log stream.

    1. Then apply your configuration:

    1. oc apply -f <file-name>.yaml

    Enabling multi-line exception detection

    Enables multi-line error detection of container logs.

    Enabling this feature could have performance implications and may require additional computing resources or alternate logging solutions.

    Log parsers often incorrectly identify separate lines of the same exception as separate exceptions. This leads to extra log entries and an incomplete or inaccurate view of the traced information.

    Example java exception

    Example ClusterLogForwarder CR

    2. kind: ClusterLogForwarder
    3. metadata:
    4.   name: instance
    5.   namespace: openshift-logging
    6. spec:
    7.   pipelines:
    8.     - name: my-app-logs
    9.       inputRefs:
    10.         - application
    11.       outputRefs:
    12.         - default
    13.       detectMultilineErrors: true

    When log messages appear as a consecutive sequence forming an exception stack trace, they are combined into a single, unified log record. The first log message’s content is replaced with the concatenated content of all the message fields in the sequence.

    Troubleshooting

    When enabled, the collector configuration will include a new section with type: detect_exceptions

    Example vector configuration section

    Example fluentd config section

    1. <label @MULTILINE_APP_LOGS>
    2.   <match kubernetes.**>
    3.     @type detect_exceptions
    4.     remove_tag_prefix 'kubernetes'
    5.     message message
    6.     force_line_breaks true
    7.     multiline_flush_interval .2
    8.   </match>