Enabling JSON logging

Logs including JSON logs are usually represented as a string inside the field. That makes it hard for users to query specific fields inside a JSON document. OpenShift Logging’s Log Forwarding API enables you to parse JSON logs into a structured object and forward them to either OpenShift Logging-managed Elasticsearch or any other third-party system supported by the Log Forwarding API.

To illustrate how this works, suppose that you have the following structured JSON log entry.

Example structured JSON log entry

Normally, the ClusterLogForwarder custom resource (CR) forwards that log entry in the message field. The message field contains the JSON-quoted string equivalent of the JSON log entry, as shown in the following example.

Example message field

{"message":"{\"level\":\"info\",\"name\":\"fred\",\"home\":\"bedrock\"",
 "more fields..."}

To enable parsing JSON log, you add parse: json to a pipeline in the ClusterLogForwarder CR, as shown in the following example.

Example snippet showing parse: json

pipelines:
- inputRefs: [ application ]
  outputRefs: myFluentd
  parse: json

When you enable parsing JSON logs by using parse: json, the CR copies the JSON-structured log entry in a structured field, as shown in the following example. This does not modify the original message field.

Example structured output containing the structured JSON log entry

{"structured": { "level": "info", "name": "fred", "home": "bedrock" },
 "more fields..."}

To enable parsing JSON logs for specific logging platforms, see .

If your JSON logs follow more than one schema, storing them in a single index might cause type conflicts and cardinality problems. To avoid that, you must configure the ClusterLogForwarder custom resource (CR) to group each schema into a single output definition. This way, each schema is forwarded to a separate index.

Structure types

You can use the following structure types in the ClusterLogForwarder CR to construct index names for the Elasticsearch log store:

• • kubernetes.labels.<key> is the Kubernetes pod label whose value is used to construct the index name.

  • openshift.labels.<key> is the pipeline.label.<key> element in the ClusterLogForwarder CR whose value is used to construct the index name.

  • kubernetes.container_name uses the container name to construct the index name.

• structuredTypeName: (string, optional) If structuredTypeKey is not set or its key is not present, OpenShift Logging uses the value of structuredTypeName as the structured type. When you use both and structuredTypeName together, structuredTypeName provides a fallback index name if the key in structuredTypeKey is missing from the JSON log data.

A structuredTypeKey: kubernetes.labels.<key> example

Suppose the following:

• Your cluster is running application pods that produce JSON logs in two different formats, “apache” and “google”.

• The user labels these application pods with logFormat=apache and logFormat=google.

• You use the following snippet in your ClusterLogForwarder CR YAML file.

In that case, the following structured log record goes to the app-apache-write index:

{
  "kubernetes":{"labels":{"logFormat": "apache", ...}}
}

And the following structured log record goes to the app-google-write index:

{
  "structured":{"name":"wilma","home":"bedrock"},
  "kubernetes":{"labels":{"logFormat": "google", ...}}
}

A structuredTypeKey: openshift.labels.<key> example

Suppose that you use the following snippet in your ClusterLogForwarder CR YAML file.

outputDefaults:
 elasticsearch:
    structuredTypeKey: openshift.labels.myLabel (1)
    structuredTypeName: nologformat
pipelines:
 - name: application-logs
   inputRefs:
   - application
   - audit
   outputRefs:
   - elasticsearch-secure
   - default
   labels:

In that case, the following structured log record goes to the app-myValue-write index:

• The Elasticsearch index for structured records is formed by prepending “app-“ to the structured type and appending “-write”.

• Unstructured records are not sent to the structured index. They are indexed as usual in the application, infrastructure, or audit indices.

• If there is no non-empty structured type, forward an unstructured record with no structured field.

It is important not to overload Elasticsearch with too many indices. Only use distinct structured types for distinct log formats, not for each application or namespace. For example, most Apache applications use the same JSON log format and structured type, such as LogApache.

For an Elasticsearch log store, if your JSON log entries follow different schemas, configure the ClusterLogForwarder custom resource (CR) to group each JSON schema into a single output definition. This way, Elasticsearch uses a separate index for each schema.

Procedure

1. Add the following snippet to your ClusterLogForwarder CR YAML file.

   outputDefaults:
    elasticsearch:
       structuredTypeKey: <log record field>
       structuredTypeName: <name>
   pipelines:
   - inputRefs:
     - application
     outputRefs: default
     parse: json

2. Optional: Use structuredTypeKey to specify one of the log record fields, as described in the preceding topic, Configuring JSON log data for Elasticsearch. Otherwise, remove this line.

3. Optional: Use structuredTypeName to specify a <name>, as described in the preceding topic, . Otherwise, remove this line.

4. For inputRefs, specify which log types to forward by using that pipeline, such as application, infrastructure, or audit.

5. Add the parse: json element to pipelines.

6. Create the CR object:

   $ oc delete pod --selector logging-infra=collector

Additional resources