Observability — Logging FAQ

• How to change the log store to the external Elasticsearch and shut down the internal Elasticsearch
• How to modify the log data retention period
• The log search page in Toolbox gets stuck when loading
• I see Internal Server Error when viewing logs in Toolbox

If you are using the KubeSphere internal Elasticsearch and want to change it to your external alternate, follow the steps below. If you haven’t enabled the logging system, refer to KubeSphere Logging System to setup your external Elasticsearch directly.

1. First, you need to update the KubeKey configuration. Execute the following command:

2. Comment out , es.elasticsearchMasterXXX and status.logging, and set es.externalElasticsearchUrl to the address of your Elasticsearch and es.externalElasticsearchPort to its port number. Below is an example for your reference.

   apiVersion: installer.kubesphere.io/v1alpha1
   kind: ClusterConfiguration
     name: ks-installer
     namespace: kubesphere-system
     ...
   spec:
     ...
     common:
       es:
         # elasticsearchDataReplicas: 1
         # elasticsearchMasterReplicas: 1
         # elasticsearchMasterVolumeSize: 4Gi
         elkPrefix: logstash
         logMaxAge: 7
         externalElasticsearchUrl: <192.168.0.2>
         externalElasticsearchPort: <9200>
     ...
   status:
     ...
     # logging:
     #  enabledTime: 2020-08-10T02:05:13UTC
     ...

3. Rerun ks-installer.

   kubectl rollout restart deploy -n kubesphere-system ks-installer

4. Remove the internal Elasticsearch by running the following command. Please make sure you have backed up data in the internal Elasticsearch.

5. ...
    options:
         es:
           index-prefix: logstash
           server-urls: http://elasticsearch-logging-data.kubesphere-logging-system.svc:9200  # Change it to the external address.

How to change the log store to Elasticsearch with X-Pack Security enabled

Currently, KubeSphere doesn’t support the integration of Elasticsearch with X-Pack Security enabled. This feature is coming soon.

You need to update the KubeKey configuration and rerun ks-installer.

1. Execute the following command:

   kubectl edit cc -n kubesphere-system ks-installer

2. Comment out status.logging and set a desired retention period as the value of es.logMaxAge (7 by default).

3. Rerun ks-installer.

   kubectl rollout restart deploy -n kubesphere-system ks-installer

I cannot find logs from workloads on some nodes using Toolbox

To find out the docker root directory path on nodes, you can run the following command. Make sure the same value applies to all nodes.

docker info -f '{{.DockerRootDir}}'

If the log search page is stuck when loading, check the storage system you are using. For example, a misconfigured NFS storage system may cause this issue.

Toolbox shows no log record today

Check if your log volume exceeds the storage limit of Elasticsearch. If so, you need to increase the Elasticsearch disk volume.

There can be several reasons for this issue:

• Network partition
• Invalid Elasticsearch host and port
• The Elasticsearch health status is red

How to make KubeSphere only collect logs from specified workloads

The KubeSphere logging agent is powered by Fluent Bit. You need to update the Fluent Bit configuration to exclude certain workload logs. To modify the Fluent Bit input configuration, run the following command:

For more information, see .