Understanding logging subsystem alerts

    Alerts are shown in the OKD web console, on the Alerts tab of the Alerting UI. Alerts are in one of the following states:

    • Firing. The alert condition is true for the duration of the timeout. Click the Options menu at the end of the firing alert to view more information or silence the alert.

    • Pending The alert condition is currently true, but the timeout has not been reached.

    Procedure

    To view the logging subsystem and other OKD alerts:

    1. In the OKD console, click Observe → Alerting.

    2. Click the Alerts tab. The alerts are listed, based on the filters selected.

    Additional resources

    The following alerts are generated by the logging collector. You can view these alerts in the OKD web console on the Alerts page of the Alerting UI.

    You can view these alerting rules in Prometheus.

    Table 2. Alerting rules
    AlertDescriptionSeverity

    ElasticsearchClusterNotHealthy

    The cluster health status has been RED for at least 2 minutes. The cluster does not accept writes, shards may be missing, or the master node hasn’t been elected yet.

    Critical

    ElasticsearchClusterNotHealthy

    The cluster health status has been YELLOW for at least 20 minutes. Some shard replicas are not allocated.

    Warning

    ElasticsearchDiskSpaceRunningLow

    The cluster is expected to be out of disk space within the next 6 hours.

    Critical

    ElasticsearchHighFileDescriptorUsage

    The cluster is predicted to be out of file descriptors within the next hour.

    Warning

    The JVM Heap usage on the specified node is high.

    ElasticsearchNodeDiskWatermarkReached

    The specified node has hit the low watermark due to low free disk space. Shards can not be allocated to this node anymore. You should consider adding more disk space to the node.

    Info

    ElasticsearchNodeDiskWatermarkReached

    The specified node has hit the high watermark due to low free disk space. Some shards will be re-allocated to different nodes if possible. Make sure more disk space is added to the node or drop old indices allocated to this node.

    Warning

    ElasticsearchNodeDiskWatermarkReached

    The specified node has hit the flood watermark due to low free disk space. Every index that has a shard allocated on this node is enforced a read-only block. The index block must be manually released when the disk use falls below the high watermark.

    Critical

    ElasticsearchJVMHeapUseHigh

    The JVM Heap usage on the specified node is too high.

    Alert

    ElasticsearchWriteRequestsRejectionJumps

    Elasticsearch is experiencing an increase in write rejections on the specified node. This node might not be keeping up with the indexing speed.

    Warning

    The CPU used by the system on the specified node is too high.

    Alert

    ElasticsearchProcessCPUHigh

    The CPU used by Elasticsearch on the specified node is too high.