Performance analyzer

    The performance analyzer plugin is installed by default in OpenSearch version 2.0 and higher.

    The following sections provide the steps for installing and configuring the performance analyzer plugin.

    The performance analyzer plugin is included in the installation for Docker and . If you need to install the performance analyzer plugin manually, download the plugin from Maven and install the plugin using the standard process. Performance analyzer will run on each node in a cluster.

    To start the performance analyzer root cause analysis (RCA) agent on a tarball installation, run the following command:

    The following command enables the performance analyzer plugin and performance analyzer RCA agent:

    To shut down the performance analyzer RCA agent, run the following command:

    1. kill $(ps aux | grep -i 'PerformanceAnalyzerApp' | grep -v grep | awk '{print $2}')

    To disable the performance analyzer plugin, run the following command:

    1. curl -XPOST localhost:9200/_plugins/_performanceanalyzer/cluster/config -H 'Content-Type: application/json' -d '{"enabled": false}'

    To uninstall the performance analyzer plugin, run the following command:

    1. bin/opensearch-plugin remove opensearch-performance-analyzer

    Configure performance analyzer

    1. # ======================== OpenSearch performance analyzer plugin config =========================
    3. # NOTE: this is an example for Linux. Please modify the config accordingly if you are using it under other OS.
    5. # WebService bind host; default to all interfaces
    6. webservice-bind-host = 0.0.0.0
    8. # Metrics data location
    9. metrics-location = /dev/shm/performanceanalyzer/
    11. # Metrics deletion interval (minutes) for metrics data.
    12. # Interval should be between 1 to 60.
    13. metrics-deletion-interval = 1
    15. # If set to true, the system cleans up the files behind it. So at any point, we should expect only 2
    16. # metrics-db-file-prefix-path files. If set to false, no files are cleaned up. This can be useful, if you are archiving
    17. # the files and wouldn't like for them to be cleaned up.
    18. cleanup-metrics-db-files = true
    20. # WebService exposed by App's port
    21. webservice-listener-port = 9600
    23. # Metric DB File Prefix Path location
    24. metrics-db-file-prefix-path = /tmp/metricsdb_
    26. https-enabled = false
    27. #Setup the correct path for certificates
    28. #certificate-file-path = specify_path
    30. #private-key-file-path = specify_path
    32. # Plugin Stats Metadata file name, expected to be in the same location
    33. plugin-stats-metadata = plugin-stats-metadata
    35. # Agent Stats Metadata file name, expected to be in the same location
    36. agent-stats-metadata = agent-stats-metadata

    To start the performance analyzer RCA agent, run the following command.

    Performance analyzer uses /dev/shm for temporary storage. During heavy workloads on a cluster, performance analyzer can use up to 1 GB of space.

    Docker, however, has a default /dev/shm size of 64 MB. To change this value, you can use the docker run --shm-size 1gb flag or a similar setting in Docker Compose.

    If you’re not using Docker, check the size of /dev/shm using df -h. The default value is probably plenty, but if you need to change its size, add the following line to /etc/fstab:

    1. tmpfs /dev/shm tmpfs defaults,noexec,nosuid,size=1G 0 0

    Then remount the file system:

    1. mount -o remount /dev/shm

    Security

    Performance analyzer supports encryption in transit for requests. It currently does not support client or server authentication for requests. To enable encryption in transit, edit performance-analyzer.properties in your $OPENSEARCH_HOME directory.

    1. vi $OPENSEARCH_HOME/config/opensearch-performance-analyzer/performance-analyzer.properties

    Change the following lines to configure encryption in transit. Note that certificate-file-path must be a certificate for the server, not a root certificate authority (CA).

    1. https-enabled = true
    3. #Setup the correct path for certificates
    4. certificate-file-path = specify_path
    6. private-key-file-path = specify_path

    If you installed OpenSearch from an RPM distribution, you can start and stop performance analyzer with systemctl.

    1. # Start OpenSearch Performance Analyzer
    2. sudo systemctl start opensearch-performance-analyzer.service
    3. # Stop OpenSearch Performance Analyzer
    4. sudo systemctl stop opensearch-performance-analyzer.service

    The following is an example response:

    1. {"Disk_Utilization":"%","Cache_Request_Hit":"count", 
    2. "Refresh_Time":"ms","ThreadPool_QueueLatency":"count",
    3. "Merge_Time":"ms","ClusterApplierService_Latency":"ms",
    4. "PublishClusterState_Latency":"ms",
    5. "Cache_Request_Size":"B","LeaderCheck_Failure":"count",
    6. "ThreadPool_QueueSize":"count","Sched_Runtime":"s/ctxswitch","Disk_ServiceRate":"MB/s","Heap_AllocRate":"B/s","Indexing_Pressure_Current_Limits":"B",
    7. "Sched_Waittime":"s/ctxswitch","ShardBulkDocs":"count",
    8. "Thread_Blocked_Time":"s/event","VersionMap_Memory":"B",
    9. "Master_Task_Queue_Time":"ms","IO_TotThroughput":"B/s",
    10. "Indexing_Pressure_Current_Bytes":"B",
    11. "Indexing_Pressure_Last_Successful_Timestamp":"ms",
    12. "Net_PacketRate6":"packets/s","Cache_Query_Hit":"count",
    13. "IO_ReadSyscallRate":"count/s","Net_PacketRate4":"packets/s","Cache_Request_Miss":"count",
    14. "ThreadPool_RejectedReqs":"count","Net_TCP_TxQ":"segments/flow","Master_Task_Run_Time":"ms",
    15. "IO_WriteSyscallRate":"count/s","IO_WriteThroughput":"B/s",
    16. "Refresh_Event":"count","Flush_Time":"ms","Heap_Init":"B",
    18. "CPU_Utilization":"cores","Cache_Query_Size":"B",
    19. "Merge_Event":"count","Cache_FieldData_Eviction":"count",
    20. "IO_TotalSyscallRate":"count/s","Net_Throughput":"B/s",
    21. "AdmissionControl_ThresholdValue":"count",
    22. "Indexing_Pressure_Average_Window_Throughput":"count/s",
    23. "Cache_MaxSize":"B","IndexWriter_Memory":"B",
    24. "Net_TCP_SSThresh":"B/flow","IO_ReadThroughput":"B/s",
    25. "LeaderCheck_Latency":"ms","FollowerCheck_Failure":"count",
    26. "HTTP_RequestDocs":"count","Net_TCP_Lost":"segments/flow",
    27. "GC_Collection_Event":"count","Sched_CtxRate":"count/s",
    28. "AdmissionControl_RejectionCount":"count","Heap_Max":"B",
    29. "ClusterApplierService_Failure":"count",
    30. "PublishClusterState_Failure":"count",
    31. "Merge_CurrentEvent":"count","Indexing_Buffer":"B",
    32. "Bitset_Memory":"B","Net_PacketDropRate4":"packets/s",
    33. "Heap_Committed":"B","Net_PacketDropRate6":"packets/s",
    34. "Thread_Blocked_Event":"count","GC_Collection_Time":"ms",
    35. "Cache_Query_Miss":"count","Latency":"ms",
    36. "Shard_State":"count","Thread_Waited_Event":"count",
    37. "CB_ConfiguredSize":"B","ThreadPool_QueueCapacity":"count",
    38. "CB_TrippedEvents":"count","Disk_WaitTime":"ms",
    39. "Data_RetryingPendingTasksCount":"count",
    40. "AdmissionControl_CurrentValue":"count",
    41. "Flush_Event":"count","Net_TCP_RxQ":"segments/flow",
    42. "Shard_Size_In_Bytes":"B","Thread_Waited_Time":"s/event",
    43. "HTTP_TotalRequests":"count",
    44. "ThreadPool_ActiveThreads":"count",
    45. "Paging_MinfltRate":"count/s","Net_TCP_SendCWND":"B/flow",
    46. "Cache_Request_Eviction":"count","Segments_Total":"count",
    47. "FollowerCheck_Latency":"ms","Heap_Used":"B",
    48. "Master_ThrottledPendingTasksCount":"count",
    49. "CB_EstimatedSize":"B","Indexing_ThrottleTime":"ms",
    50. "Master_PendingQueueSize":"count",
    51. "Cache_FieldData_Size":"B","Paging_MajfltRate":"count/s",
    52. "ThreadPool_TotalThreads":"count","ShardEvents":"count",
    53. "Net_TCP_NumFlows":"count","Election_Term":"count"}

    The (RCA) framework uses the information from performance analyzer to inform administrators of the root cause of performance and availability issues that their clusters might be experiencing.

    Enable the RCA framework

    To enable the RCA framework, run the following command:

    1. curl -XPOST http://localhost:9200/_plugins/_performanceanalyzer/rca/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}'

    If you encounter the curl: (52) Empty reply from server response, run the following command to enable RCA:

    1. curl -XPOST https://localhost:9200/_plugins/_performanceanalyzer/rca/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}' -u 'admin:admin' -k

    To request all available RCAs, run the following command:

    1. GET localhost:9600/_plugins/_performanceanalyzer/rca

    To request a specific RCA, run the following command:

    The following is an example response:

    Further documentation on the use of performance analyzer and RCA can be found at the following links: