TLS troubleshooting

    and the files in config/opensearch-security/ are in the YAML format. A linter like can help verify that you don’t have any formatting errors.

    View contents of PEM certificates

    You can use OpenSSL to display the content of each PEM certificate:

    Then ensure that the value matches the one in opensearch.yml.

    For more complete information on a certificate:

    1. openssl x509 -in node1.pem -text -noout

    The Security plugin uses the string representation of Distinguished Names (RFC1779) when validating node certificates.

    If parts of your DN contain special characters (e.g. a comma), make sure you escape it in your configuration:

    1.   - 'CN=node-0.example.com,OU=SSL,O=My\, Test,L=Test,C=DE'

    You can have whitespace within a field, but not between fields.

    Bad configuration

    1. plugins.security.nodes_dn:
    2.   - 'CN=node-0.example.com, OU=SSL,O=My\, Test, L=Test, C=DE'

    Good configuration

    1. plugins.security.nodes_dn:
    2.   - 'CN=node-0.example.com,OU=SSL,O=My\, Test,L=Test,C=DE'

    Sometimes the IP address in your certificate is not the one communicating with the cluster. This problem can occur if your node has multiple interfaces or is running on a dual stack network (IPv6 and IPv4).

    1. SSL Problem Received fatal alert: certificate_unknown javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

    You might also see the following message in your cluster’s master log when the new node tries to join the cluster:

    Check the IP address in the certificate:

    1. IPAddress: 2001:db8:0:1:1.2.3.4

    In this example, the node tries to join the cluster with the IPv4 address of 10.0.0.42, but the certificate contains the IPv6 address of 2001:db8:0:1:1.2.3.4.

    TLS certificates are organized in a certificate chain. You can check with keytool that the certificate chain is correct by inspecting the owner and the issuer of each certificate. If you used the demo installation script that ships with the Security plugin, the chain looks like:

    Node certificate

    1. Owner: CN=node-0.example.com, OU=SSL, O=Test, L=Test, C=DE
    2. Issuer: CN=Example Com Inc. Signing CA, OU=Example Com Inc. Signing CA, O=Example Com Inc., DC=example, DC=com

    Signing certificate

    1. Owner: CN=Example Com Inc. Signing CA, OU=Example Com Inc. Signing CA, O=Example Com Inc., DC=example, DC=com
    2. Issuer: CN=Example Com Inc. Root CA, OU=Example Com Inc. Root CA, O=Example Com Inc., DC=example, DC=com

    Root certificate

    2. Issuer: CN=Example Com Inc. Root CA, OU=Example Com Inc. Root CA, O=Example Com Inc., DC=example, DC=com

    From the entries, you can see that the root certificate signed the intermediate certificate, which signed the node certificate. The root certificate signed itself, hence the name “self-signed certificate.” If you’re using separate keystore and truststore files, your root CA can most likely in the truststore.

    Generally, the keystore contains client or node certificate and all intermediate certificates, and the truststore contains the root certificate.

    If you have multiple entries in the keystore and you are using aliases to refer to them, make sure that the configured alias in opensearch.yml matches the one in the keystore. If there is only one entry in the keystore, you do not need to configure an alias.

    In order to view information about the certificates stored in your keystore or truststore, use the keytool command like:

    1. keytool -list -v -keystore keystore.jks

    Check SAN hostnames and IP addresses

    The valid hostnames and IP addresses of a TLS certificates are stored as SAN entries. Check that the hostname and IP entries in the SAN section are correct, especially when you use hostname verification:

    If you are using OIDs to denote valid node certificates, check that the SAN extension for your node certificate contains the correct OIDName:

    1. Certificate[1]:
    2. Owner: CN=node-0.example.com, OU=SSL, O=Test, L=Test, C=DE
    3. ...
    4. Extensions:
    5. #5: ObjectId: 2.5.29.17 Criticality=false
    6. SubjectAlternativeName [
    7.   ...
    8.   OIDName: 1.2.3.4.5.5

    Check EKU field for node certificates

    Node certificates need to have both serverAuth and clientAuth set in the extended key usage field:

    1. #3: ObjectId: 2.5.29.37 Criticality=false
    2. ExtendedKeyUsages [
    3.   serverAuth
    4.   clientAuth
    5. ]

    The Security plugin disables TLS version 1.0 by default; it is outdated, insecure, and vulnerable. If you need to use TLSv1 and accept the risks, you can enable it in opensearch.yml:

    1. plugins.security.ssl.http.enabled_protocols:
    2.   - "TLSv1"
    3.   - "TLSv1.1"
    4.   - "TLSv1.2"

    Supported ciphers

    TLS relies on the server and client negotiating a common cipher suite. Depending on your system, the available ciphers will vary. They depend on the JDK or OpenSSL version you’re using, and whether or not the JCE Unlimited Strength Jurisdiction Policy Files are installed.

    For legal reasons, the JDK does not include strong ciphers like AES256. In order to use strong ciphers you need to download and install the . If you don’t have them installed, you might see an error message on startup:

    1. [INFO ] AES-256 not supported, max key length for AES is 128 bit.
    2. That is not an issue, it just limits possible encryption strength.
    3. To enable AES 256 install 'Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files'

    The Security plugin still works and falls back to weaker cipher suites. The plugin also prints out all available cipher suites during startup:

    1. [INFO ] sslTransportClientProvider:
    2. TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, ...]