请注意,在我更新证书之前,整个集群都运行良好。我在使用 Elasticsearch 7.5。
我使用 certificate-util 更新了我的证书 - 创建了一个包含我所有服务器和 Kibana 节点的 server.yml,使用 -pem、-multiple 和 --keep-ca-key 运行该实用程序,传递了 ca.crt以及发送给节点的各种节点证书,确保 elasticsearch.yml 和 kibana.yml 文件适当地指向它们,并进行滚动重启。然后一切都崩溃了。
主要问题——我目前至少遇到的问题——是我在主题中遇到了错误。完整的,来自与之通信的 Elastic 节点的日志:
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:473) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:281) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:600) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:554) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) [netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.43.Final.jar:4.1.43.Final]
at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:324) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:267) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:262) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkClientCerts(CertificateMessage.java:1275) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1178) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1159) ~[?:?]
at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1247) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:691) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1192) ~[?:?]
at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1502) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1516) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1400) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:503) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]
... 16 more
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:349) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:258) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:138) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkClientCerts(CertificateMessage.java:1253) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1178) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1159) ~[?:?]
at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1247) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:691) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1192) ~[?:?]
at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1502) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1516) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1400) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:503) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]
... 16 more
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:158) ~[?:?]
at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:84) ~[?:?]
at java.security.cert.CertPathValidator.validate(CertPathValidator.java:309) ~[?:?]
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:344) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:258) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:138) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkClientCerts(CertificateMessage.java:1253) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1178) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1159) ~[?:?]
at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1247) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:691) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1192) ~[?:?]
at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1502) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1516) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1400) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:503) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]
... 16 more
我所有的节点都给出了类似的错误。
从表面上看,它看起来相当简单 - 你会认为 ca.crt 在 Kibana 节点上是错误的(至少,这是我从搜索中发现的。)如果它是错误的,我不知道为什么 - 它实际上是与我在所有其他节点上获得的 ca.crt 相同。
据我所知,我没有明确配置密钥库,elasticsearch-keystore 的输出如下所示:
[root@Kibana Node kibana]# /usr/share/elasticsearch/bin/elasticsearch-keystore list
bootstrap.password
keystore.seed
xpack.security.authc.realms.active_directory.platform_ad.secure_bind_password
如果 Java 有一个密钥库,我也找不到它,至少找不到名称“keystore”——如果有人知道可以在哪里配置它,下一步可能会很好。
我不确定我还能做些什么来解决问题 - 我的 Google-fu 让我失望了(我发现的一切都指向 CA 确实是问题,以最简单的方式 - 它们不匹配。如果在这种情况下它们不匹配,这是证书实用程序的错误,但将它们与 openssl 进行比较并没有发现任何差异。)
快速总结:用于为集群生成所有证书的同一个 ca.crt 在 Kibana 上似乎不受信任,但似乎在其他节点上工作。
因此,事实证明这是一个偷偷摸摸的双重问题。
第一个问题是我在我的配置中的一个ca.crt 条目上输入了我使用的路径 - 想想“cert”与“certs”这使得它找不到证书(并不是它实际上告诉我在记录...)这很可能掩盖了第二个问题。
第二个问题似乎是 Elastic 不喜欢我更改使用的默认路径 $ES_HOME$/certs/.crt 和 $ES_HOME$/certs/ca/ca.crt。与我一起进行故障排除的人注意到证书设置文档中的一个小玩笑,基本上说“如果您更改它指向的路径,Elasticsearch 有时会在查找证书时遇到问题,无论您是否正确设置了权限。” 再一次,不像它在日志中的任何地方告诉我的那样......
这是非常愚蠢的,但是将路径更改回默认路径并将证书移回原始文件夹使其按预期获取证书并在第一次拼写错误修复无效后进行通信。