Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Several test failures in :x-pack:plugin:security:test in 6.3 #30869

Closed
droberts195 opened this issue May 25, 2018 · 10 comments
Closed

Several test failures in :x-pack:plugin:security:test in 6.3 #30869

droberts195 opened this issue May 25, 2018 · 10 comments
Assignees
@jkakavas
Labels
:Security/Security Security issues without another label >test-failure Triaged test failures from CI

Comments

@droberts195
Copy link
Contributor

droberts195 commented May 25, 2018 •
edited
Loading

In https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.3+multijob-unix-compatibility/os=sles/86/console the following tests failed:

org.elasticsearch.xpack.security.authc.esnative.NativeRealmIntegTests testAddUserAndRoleThenAuth
org.elasticsearch.xpack.security.authz.IndexAliasesTests testGetAliasesCreateOnlyPermissionStrict
org.elasticsearch.xpack.security.authz.IndexAliasesTests testGetAliasesCreateOnlyPermissionStrict
org.elasticsearch.xpack.security.audit.logfile.AuditTrailSettingsUpdateTests testInvalidFilterSettings
org.elasticsearch.xpack.security.transport.ssl.SslMultiPortTests testThatTransportClientWithOnlyTruststoreCanConnectToNoClientAuthProfile
org.elasticsearch.xpack.security.transport.ssl.SslMultiPortTests testThatTransportClientWithOnlyTruststoreCannotConnectToDefaultProfile
org.elasticsearch.xpack.security.transport.ssl.SslMultiPortTests testThatTransportClientWithOnlyTruststoreCannotConnectToDefaultProfile

None of these reproduce individually. These are the REPRO commands:

./gradlew :x-pack:plugin:security:test -Dtests.seed=22E2031847F1522 -Dtests.class=org.elasticsearch.xpack.security.audit.logfile.AuditTrailSettingsUpdateTests -Dtests.method="testInvalidFilterSettings" -Dtests.security.manager=true -Dtests.locale=sr-Latn-RS -Dtests.timezone=America/Argentina/Rio_Gallegos
./gradlew :x-pack:plugin:security:test -Dtests.seed=22E2031847F1522 -Dtests.class=org.elasticsearch.xpack.security.transport.ssl.SslMultiPortTests -Dtests.method="testThatTransportClientWithOnlyTruststoreCanConnectToNoClientAuthProfile" -Dtests.security.manager=true -Dtests.locale=sr-Latn-BA -Dtests.timezone=Chile/Continental
./gradlew :x-pack:plugin:security:test -Dtests.seed=22E2031847F1522 -Dtests.class=org.elasticsearch.xpack.security.transport.ssl.SslMultiPortTests -Dtests.method="testThatTransportClientWithOnlyTruststoreCannotConnectToDefaultProfile" -Dtests.security.manager=true -Dtests.locale=sr-Latn-BA -Dtests.timezone=Chile/Continental
./gradlew :x-pack:plugin:security:test -Dtests.seed=22E2031847F1522 -Dtests.class=org.elasticsearch.xpack.security.transport.ssl.SslMultiPortTests -Dtests.method="testThatTransportClientWithOnlyTruststoreCannotConnectToDefaultProfile" -Dtests.security.manager=true -Dtests.locale=sr-Latn-BA -Dtests.timezone=Chile/Continental
./gradlew :x-pack:plugin:security:test -Dtests.seed=22E2031847F1522 -Dtests.class=org.elasticsearch.xpack.security.authc.esnative.NativeRealmIntegTests -Dtests.method="testAddUserAndRoleThenAuth" -Dtests.security.manager=true -Dtests.locale=el-CY -Dtests.timezone=Africa/Algiers
./gradlew :x-pack:plugin:security:test -Dtests.seed=22E2031847F1522 -Dtests.class=org.elasticsearch.xpack.security.authz.IndexAliasesTests -Dtests.method="testGetAliasesCreateOnlyPermissionStrict" -Dtests.security.manager=true -Dtests.locale=ru-RU -Dtests.timezone=Africa/Malabo
./gradlew :x-pack:plugin:security:test -Dtests.seed=22E2031847F1522 -Dtests.class=org.elasticsearch.xpack.security.authz.IndexAliasesTests -Dtests.method="testGetAliasesCreateOnlyPermissionStrict" -Dtests.security.manager=true -Dtests.locale=ru-RU -Dtests.timezone=Africa/Malabo

There are some exceptions in the log file indicating SSL setup problems, for example:

  1> [2018-05-25T08:17:49,624][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [node_sc2] exception caught on transport layer [NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:56147, remoteAddress=/127.0.0.1:51564}], closing connection
  1> io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 455300000064000000000000000108004d3603010d417574686f72697a6174696f6e2e4261736963206447567a64463931633256794f6e67746347466a617931305a584e304c58426863334e3362334a6b0016696e7465726e616c3a7463702f68616e647368616b6500
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
  1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
  1> Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 455300000064000000000000000108004d3603010d417574686f72697a6174696f6e2e4261736963206447567a64463931633256794f6e67746347466a617931305a584e304c58426863334e3362334a6b0016696e7465726e616c3a7463702f68616e647368616b6500
  1> 	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1106) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	... 15 more

Also:

  1> io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
  1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
  1> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
  1> 	at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529) ~[?:?]
  1> 	at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[?:?]
  1> 	at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) ~[?:?]
  1> 	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]
  1> 	at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_172]
  1> 	at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:281) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	... 15 more
  1> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
  1> 	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
  1> 	at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) ~[?:?]
  1> 	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) ~[?:?]
  1> 	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) ~[?:?]
  1> 	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1614) ~[?:?]
  1> 	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[?:?]
  1> 	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
  1> 	at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
  1> 	at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
  1> 	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_172]
  1> 	at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
  1> 	at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	... 15 more
  1> Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
  1> 	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) ~[?:?]
  1> 	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) ~[?:?]
  1> 	at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]
  1> 	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
  1> 	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) ~[?:?]
  1> 	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:?]
  1> 	at org.elasticsearch.xpack.core.ssl.SSLService$ReloadableTrustManager.checkServerTrusted(SSLService.java:594) ~[x-pack-core-6.3.0-SNAPSHOT.jar:6.3.0-SNAPSHOT]
  1> 	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1601) ~[?:?]
  1> 	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[?:?]
  1> 	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
  1> 	at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
  1> 	at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
  1> 	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_172]
  1> 	at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
  1> 	at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	... 15 more
  1> Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
  1> 	at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[?:?]
  1> 	at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[?:?]
  1> 	at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) ~[?:1.8.0_172]
  1> 	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) ~[?:?]
  1> 	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) ~[?:?]
  1> 	at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]
  1> 	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
  1> 	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) ~[?:?]
  1> 	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:?]
  1> 	at org.elasticsearch.xpack.core.ssl.SSLService$ReloadableTrustManager.checkServerTrusted(SSLService.java:594) ~[x-pack-core-6.3.0-SNAPSHOT.jar:6.3.0-SNAPSHOT]
  1> 	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1601) ~[?:?]
  1> 	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[?:?]
  1> 	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
  1> 	at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
  1> 	at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
  1> 	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_172]
  1> 	at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
  1> 	at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	... 15 more

And:

  1> [2018-05-25T08:17:38,683][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [node_s1] exception caught on transport layer [NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:30280, remoteAddress=/127.0.0.1:37174}], closing connection
  1> io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: null cert chain
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
  1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
  1> Caused by: javax.net.ssl.SSLHandshakeException: null cert chain
  1> 	at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529) ~[?:?]
  1> 	at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[?:?]
  1> 	at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) ~[?:?]
  1> 	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]
  1> 	at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_172]
  1> 	at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:281) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	... 15 more
  1> Caused by: javax.net.ssl.SSLHandshakeException: null cert chain
  1> 	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
  1> 	at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[?:?]
  1> 	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) ~[?:?]
  1> 	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:318) ~[?:?]
  1> 	at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1935) ~[?:?]
  1> 	at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:237) ~[?:?]
  1> 	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
  1> 	at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
  1> 	at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
  1> 	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_172]
  1> 	at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
  1> 	at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	... 15 more
  1> [2018-05-25T08:17:38,694][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [programmatic_transport_client] exception caught on transport layer [NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:37174, remoteAddress=/127.0.0.1:30280}], closing connection
  1> io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Received fatal alert: bad_certificate
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
  1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
  1> Caused by: javax.net.ssl.SSLException: Received fatal alert: bad_certificate
  1> 	at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]
  1> 	at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[?:?]
  1> 	at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634) ~[?:?]
  1> 	at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800) ~[?:?]
  1> 	at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083) ~[?:?]
  1> 	at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907) ~[?:?]
  1> 	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]
  1> 	at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_172]
  1> 	at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:281) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	... 15 more
@droberts195 droberts195 added >test-failure Triaged test failures from CI v6.3.0 :Security/Security Security issues without another label labels May 25, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-security

@jpountz jpountz removed the v6.3.0 label Jun 13, 2018
@jkakavas
Copy link
Member

This one looks like the node attempted to communicate over http

455300000064000000000000000108004d3603010d417574686f72697a6174696f6e2e4261736963206447567a64463931633256794f6e67746347466a617931305a584e304c58426863334e3362334a6b0016696e7465726e616c3a7463702f68616e647368616b6500

-->

ESd��M6��
Authorization.Basic dGVzdF91c2VyOngtcGFjay10ZXN0LXBhc3N3b3Jk�internal:tcp/handshake

and dGVzdF91c2VyOngtcGFjay10ZXN0LXBhc3N3b3Jk is Base64 encoded test_user:x-pack-test-password

None of these tests have failed since, but keeping this open until I ca n figure out what could have caused the failure

@jkakavas jkakavas self-assigned this Jun 15, 2018
@original-brownbear original-brownbear mentioned this issue Jun 28, 2018
Merged
@imotov
Copy link
Contributor

imotov commented Jul 9, 2018

I cannot quite figure out if this is the same issue or something completely different:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.3+multijob-unix-compatibility/os=sles/174/consoleText

@ywelsch
Copy link
Contributor

ywelsch commented Jul 11, 2018

More of this?
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.3+multijob-windows-compatibility/177

@danielmitterdorfer
Copy link
Member

Another instance in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=java10,ES_RUNTIME_JAVA=java11,nodes=virtual&&linux/209/console

The reproduction line is:

./gradlew :x-pack:plugin:security:test \
  -Dtests.seed=B0DEA59D380A135F \
  -Dtests.class=org.elasticsearch.xpack.security.transport.ssl.SslMultiPortTests \
  -Dtests.method="testThatTransportClientWithOnlyTruststoreCannotConnectToClientProfile" \
  -Dtests.security.manager=true \
  -Dtests.locale=sr-RS \
  -Dtests.timezone=US/Pacific

@cbuescher
Copy link
Member

@jkakavas I got a recent one here:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=debian/87/consoleFull

Doesn't reproduce, but I guess we don't mute these to gather more info and they fail not too often?

@aaparent

This comment has been minimized.

Sign in to view
@jkakavas

This comment has been minimized.

Sign in to view
@aaparent

This comment has been minimized.

Sign in to view
@jkakavas
Copy link
Member

I did a poor job of looking into the actual failure here and capturing the logs and what actually failed in these tests. There is no way to get to the bottom of this now, as the build logs are not available any more, so I will close this to remove noise from the list of >test-failures and re-open if there is a similar issue in later versions and investigate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jkakavas jkakavas

Labels
:Security/Security Security issues without another label >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@jpountz @imotov @danielmitterdorfer @ywelsch @droberts195 @jkakavas @cbuescher @elasticmachine @aaparent