You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Please note! This is a template to file a bug for the OpenSearch security plug-in. If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our vulnerability reporting page or directly via email to aws-security@amazon.com. Please do not create a public GitHub issue.
Describe the bug
We need to set up 1.3 test cluster for PenTest. So I tried to create a 1.3 cluster with multi-node cdk. The test cluster created successfully. But the cluster can't work.
OpenSearch can start but can't access any API like _cat/indices
Login to one EC2 node in ClientASG and check log in /opensearch-1.3.0/logs/ml-pentest.log
Can find such error
[2022-02-16T00:06:15,978][WARN ][r.suppressed ] [client-9a222d1f] path: /_nodes, params: {settings_filter=plugins.security.ssl.transport.pemkey_filepath,plugins.security.cert.oid,plugins.security.enable_snapshot_restore_privilege,plugins.security.audit.config.pemtrustedcas_filepath,reindex.ssl.supported_protocols,opendistro_security.compliance.history.external_config_enabled,plugins.security.ssl.transport.truststore_password,plugins.security.ssl.transport.keystore_alias,plugins.security.ssl.transport.keystore_type,plugins.security.check_snapshot_restore_write_privileges,plugins.security.advanced_modules_enabled,reindex.ssl.truststore.password,opendistro_security.*,plugins.security.ssl.transport.truststore_alias,plugins.security.unsupported.accept_invalid_config,plugins.security.audit.config.webhook.format,plugins.security.audit.config.webhook.ssl.pemtrustedcas_filepath,plugins.security.audit.config.pemkey_password,plugins.security.background_init_if_securityindex_not_exist,plugins.security.ssl.transport.enabled,plugins.security.audit.config.webhook.ssl.verify,plugins.security.ssl.transport.keystore_keypassword,plugins.security.protected_indices.roles,plugins.security.audit.config.index,plugins.security.ssl.http.keystore_alias,plugins.security.audit.config.webhook.url,plugins.security.allow_unsafe_democertificates,plugins.security.unsupported.restapi.allow_securityconfig_modification,plugins.security.allow_default_init_securityindex,plugins.security.ssl.http.truststore_type,plugins.security.ssl.transport.keystore_password,plugins.security.audit.config.log4j.logger_name,reindex.ssl.keystore.key_password,reindex.ssl.truststore.type,plugins.security.ssl.http.keystore_filepath,plugins.security.kerberos.krb5_filepath,plugins.security.ssl.transport.keystore_filepath,plugins.security.ssl.client.external_context_id,plugins.security.ssl.transport.pemcert_filepath,plugins.security.unsupported.inject_user.enabled,plugins.security.ssl.http.pemkey_password,opendistro_security.audit.enable_rest,reindex.ssl.key_passphrase,opendistro_security.audit.resolve_bulk_requests,plugins.security.restapi.password_validation_regex,plugins.security.unsupported.allow_now_in_dls,plugins.security.audit.config.type,plugins.security.ssl.transport.truststore_type,plugins.security.audit.threadpool.max_queue_len,plugins.security.audit.config.pemcert_filepath,plugins.security.audit.config.password,plugins.security.ssl.transport.enforce_hostname_verification,plugins.security.unsupported.restore.securityindex.enabled,plugins.security.*,plugins.security.config_index_name,plugins.security.audit.config.pemtrustedcas_content,plugins.security.ssl.transport.pemtrustedcas_filepath,reindex.ssl.truststore.path,plugins.security.ssl.http.pemcert_filepath,reindex.ssl.keystore.password,reindex.ssl.certificate_authorities,plugins.security.compliance.disable_anonymous_authentication,opendistro_security.audit.resolve_indices,plugins.security.audit.config.pemcert_content,plugins.security.ssl.http.truststore_password,plugins.security.ssl.http.crl.prefer_crlfile_over_ocsp,plugins.security.audit.config.pemkey_filepath,opendistro_security.compliance.history.read.metadata_only,opendistro_security.compliance.history.write.log_diffs,plugins.security.ssl.transport.extended_key_usage_enabled,plugins.security.unsupported.load_static_resources,plugins.security.compliance.salt,plugins.security.filter_securityindex_from_all_requests,reindex.ssl.certificate,plugins.security.ssl.http.crl.validate,reindex.ssl.verification_mode,opendistro_security.audit.enable_transport,plugins.security.ssl.http.crl.validation_date,plugins.security.audit.config.enable_ssl_client_auth,plugins.security.ssl.http.pemtrustedcas_filepath,plugins.security.ssl.http.keystore_keypassword,plugins.security.ssl_only,opendistro_security.compliance.history.write.metadata_only,opendistro_security.audit.log_request_body,plugins.security.unsupported.inject_user.admin.enabled,plugins.security.audit.config.webhook.ssl.pemtrustedcas_content,plugins.security.ssl.http.pemkey_filepath,plugins.security.audit.config.username,plugins.security.ssl_cert_reload_enabled,plugins.security.ssl.http.crl.disable_crldp,plugins.security.audit.threadpool.size,plugins.security.roles_mapping_resolution,plugins.security.audit.config.pemkey_content,reindex.ssl.keystore.path,plugins.security.ssl.http.enabled,plugins.security.kerberos.acceptor_keytab_filepath,plugins.security.system_indices.enabled,plugins.security.audit.config.cert_alias,reindex.ssl.client_authentication,reindex.ssl.keystore.type,plugins.security.audit.config.log4j.level,plugins.security.ssl.transport.truststore_filepath,plugins.security.audit.type,plugins.security.disabled,reindex.ssl.cipher_suites,plugins.security.disable_envvar_replacement,plugins.security.restapi.password_validation_error_message,plugins.security.ssl.http.crl.check_only_end_entities,opendistro_security.compliance.history.internal_config_enabled,opendistro_security.audit.exclude_sensitive_headers,plugins.security.ssl.http.enable_openssl_if_available,plugins.security.ssl.http.clientauth_mode,plugins.security.protected_indices.enabled,plugins.security.unsupported.disable_rest_auth_initially,reindex.ssl.key,plugins.security.ssl.http.crl.file_path,plugins.security.audit.config.enable_ssl,plugins.security.kerberos.acceptor_principal,plugins.security.cert.intercluster_request_evaluator_class,reindex.ssl.keystore.algorithm,plugins.security.audit.config.verify_hostnames,plugins.security.ssl.http.keystore_type,plugins.security.ssl.http.truststore_filepath,plugins.security.cache.ttl_minutes,plugins.security.ssl.transport.pemkey_password,plugins.security.system_indices.indices,plugins.security.ssl.transport.enable_openssl_if_available,plugins.security.ssl.http.keystore_password,plugins.security.ssl.http.crl.disable_ocsp,plugins.security.ssl.http.truststore_alias,plugins.security.ssl.transport.principal_extractor_class,plugins.security.protected_indices.indices,plugins.security.ssl.transport.resolve_hostname,plugins.security.unsupported.disable_intertransport_auth_initially, filter_path=nodes.*.version,nodes.*.http.publish_address,nodes.*.ip}
org.opensearch.OpenSearchSecurityException: Unexpected exception cluster:monitor/nodes/info
at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:376) [opensearch-security-1.3.0.0.jar:1.3.0.0]
at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:154) [opensearch-security-1.3.0.0.jar:1.3.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:99) [opensearch-performance-analyzer-1.3.0.0.jar:1.3.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:169) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:97) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:108) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:95) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:433) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:730) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.client.support.AbstractClient$ClusterAdmin.nodesInfo(AbstractClient.java:813) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.rest.action.admin.cluster.RestNodesInfoAction.lambda$prepareRequest$0(RestNodesInfoAction.java:87) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:128) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.security.filter.SecurityRestFilter$1.handleRequest(SecurityRestFilter.java:126) [opensearch-security-1.3.0.0.jar:1.3.0.0]
at org.opensearch.rest.RestController.dispatchRequest(RestController.java:306) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:392) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.rest.RestController.dispatchRequest(RestController.java:235) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.security.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [opensearch-security-1.3.0.0.jar:1.3.0.0]
at org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:361) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:440) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:351) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:55) [transport-netty4-client-1.3.0.jar:1.3.0]
at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:41) [transport-netty4-client-1.3.0.jar:1.3.0]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:71) [transport-netty4-client-1.3.0.jar:1.3.0]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1371) [netty-handler-4.1.73.Final.jar:4.1.73.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234) [netty-handler-4.1.73.Final.jar:4.1.73.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283) [netty-handler-4.1.73.Final.jar:4.1.73.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.73.Final.jar:4.1.73.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.73.Final.jar:4.1.73.Final]
at java.lang.Thread.run(Thread.java:833) [?:?]
Expected behavior
Cluster should work correctly
Plugins
All plugins.
Screenshots
No
Host/Environment (please complete the following information):
m5.xlarge, Amazon Linux
The text was updated successfully, but these errors were encountered:
@ylwu-amzn Thanks for reporting the issue! But I'm unable to reproduce the issue. I got the expected response on 1.3 cluster(single node). Is there any misconfiguration on the cluster?
GET _cat/indices
yellow open security-auditlog-2022.02.17 2fKog0RzSECpSgI2JotEEw 1 1 5 0 80kb 80kb
green open .kibana_92668751_admin_1 M40rhZvoTFy3Ig7zi0O-3Q 1 0 1 0 5kb 5kb
green open .kibana_1 bXucRphtQVGd_7ay5AdyMA 1 0 6 6 33.2kb 33.2kb
green open .opendistro_security o9rPNe2LT0GcS_9hqGI2rg 1 0 9 0 59.6kb 59.6kb
Please note! This is a template to file a bug for the OpenSearch security plug-in. If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our vulnerability reporting page or directly via email to aws-security@amazon.com. Please do not create a public GitHub issue.
Describe the bug
We need to set up 1.3 test cluster for PenTest. So I tried to create a 1.3 cluster with multi-node cdk. The test cluster created successfully. But the cluster can't work.
To Reproduce
Steps to reproduce the behavior:
main
branch code_cat/indices
ClientASG
and check log in/opensearch-1.3.0/logs/ml-pentest.log
Expected behavior
Cluster should work correctly
Plugins
All plugins.
Screenshots
No
Host/Environment (please complete the following information):
The text was updated successfully, but these errors were encountered: