[BUG] Securityadmin error: Not a valid protocol version: This is not an HTTP port.

[prudhvigodithi]

What is the bug?
Running /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh on port 9300 throws error Caused by: org.apache.http.ParseException: Not a valid protocol version: This is not an HTTP port
Full Error log:

**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to my-first-cluster.default.svc.cluster.local:9300 ... done
ERR: An unexpected RuntimeException occured: error while performing request
Trace:
java.lang.RuntimeException: error while performing request
	at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:932)
	at org.opensearch.client.RestClient.performRequest(RestClient.java:307)
	at org.opensearch.client.RestClient.performRequest(RestClient.java:295)
	at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:464)
	at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:161)
Caused by: org.apache.http.ProtocolException: Not a valid protocol version: This is not an HTTP port
	at org.apache.http.impl.nio.codecs.AbstractMessageParser.parse(AbstractMessageParser.java:209)
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:245)
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.http.ParseException: Not a valid protocol version: This is not an HTTP port
	at org.apache.http.message.BasicLineParser.parseProtocolVersion(BasicLineParser.java:148)
	at org.apache.http.message.BasicLineParser.parseStatusLine(BasicLineParser.java:366)
	at org.apache.http.impl.nio.codecs.DefaultHttpResponseParser.createMessage(DefaultHttpResponseParser.java:112)
	at org.apache.http.impl.nio.codecs.DefaultHttpResponseParser.createMessage(DefaultHttpResponseParser.java:50)
	at org.apache.http.impl.nio.codecs.AbstractMessageParser.parseHeadLine(AbstractMessageParser.java:156)
	at org.apache.http.impl.nio.codecs.AbstractMessageParser.parse(AbstractMessageParser.java:207)
	... 11 more

From 2.0 release notes I see Transport client is deprecated, however the documentation still says for -p

-p | OpenSearch port. Default is 9300—not the HTTP port.

How can one reproduce the bug?
Executing /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh on port 9300

What is the expected behavior?
For 1.3.3, this error wont show up and securityadmin.sh executes without any error, connects to cluster and creates a security index, works as expected.

What is your host/environment?

• OS: 2.0.1

Do you have any additional context?
Following the issue
opensearch-project/security#1578, I see the TransportClient is deprecated, so ideally the expected port is rest http port, could be even 9200 for OpenSearch 2.0.1, but however the documentation still says to use Default 9300 port.

This issue is raised to help OpenSearch Kubernetes Operator compatible with 2.0.0 series of OpenSearch.
opensearch-project/opensearch-k8s-operator#176

[pawelw1]

@prudhvigodithi The 2.0.x documentation has been updated with the correct port.

https://opensearch.org/docs/latest/security-plugin/configuration/security-admin/#basic-usage

[prudhvigodithi]

Hey @pawelw1, https://opensearch.org/docs/latest/security-plugin/configuration/security-admin/#opensearch-settings, I see as 9300 as transport port?, but ya basic usage shows 9200, should we consider changing the doc that says ? -p | OpenSearch port. Default is 9300—not the HTTP port. or is there any specific reason to add 9300 ?
Thank you

[prudhvigodithi]

Found a similar forum post
https://forum.opensearch.org/t/securityadmin-sh-uses-http-port-9200-and-not-transport-port-9300-in-opensearch-2-0-0/9760

[salacr]

Hi
I don't know if it's related but when I change the port to 9200 I get an error

"javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?"

[pawelw1]

@prudhvigodithi I know that cross-cluster replication and search still using port 9300.

[pratikshavyas]

Hi @Naarcha-AWS
Is this issue related to https://forum.opensearch.org/t/securityadmin-sh-errors-when-connecting-opensearch-through-http/9806 ?
Here it is mentioned that "It is needed to enable SSL on HTTP to get securityadmin.sh working."
If this issue is related to this thread and I see that it is marked as bug and milestone is added as 2022-Q3, then can you please share plan on fixing this and what is the way moving forward?
Thanks.

[JeffHuss]

Looking in to this. The security script invokes https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/tools/SecurityAdmin.java which seems to indicate that the default port is 9200.

@pratikshavyas this issue is filed against the documentation. Can you please help me understand what you mean when you ask "can you please share plan on fixing this and what is the way moving forward?"

[pratikshavyas]

Hi @JeffH-AWS
I wanted to check about "plugins.security.ssl.http.enabled" setting in opensearch. This value can be set to false to disable TLS on Rest layer. However, If I set it to false, securityadmin.sh scripts fails with "javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?" error.
So, I wanted to know if there is any plan on fixing this.

[pawelw1]

@pratikshavyas The reason why securityadmin.sh fails is the fact that you must use it with admin TLS certificates defined in plugins.security.authcz.admin_dn of opensearch.yml file. TLS certificate authentication always runs over a secured connection and securityadmin.sh only follows that logic. You cannot make a secured connection with an unsecured endpoint.

That's why you must enable SSL on HTTP by setting plugins.security.ssl.http.enabled to true to use that script. This is not a bug but an effect of disabling authentication and authorization on the transport layer and moving it to the HTTP endpoint.

The mentioned option allows you to disable SSL on HTTP, however securityadmin.sh script is an exception to that.

[cliu123]

In security plugin, the transport port should be changed to http port

[pawelw1]

@cliu123 I think the documentation is correct. This regards the OpenSearch service and not securityadmin.sh.
Port 9300 is still in use. The -p option defines the transport port number. If not selected it will be selected from the 9300 - 9400 pool.
Only authentication and authorization were dropped in the transport layer.

[cliu123]

@cliu123 I think the documentation is correct. This regards the OpenSearch service and not securityadmin.sh. Port 9300 is still in use. The -p option defines the transport port number. If not selected it will be selected from the 9300 - 9400 pool. Only authentication and authorization were dropped in the transport layer.

You're right. If the documentation is not specific for security admin tool, then it's good.
But security admin tool doesn't use 9300 anymore, so the security admin tool needs to be updated to save confusion. I see @setiah already request the author of the PR to update that. Thanks!
The security admin tool has multiple issues, and we plan to deprecate it at some time. Please use security REST API as much as you can.

[pawelw1]

@cliu123 I think the documentation is missing a statement that would explain exactly what was deprecated in terms of the transport layer.

1. securityadmin.sh uses port 9200 instead of port 9300
2. deprecation of the authentication and authorization in the transport layer doesn't deprecate transport layer communication and OpenSearch still requires ports 9300 - 9400.

[prudhvigodithi]

SecurityAdmin for OpenSearch 1.x the default port is still 9300 (which is the transport port), for 2.x the port is http port 9200 or the http port with environmental variable cluster setting http.port
May be we have to consider updating the documentation in accordingly.

[cliu123]

1. securityadmin.sh uses port 9200 instead of port 9300

Correct!

2. deprecation of the authentication and authorization in the transport layer doesn't deprecate transport layer communication and OpenSearch still requires ports 9300 - 9400.

I don't think the authN and authZ in transport layer have been deprecated. Only TransportClient has been removed and has replaced with HighLevelRestClient.

[pawelw1]

@cliu123 TransportClient authentication and authorization according to opensearch-project/security#1701

[pawelw1]

@pratikshavyas Version 1.x is still using TransportClient and port 9200. This change was introduced in version 2.0.0

https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.0.0.md

[JeffHuss]

It sounds like we had a few different topics conflated here.

1. Expected behavior when running securityadmin.sh while plugins.security.ssl.http.enabled is set to false..
2. Accuracy of the documentation regarding the port being used by the securityadmin.sh script and by intra-cluster communication.

Regarding item 1, it sounds like this has been cleared up. Does anything further need answered here?

If we're all good on that front, what needs adjusted in the docs and for what specific version(s)?

[JeffHuss]

Putting this on the backlog until someone is able to chime in about my previous question.