-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] Breaking changes with opensearch.http.override_main_response_version=true #3023
Comments
@shwetathareja Could you help find a repro case where this is true (using |
@mch2 @saratvemulapalli I believe we were originally planning to do (a) for 2.0, is that accurate? do we no longer want to? |
While 2.0.0 introduces breaking changes to the rest interface, the clients abstraction the rest calls. Is that abstraction enough that a 1.x client is still logically comparable via the abstraction layer? Assuming it is, we add support for OpenSearch 2.X in a new OpenSearch client 1.X release on inspection of the api version, it would be valuable to allow rolling backends forward while front ends only need the minor update. |
We have similar case on Inclusive naming and the plan to make sure you can do rolling upgrade without any issues. I would avoid doing the upgrade on server and client on the same time. |
OpenSearch 2.0 removed the classic mapping templates. Clients may be expecting these are available if OpenSearch is advertising itself as a 7.10.2 cluster. I pushed for the addition of this property ( |
This is how I understand it. Now us being at 2.0, this override is not needed anymore. Because the clients on older major (i.e 1.x) cannot talk to OpenSearch 2.0 as we have many breaking changes in our Rest APIs. |
Not entirely correct. Since OpenSearch Core 1.0.0 GA is nothing more than refactored, reversioned Legacy 7.10.x OpenSearch 2.0 is wire compatible back to legacy 7.10.0. We need to retain spoofing to ensure compatibility w/ 7.10.0 nodes. OpenSearch 3.0 will remove any/all compatibility w/ legacy versions so spoofing can be removed at that time. |
Correct. Types were deprecated in legacy 7.x (Nov 1, 2018); ergo they were removed since clients had over three years to stop depending on types. |
We use a couple supporting tools that still depend on the 7.x ES client libraries. We look forward to disabling the 7.10.2 spoofing as soon as possible to remove our reliance on this workaround. I based our plans on the answer I received at the community meeting on 2021-10-19: "Do we expect the ES version compatibility flag to be removed in 2.0?" Kyle's answer was to expect the version compatibility flag to be re-evaluated prior to 3.0. If removed, it would be removed no earlier than 3.0. If this is no longer possible and it becomes necessary for us to remain on 1.x until we no longer rely on the workaround, will 1.x continue to get security patches for while? |
Interesting.. thanks for letting me know. |
@saratvemulapalli I think it is reversed: Rest clients which support 2.0 and we do want to support them for 7.10. That would be the upgrade path: upgrade client to 2.0, it should work for both versions, than start upgrading clusters. |
Every major version is expected to be backward compatible with last minor version of majorVersion - 1. So OpenSearch 2.0 should be backward compatible with ES 7.10. But this flag - @nknize this flag |
I think there is some dissonance between OpenSearch 2.0.0 saying it's 7.10.2 yet not having the same API (after the removal of mapping types). It would be completely logical for someone to have a tool or script that tests for 7.10.2 and continues to use mapping types (maybe only complaining in the logs or something). Now if they move to OpenSearch 2.0.0 w/ I'm not sure the project should be shy about 2.0.0 having breaking changes - that's what major versions are for after all. If put myself in the mindset of a user, I'd rather the software prevent something nasty at the start than it cropping up in a running process. |
The main purpose of the Keeping this flag for 2.0 would probably mean to return not 7.10 but 8.0, to match Elasticsearch breaking API changes, but none of the Elasticsearch native clients would support that. I suspect there is small number of projects that prefer to use raw HTTP and not rely on any Elasticsearch / Opensearch clients at all, like [1], some of those could check for version in response. I think removing |
That is the correct upgrade path for clients depending on the HLRC.
Correct.
💯
Agree. I've slept since we removed the transport client December of last year so we don't need to retain transport client compatibility (nor should the REST client spoofing matter here). +1 for removing the |
|
|
I've opened up an issue to remove the documentation for this feature from 2.0 and up. |
but is the upgrade path from ES 7.10.2 HLRC and/or OpenSearch 1.x HLRC to OpenSearch 2.x HLRC or to opensearch-java? elastic said their elasticsearch-java is now the official java client and should be used instead and e.g. spring-data-elasticsearch is now refactoring to support both clients (and eventually also an OpenSearch client): spring-projects/spring-data-elasticsearch#1973 & spring-projects/spring-data-elasticsearch#1853 upgrading from one HLRC to another HLRC is probably easiest (for those projects using it directly) but moving directly to opensearch-java is probably the more long-term approach. and since opensearch-java is not part of OpenSearch itself i would presume it would lend itself much more to being compatible over multiple OpenSearch major versions than the HLRC which is an integral part of OpenSearch and makes use of internal OpenSearch libraries? |
Many users of Beats and Logstash have switched to OpenSearch and Elastic has made it clear that these products do not intend to support OpenSearch. There are explicit version checks to verify compatibility with Elasticsearch that were circumvented using the spoofing mechanism. Without the spoof users of some filebeat processors are now DOA after upgrading to 2.0. I have opened a PR to revert the removal of spoofing so these affected users can continue to function by explicitly setting the version. Yes, breaking compatibility with upgraded clients might be a side effect if users explicitly set this setting to function with older versions of Beats and Logstash. For this reason we need the REST API Versioning mechanism so this will, hopefully, be a temporary situation. Longer term we need a plan to either fork beats and logstash or replace it with an alternative like FluentBit. |
Hello everyone I'm currently using org.elasticsearch : elasticsearch-hadoop and elasticsearch-hadoop-mr since there is no opensearch-hadoop library for JAVA Now with Opensearch 2.0.0 i cannot use anymore this option, so i 'm currently blocked . Anyone has an idea on a workaround ? |
I was in same boat and found that If all you need is support for indexing, it is a single line change: Just change that line to:
Then just use Hope this helps. |
OpenSearch 2.0.1 has been released a few days (weeks?) ago which has the option back in, see the release notes |
There are a number of breaking changes in 2.0 RESTful APIs. I suspect (still need to confirm) that breaks
opensearch.http.override_main_response_version=true
(https://opensearch.org/docs/latest/clients/agents-and-ingestion-tools/index/).I think we have two options.
a) Remove
opensearch.http.override_main_response_version=true
.b) Ensure backwards compatibility.
This is tangentially related to opensearch-project/opensearch-clients#17.
The text was updated successfully, but these errors were encountered: