Rest API version compatibility #51816
Comments
|
reserving first comment for additional information |
|
pinging @elastic/es-clients |
Providing a REST infrastructure to handle compatible APIs. Rest APIs that were removed in 8 will be available only if an HTTP request had a compatible header. At the moment is is expecting an Accept header to contain application/vnd.elasticsearch+json;compatible-with=7. This might be changed though when #52370 is resolved. The REST changes contain enriching RestRequest with a parameter Compatible-With. The value is taken from Accept header. See RestRequest creation. Used when deciding if a request to a compatible handler can be dispatched. See RestController. //TODO this at the moment can be simplified to only use a value from a header. However in rest layer we often have only access to Params, so headers won't be accessible when implementing some compatible code enriching XContentBuilder with a compatible version value. See AbstractRestChannel. Used when shape of a response is changed. see DocWriteResponse and GetResult MediaType parsing change - see XContentType. because we expect a different media type now application/vnd.elasticsearch+json;compatible-with=7 (when previously application/json or similar) testing changes - compat tests will have a compatible header set. see AbstractCompatRestTest.java Compatible rest handlers - based on RestIndexAction and RestGetAction. See modules/rest-compatibility. These extend already existing handlers and register them in a RestCompatPlugin under a typed paths. They will only be accessible when a compatible header is present (use of compatibilityRequired method from RestHandler interface) Without this PR 283 tests from 7.x are failing. - rest api spec tests only with this PR 228 tests are passing - with almost all tests from index/* package passed (missing 2 tests that require change for include_type_param, to be added in a new PR) relates #51816
|
Will this be available in ES 8? Does it mean that from ES 7 and onwards, we don't have to think about breaking Rest Api during the upgrade? |
|
@Musfiqur01 yes it will be available in ES8 accepting version 7 requests. This will allow you to upgrade your server first and your application with a client later. You will have to deal with breaking changes during your application upgrade. |
Content-Type and Accept headers can be populated with a versioned form of media types like application/vnd.elasticsearch+json;compatible-with=7 when previously it was simple application/json or (cbor, yaml..) - this is still supported. Extending MediaTypeParser to validate the parameters. relates #51816
Per REST endpoint media types declaration allows to make parsing/validation more strict. If a media type was declared only in one endpoint (for instance CSV in SQL endpoint) it should not be allowed to parse that media type when using other endpoints. However, the Compatible API need to be able to understand all media types supported by Elasticsearch in order to parse a compatible-with=version parameter. This implies that endpoints need to declare which media type they support and how to parse them (if introducing new media types - like SQL). How to parse: MediaType interface still serves as an abstraction on top of XContentType and TextFormat. It also has a declaration of mappings String-MediaType with parameters. Parameters declares the names of parameters and regex to validate its values. This instructs how to perform the parsing. For instance - XContentType.JSON has the mapping of application/vnd.elasticsearch+json -> JSON and allows parameters compatible-with=\d and charset=utf-8 MediaTypeParser was simplified into ParsedMediaType class with static factory method for parsing. How to declare: RestHandler interface is extended with a validAcceptMediaTypes which returns a MediaTypeRegistry - a class that encapsulates mappings of string (type/subtype) to MediaType, allowed parameters and formatPathParameter values. We only need to allow of declaration of valid media types for Accept header. Content-Type valid media types are fixed to XContentType instances - json, yaml, smile, cbor. relates #51816
Per https://tools.ietf.org/html/rfc7231#section-3.1.1.1 MediaType can have spaces around parameters pair, but do not allow spaces within parameter. That means that parameter pair forbids spaces around `=` follow up after elastic#64406 (comment) relates elastic#51816
Per https://tools.ietf.org/html/rfc7231#section-3.1.1.1 MediaType can have spaces around parameters pair, but do not allow spaces within parameter. That means that parameter pair forbids spaces around = follow up after #64406 (comment) relates #51816
Browsers are sending media ranges with quality factors on Accept header. We should ignore the value and responde with applicaiton/json closes elastic#64689 relates elastic#51816
v7compatibilityNotSupportedTests was introduced to make it easier to track tests that have been identified as not needing compatible changes and those that still need to be checked. We have checked all tests now and the separate list is no longer needed. relates elastic#51816 relates elastic#73912
Related to elastic#51816 / elastic#68905. Adds back the _xpack/ml routes when using rest compatibility for a request.
|
How can I disable this in latest Java client? |
|
@kid1412621 why do you want to disable it? Rest api compatibility is a server feature that is used when an old client is used against new server (v8 server vs v7 client). You can disable this in your client by using |
'Coz I'm using 7.10 on the server side, it seems not support this header. |
|
@kid1412621 good point! The ability to parse Looks like you are using a newer client than a server? |
Yeah,client: 7.16. server: 7.10. |
Hi there, it seems there is no |
|
This will be included in 8.0 of Elasticsearch. |
Systems/clients that wish to interact with Elasticsearch are required to do so over the REST API. Breaking changes to the REST API (in major versions) can introduce challenges for these systems/clients.
The preferred upgrade path is 1) upgrade Elasticsearch 2) upgrade the systems/clients that are interacting with Elasticsearch. However, major version upgrades are more likely 1) upgrade systems/clients to be compatible with the next major version of Elasticsearch 2) upgrade Elasticsearch. Upgrading the systems/clients that interact with Elasticsearch before upgrading Elasticsearch can create an high orchestration upgrade event. The administrators of Elasticsearch need to know that all systems that interact with Elasticsearch are compatible with the new major version before they can upgrade.
Rest API version compatibility seeks to address better de-coupling of systems/clients and server upgrades. Rest API version compatibility will ensure that requests and responses that worked in the prior major version will continue to function in the next major version without error (but may behave differently). The intent is to allow Administrators to upgrade the Elasticsearch server to new major versions without requiring (or at least minimal) changes to the systems/clients that talk to Elasticsearch. Further, Rest API version compatibility will not require specific deployment architectures or complex configuration. Rest API version compatibility will be enabled/disabled per HTTP request via an HTTP header.
Rest API version compatibility can be defined along two axis: 1) shape 2) behavior.
Shape
Shape is being defined as the structure of the HTTP request. Shape includes the JSON body for the request/response, the HTTP verb (GET, POST, etc.), the HTTP path (/foo/bar), HTTP parameters (?foo=bar), etc. Changes to the shape of request and responses (with minor exception) will be covered by REST API version compatibility.
For example, in 8.0 REST paths that started _xpack will no longer be available and will be replaced with a similar path but without the
_xpackin the URL. If a system/client relied upon any of those URLs with_xpackin the path, those systems/clients would need to be upgraded prior to upgrading Elasticsearch from 7.x to 8.x.With Rest API version compatibility (assuming it was requested via the header), those paths will still be honored since they were valid in 7.x. Any shape honored in 7.x will be honored in 8.x when Rest API version compatibility is requested. The request header that enables Rest API version compatibility can be requested to be compatible with a specific major version, for example, a request can request compatibility with v7. When Elasticsearch server is running 7.x and compatibility is requested for v7, the header will essentially be ignored. However, as soon as Elasticsearch is upgraded (from 7.x to 8.x) and the system/client is still sending a header requesting compatibility with v7, the 8.x server will read that header and ensure that any shape elements that worked in 7.x will continue to work in 8.x. This type of compatibility is a close cousin to traditional wire or binary compatibility.
For these shape based breaking changes across major versions, Rest API version compatibility allows the server to be upgraded independently to any required changes for the client.
Behavior
Rest API version compatibility as described here is distinctly different then most SAAS version-ed APIs. SAAS offerings can allow for stronger guarantees for how a service behind the version of REST API behaves then software you can download and run yourself. With Rest API version compatibility, the shape of the REST API (with minor exception) will be guaranteed to be compatible with the previous minor version, however, the behavior of the server may differ from the last prior major version.
For example, the default number of shards per index was changed in major version. Had Rest API version compatibility existed then it would have no impact on that change since that change was strictly about how Elasticsearch behaves. Many changes across majors (and minors) are changes to the behavior of the service and will be not influenced by REST API version compatibility.
Another example is that in 8.0 many of the deprecated discovery.zen are removed since a new cluster coordination subsystem was implemented in 7.x and the old one will be completely removed in 8.0. Since cluster coordination (and the related settings) is a behavior, not an REST API, it is not part Rest API version compatibility.
Shape && Behavior
Quite often shape and behavior change go hand-in-hand. Not all changes to the shape are as simple as a changing URL. Many of the breaking changes to the REST API also include some form of behavior change as well.
For 8.0 the most notable change that encompasses both shape and behavior is the complete removal of types. In versions prior to 8.0, many REST APIs supported the concept of a type within an index. 6.x and 7.x deprecated all functionality related to types, so that in 8.0 types will be completely removed. However, many REST API shape elements in 7.x are tightly correlated to the concept of a type. For example, in 7.x you could create an index with a custom type (e.g.
PUT foo/custom_type/1 { "a" :"b"}). While that functionality is deprecated in 7.x, it is still supported in 7.x. However in 8.0, how can we still support the type based URLs for Rest API version compatibility if types will be completely removed ?A form of behavior compatibility will be introduced on a case-by-case basis. Behavior compatibility’s primary focus is to best-effort honor the intent of the request and response and provide shape compatibility. In most cases behavior compatibility will not be identical to the prior version.
For the complete removal of types, the behavior compatibility strategy will be to internally map any type based request to a typeless based request. For example, (if compatibility is requested) the server will still accept typed based request (e.g.
PUT foo/custom_type/1 { "a" :"b"}), but will internally convert to a typeless request (e.g.PUT foo/_doc/1 { "a" :"b"}). The goal here is to ensure that the requested shape is still compatible, honoring the intent of the request, but not behave exactly as it did in the prior version.The behavior compatibility will be best effort to honor the intent of the request and preserve the format of the response. However, this will not always be sufficient for all use cases. In the above example, if the application logic was dependent on
custom_typethe behavior compatibility would not be sufficient. The service/client will still need to upgrade prior to Elasticsearch for those use-cases. Since compatibility will be requested per HTTP request via a header, clients can opt-out or opt-in per request.The documentation will be updated to help clearly identify exactly how the behavior compatibility will be achieved to allow consumers better understand if compatibility will work for their use cases.
Deprecations and Compatibility
Elasticsearch follows a pattern of only introducing breaking changes in major release (with very few exceptions). All breaking changes are deprecated in a major version before they are implemented in the next major version. Rest API version compatibility will not change any of the existing deprecation or release strategies. However, Rest API version compatibility will extend the ability to use deprecated API for a full extra major release if (and only if) REST compatibility is requested for that API and version. Also, note that due the the case-by-case of behavior compatibility, the behavior that backs the REST API may not be sufficient for all use cases.
Additionally, a new type of warning will be introduced. A compatibility warning, much like the deprecation warnings will be introduced to help systems/clients understand where and what REST API version compatibility is doing. REST API version compatibility should be viewed as a compatibility layer to help systems/clients decouple their upgrade life-cycles for each other. It is not a long term strategy to keep using deprecated features.
Scope
Rest layer wire compatibility, but not behavior compatibility.
Duration
~ 12 months
List of high level tasks
TODO
Parking lot (things discussed, but need to revisit)
[ ] The relationship between
AcceptandContent-Typeheaders with/out compatibility. see #53228 (comment) PR to address thisThe text was updated successfully, but these errors were encountered: