Yet another Overlay proposal

[darrelmiller]

A continuation of the conversation in #35 but with a new approach.

Updated 2018/12/06 based on Mike's suggestions.

---

OpenAPI Overlays

In recent months we have been discussing various use cases for overlays and various solutions. The following proposal takes a somewhat more radical approach to the problem. It is a more ambitious proposal than the others we have seen before but the additional complexity does allow for supporting many of the scenarios that have been discussed to date.

Overlay Document

An overlay document contains a list of Update Objects that are to be applied to the target document. Each Update Object has a target property and a value property. The target property is a JMESPath query that identifies what part of the target document is to be updated and the value property contains an object with the properties to be overlayed.

Overlay Object

This is the root object of the OpenAPI Overlay document.

Fixed Fields

Field Name	Type	Description
overlay	string	Version of the Overlay specification that this document conforms to.
info	[Info Object]	Identifying information about the overlay.
extends	url	URL to an OpenAPI document this overlay applies to.
updates	[Update Object]	A list of update objects to be applied to the target document.

The list of update objects MUST be applied in sequential order to ensure a consistent outcome. This enables objects to be deleted in one update and then re-created in a subsequent update.

Info Object

This object contains identifying information about the OpenAPI Overlay document.

Fixed Fields

Field Name	Type	Description
title	string	A human readable description of the purpose of the overlay.
version	string	A version identifer for indicating changes to an overlay document.

Update Object

This object represents one or more changes to be applied to the target document at the location defined by the target JMESPath.

Fixed Fields

Field Name	Type	Description
target	string	A JMESPath expression referencing the target objects in the target document.
value	Any	An object with the properties and values to be updated in the target document.

The properties of the Value Object MUST be compatible with the target object referenced by the JMESPath key. When the Overlay document is applied, the properties in the Value Object replace properties in the target object with the same name and new properties are appended to the target object.

Structured Overlays Example

When updating properties throughout the target document it may be more efficient to create a single Update Object that mirrors the structure of the target document. e.g.

overlay: 1.0.0
info:
  title: Structured Overlay
  version: 1.0.0
updates:
- target: "@"
    value:
      info:
        x-overlay-applied: structured-overlay
      paths:
        "/":
          summary: "The root resource"
          get:
            summary: "Retrieve the root resource"
            x-rate-limit: 100
        "/pets":
          get:
            summary: "Retrieve a list of pets"
            x-rate-limit: 100
      components:
      tags:

Targeted Overlays

Alternatively, where only a small number of updates need to be applied to a large document, each Update Object can be more targeted.

overlay: 1.0.0
info:
  title: Structured Overlay
  version: 1.0.0
updates:
- target: paths."/foo".get
    value:
        description: This is the new description
- target: paths."/bar".get
    value:
        description: This is the updated description
- target: paths."/bar"
    value:
        post:
            description: This is an updated description of a child object
            x-safe: false

Wildcard Overlays Examples

One significant advantage of using the JMESPath syntax that it allows referencing multiple nodes in the target document. This would allow a single update object to be applied to multiple target objects using wildcards.

overlay: 1.0.0
info:
  title: Update many objects at once
  version: 1.0.0
updates:
- target: paths.*.get
    value:
      x-safe: true
- target: paths.*.get.parameters[?name=='filter' && in=='query']
    value:
      schema:
        $ref: "/components/schemas/filterSchema"

Array Modification Examples

Due to the fact that we can now reference specific elements of the parameter array, it does open the possibilty to being able to add parameters and potentially remove them using a null value.

overlay: 1.0.0
info:
  title: Add an array element
  version: 1.0.0
updates:
- target: paths.*.get.parameters[length(@)]
    value: 
      name: newParam
      in: query

overlay: 1.0.0
info:
  title: Remove a array element
  version: 1.0.0
updates:
- target: $.paths[*].get.parameters[? name == 'dummy']
    value: null

Proposal Summary

Benefits

• This approach addresses the two distinct approaches of structured overlay vs targeted overlay which suits distinct but equally valid scenarios.
• Addresses the problem of modifying the parameters array and removes the need to replace the entire array when a small change is required.
• Allows sets of related overlays to be stored in a same file.
• Enables updating a set of objects based on a pattern. This might be an effective way of apply common behaviour across many operations in an API.

Challenges

• Tooling will need a JMESPath implementation.
• Large overlays may be slow to process.
• Multiple complex pattern based overlays may cause overlapping updates causing confusing outcomes.

[darrelmiller]

Should this be a distinct spec from OpenAPI?

[darrelmiller]

Should the core document reference the overlay or should the overlay reference the core document? Or should the relationship be dynamic and external to the spec?

Should overlays be composable? Can I reference one overlay from another?

[darrelmiller]

Consensus:

• Core document should not reference the overlay
• Separate the Overlay spec from OAS. (Don't know if it should have a dependency on OAS?)
• No strong objection to JSONPath, but needs to have an actual written spec.
• No objections to extra complexity of structural and targeted overlays

[darrelmiller]

Suggestion for 3.1 that extensions that have a null value have the same meaning as the extension not being present.

[MikeRalphson]

See also http://jmespath.org

One reason I suggest it as an alternative, is that JSONPath defers 'scripting' expressions to the underlying language.

Expressions of the underlying scripting language () can be used as an alternative to explicit names or indices

which we of course have no control over, potentially leading to incompatible overlay documents. In naive implementations this would be a security risk too.

JSONPath's dual dotted and bracketed syntaxes also don't help with trying to get across clear intent.

Just for fun I've implemented the above (but with jmespath). Minimally implemented, it was 22 lines of code (4 of which were blank).

Question: how (if?) are $refs handled a) within the overlay document, and b) if they are found as the result of a target expression in the OAS document being extended? Must that document be de-reffed first?

Question: should we allow a source attribute mututally-exclusive with the value attribute? This would also take a *path expression, and allow moving dynamic values around in the document.

I think I'm in favour of publishing this as a separate spec., because there's really nothing which is OpenAPI-specific about it. Maybe we could issue a Technical Note or similar? (I'd also like to do one for an errata for v2.0 of the main spec.)

[MikeRalphson]

That said, if we prefer to align things a little more to the OAS, then I'd be tempted to slightly rework the structure as follows:

overlay: 1.0.0
info:
  version: 1.0.0
  title: My Nice Overlay
extends: ./openapi.yaml # an optional uri_reference to the base openapi document OR another overlay document
updates:
- target: info
  value:
    description: Overlay applied

[MikeRalphson]

@darrelmiller here by way of an example is an overlay which translates all response object descriptions into German.

[fmvilas]

Let's work on this together. It's gonna be implemented soon in AsyncAPI. BTW, I like the direction it's taking.

[webron]

Just a couple of observations...

Behavior:

• In RAML Overlay equivalent? #35, @jstoiko brought up that in RAML a core definition can be invalid as a result of having overlays defined, as those would make it valid. Is this the case with the OpenAPI overlays? The downside of allowing this is that it makes validation more complicated for the core documents, while the upside is that it allows more flexibility. Another downside - it suggests that the OpenAPI Specification is aware of Overlays.
• It's unclear from the description of multiple overlays can be applied on a single OpenAPI definition. Can overlays be applied to overlays?
• There's a hidden assumption that the post-processed result of a definition and overlay(s) produce a valid definition. Suggest adding a clarification that it may not be the case (this is to ease validation).
• Agreed that it should be an independent spec.

General structure:

• In the 3 objects (Overlay, Info, Update), it's not clear which properties are REQUIRED.
• Are we offering any reusabilty here? Supporting $ref for the Update Object for example.

Overlay Object:

• Missing description(s) field(s).
• extends - unclear if this is required or not, and it seems to suggest the ability to extend a single document only. Do we really need to specify this in the document? Do we need a way to support multiple documents?

Update Object:

• value - saying that the structure of this property (as typed Any) MUST be compatible with the target is a too strong assertion and can lead to a validation nightmare. Suggest changing the assertion to SHOULD. The downside of this is that it may make debugging more complicated.
• target - haven't read the full JMESPath spec yet but:
  • I like the level of flexibility it gives in controlling what's affected.
  • JMESPath does not seem to be 'official'. What would we be referencing from the the the Overlay spec? It also seems non-versioned.
  • By introducing yet-another-expression-language, we're increasing the complexity of the spec. We need to keep in mind that not all our users are very technical, and some are struggling even with things like JSON Schema and JSON References. We can say this is a tooling problem, but we need to be aware of this.
  • As a full expression language, do we support all of it? I'm sure that would lead to edge cases and some crazy things, but it may not be a real concern.
• The use of null to remove properties from a definition is problematic, as we mentioned that it's a valid value for example/examples. We can say that if no value is defined, then it's an implicit delete operation, but it would be more verbose.
• (raised by @MikeRalphson as well) How is $ref handled within a value? Is it taken as an as-is value or is it resolved and then used?

[jstoiko]

Food for thoughts:

• if this is an independent Spec that doesn't necessarily have to be syntactically close to OAS
• and arguably would be less confusing if dramatically different (syntactically)
• and the "patching" aspect is something that we all want to be as flexible as possible (i.e. patching arrays)
• and tooling is also key (read: use existing tooling)

why not use an existing diffing format? I am thinking Unified Diff or a variation thereof.

This would cover both "diffing" and "patching" use cases, with all the benefits of using an existing (and widely used) format and not having to maintain "yet another"(tm) Spec.

[MikeRalphson]

@jstoiko two 'objections' to using a format such as unified diff or json-patch:

1. these formats are hard to write by hand and limit the use of overlays to diffing and patching by machine
2. these formats do not allow targetting multiple paths within a document without repetition

[MikeRalphson]

@webron thoughts on your notes (in case I forget during the call):

extends was intended to be optional, and I wouldn't think allowing an overlay to target an overlay directly would be sensible. However, an overlay could target the result of applying an overlay (or chain of overlays) to a source OAS document. Fanning out to multiple values of extends is possible, but how useful and common would that use-case be in practice?

An OAS document which is invalid because overlays have not been applied is invalid, period. As long as we do not specify that tooling MUST validate the target OAS document before applying overlays we leave ourselves an exit hatch. However, in the real world tools using a parser to ingest the target OAS document will likely choke on an invalid one.

We could reach out to the JMesPath maintainer and see whether there is any appetite for producing an RFC (possibly supported by TDC contributions).

I would strongly suggest supporting all of whatever expression language we choose, as supporting a subset hobbles the ability of tooling to use existing libraries. (This is another reason not to use JsonPath.)

[shijinkui]

hi, all
The mutiple language is strongly need in international product. I recommand use the i18n property file optitionly which can be easy translated into other language by linguister without changing API defination file.
Grammer like：$ref: '@abc/en.properties/key_pets_post_desc'.
At same time support to use description text directly.

By the way, i18n is so import, we shouldn't hesitate so long.

Bests, JK

openapi: "3.0.0"
info:
  version: 1.0.0
  title: Swagger Petstore
  description: A sample API that uses a petstore as an example to demonstrate features in the
servers:
  - url: http://petstore.swagger.io/api
paths:
  /pets:
    get:
      description: Returns all pets from the system that the user has access to
    post: 
      description:  $ref: '@abc/en.properties/key_pets_post_desc'

[MikeRalphson]

@shijinkui Your example isn't valid YAML syntax:

Nested mappings are not allowed in compact mappings at line 13, column 21:
      description:  $ref: '@abc/en.properties/key_pets_post_desc'
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^…

Even if we changed all summary and description strings to be strings or objects containing a $ref object, how does referencing a single external resource containing a string help with i18n? The overlay solution proposed in this issue does actually address this (i.e. having one overlay per target language).

[msche]

Dear community, I started development on tooling to apply overlays to JSON and YAML files. I based it on the specs of the openapi overlays. A very early alpha version can be found at GitHub > JayOverlay. Currently it has a gradle plugin to apply the overlays but working on CLI and maven plugin.

[handrews]

@darrelmiller @lornajane want to transfer this issue to the Overlays repo? I don't have permissions there.

[lornajane]

Thanks! Although I'm not sure what to do with these issues that have been open a while and where things like Overlays have moved on. I'll close this one as I feel that we have taken the action that it needs.