"$schema" guidance for OAS 3.1

[handrews]

#1977 includes support for $schema along with all of the other JSON Schema keywords. This is really important for being able to experiment with things like new code gen vocabularies or using Hyper-Schema and OAS together, but we'll want to put some boundaries on it. We don't want people thinking they can just shove anything in there and expect tooling to support it.

I opened PR #2016 primarily to have something concrete to reference for this.

TL;DR:

• The default meta-schema for OAS Schema objects should describe the JSON Schema Core and Validation keywords, plus OAS's extensions and compatibility adjustments (e.g. exclusive*).
• Will OAS require that all meta-schemas forbid unknown keywords that do not begin with x-? Requiring this would mean that the standard JSON Schema meta-schemas cannot be used, even though they are compatible, which would seem to defeat the purpose of JSON Schema compatibility
• Will OAS allow using additional optional vocabularies (meaning that tooling that doesn't care about them can safely ignore them) that use keywords without x- prefixes? Examples would be hyper-schema and some future hypothetical code generation vocabulary that might get designed outside of OAS.

Default $schema value

I used https://spec.openapis.org/oas/3.1/meta/validation/2019-10 as a placeholder URI for what I think the default Schema Object meta-schema should be. This meta-schema declares that all of the JSON Schema Core and Validation semantics are present, with the same level of requirement as the standard https://json-schema.org/draft/2019-09/schema meta-schema (everything except the Format vocabulary is required, while support for format is somewhat best effort- as far as I can tell, that matches how OpenAPI manages format).

The meta-schema also requires support for OpenAPI extension keywords, and compatibility syntax for exclusiveMinimum and exclusiveMaximum. It uses a somewhat tweaked version of the standard validation meta-schema plus a meta-schema for the extensions in order for this to work. It also enforces that the only extension keywords allowed must start with x-.

OpenAPI should define this as its default for $schema (as a special-purpose implementation, it can determine its own default assumptions.

Possible alternate values

The standard core/validation meta-schema

To facilitate interoperability with regular JSON Schema implementations, we should allow the standard https://json-schema.org/draft/2019-09/schema meta-schema. As it declares strictly a subset of the keyword semantics and syntax declared by the OAS Schema object meta-schema, this should work just fine.

The only exception to compatibility is that it does not forbid unknown keywords that start with something other than x-. I think it is fine for us to document that just because a meta-schema does not enforce that restriction, that does not mean that OpenAPI implementations must suddenly support non-x--prefixed extension keywords.

The standard hyper-schema meta-schema

If we want to encourage experimentation with Hyper-Schema, we should also allow https://json-schema.org/draft/2019-09/hyper-schema. Of course, that introduces two additional non-x--prefixed keywords, links and base. This is where things get interesting.

If we want to have a strict adherence to the x- policy, then that would mean that in order to use Hyper-Schema, its vocabulary should be added to the default Schema Object meta-schema (probably with value false indicating that tools can decide to process the schema even if they do not understand Hyper-Schema- Hyper-Schema is safe for tools that don't care about it to ignore).

OR, we could say that only keywords that are defined outside of the JSON Schema vocabulary concept need to be prefixed with x-. So any keyword with semantics defined by a vocabulary referenced in the meta-schema's $vocabulary object would be allowable. Only keywords that do not have formally defined semantics need an x- prefix. I assume the x- prefix was designed to avoid future name clashes. The $vocabulary system reduces the need for that somewhat, I think.

Future code-gen or other vocabularies

The main reason I wanted to bring up this idea of allowing keywords from defined vocabularies to not be prefixed with x- is that one of the goals of updating to 2019-09 is to allow using the vocabulary concept to facilitate use cases like code generation.

This is something for which there is substantial demand both inside and outside of the OpenAPI ecosystem. If a new vocabulary is developed for this outside of OpenAPI, then it will not have x--prefixed keywords. It would seem to me to be important to allow tools to add support for such a vocabulary without forcing the existence of a parallel vocabulary with all of the keywords renamed with x- prefixes.

Vocabulary-based compatibility

One of the key reasons for separating vocabularies and meta-schemas is that new/unknown meta-schemas are expected to be far more common than new/unknown vocabularies.

So, defining what is allowable in $schema based on the vocabulary set is the most flexible option. In this approach, any meta-schema declaring a subset of the vocabularies in https://spec.openapis.org/oas/3.1/meta/validation/2019-10 would be allowed (which would include the standard core/validation meta-schema).

The question would again be whether additional vocabularies that do not contradict those declared in https://spec.openapis.org/oas/3.1/meta/validation/2019-10 would also be allowed, even though their keywords do not start with x- (for example, Hyper-Schema, or a code gen vocabulary). A good option might be to say that they are, but only if they are declared with false (meaning that tooling can safely ignore them if it is not designed to work with them).

This exact scenario is why the true/false vocabulary declarations exist: we did not want to force all tooling to understand irrelevant vocabularies that do not impact validation outcomes.

[handrews]

(added a TL;DR up top for those who got the long initial email and decided not to read it 😄 )

[philsturgeon]

For these two:

Will OAS require that all meta-schemas forbid unknown keywords that do not begin with x-? Requiring this would mean that the standard JSON Schema meta-schemas cannot be used, even though they are compatible, which would seem to defeat the purpose of JSON Schema compatibility
Will OAS allow using additional optional vocabularies (meaning that tooling that doesn't care about them can safely ignore them) that use keywords without x- prefixes? Examples would be hyper-schema and some future hypothetical code generation vocabulary that might get designed outside of OAS.

Yes to both, and will be part of #1977. I've taken a swing at the wording but I'm sure it'll take some polishing.

[handrews]

@philsturgeon the text at https://github.com/OAI/OpenAPI-Specification/pull/1977/files#diff-b83206d953b4be5d312b068f82ba0217R2324 indicates "no" to the first and "yes" to the second, which I think is actually what we want.

In any event, we should resolve this as a follow-up to 1977 and not burden it with further subtleties and wording debates.

[philsturgeon]

@handrews I believe we've got the answers we needed here, what is the next move?

[handrews]

@handrews I believe we've got the answers we needed here, what is the next move?

@philsturgeon no, we haven't had this answered. You can't answer "yes to both" because they contradict each other (I could have phrased that question better, sorry). PR 1977 at last check implied no/yes, which I think is correct but needs confirmation from @OAI/tsc

[philsturgeon]

@handrews the answer I was referring to was not my earlier comment, but the PR 1977 implying no/yes. So... yeah who is the best person to ask? If we ask a specific person like... @darrelmiller then better chance of an answer.

[handrews]

@philsturgeon I'm still confused. I assume you are referring to this part of PR 1977, which does say that everything is allowed, but my recollection was that we deferred a final decision on that and needed TSC consideration.

[Edit: Was not at my best when posting this because (gestures vaguely at pandemic), cleaned up the tone a bit- my apologies.]

[darrelmiller]

I do not recall having this conversation. I do remember reading about the issue and thinking that getting an answer is going to hurt my brain.

I will review the material and thing more about this. I'm going to /cc @OAI/tsc and ask for folks to form opinions.

I do not believe we want to allow arbitrary keywords and just ask tooling to ignore them. Should we allow $schema to introduce additional keywords. That I will have a better answer for once I have read Henry's essay.

[handrews]

@darrelmiller note that we're considering treating unknown keywords as annotations per json-schema-org/json-schema-spec#698

If/when we have a machine-readable indication of what keywords are part of a vocabulary, then we could restrict that behavior to unknown vocabularies. But right now we can't tell which keywords are part of an unknown vocabulary and which are just unknown entirely.

[philsturgeon]

Aha! We have a plan.

• We should add text to identify the schema as using the OpenAPI vocabulary with URI as the Id. The Id could be, https://spec.openapis.org/oas/3.1/schema-object This conversation can continue in the spec channel.
• Schema objects should be assumed to identify as using the $schema of the id defined in the last bullet point.
• For external tools to process the JSON Schema as using an OpenAPI vocabulary users would need to include this id in the schema.

Originally posted by @darrelmiller in #2188 (comment)

I'll take charge of writing up some words for this, pending a chat on OpenAPI Slack with @Relequestual once he gets an invite.

[handrews]

@philsturgeon @Relequestual @darrelmiller

We should add text to identify the schema as using the OpenAPI vocabulary with URI as the Id. The Id could be, https://spec.openapis.org/oas/3.1/schema-object This conversation can continue in the spec channel.

As a reminder, an example of both the dialect and vocabulary meta-schemas and possible $id values for them has been posted as PR #2016 for quite some time. While the exact $id values there are not important, please make sure that the wording here is broadly compatible with that arrangement. Or comment on that PR about what needs to change and why.

Schema objects should be assumed to identify as using the $schema of the id defined in the last bullet point

meta-schemas are set on a per-resource basis, and the main OAS file is a resource. So schema objects in the OAS file are part of that resource, and that resource can (and should) define the default meta-schema like this.

For external tools to process the JSON Schema as using an OpenAPI vocabulary users would need to include this id in the schema.

Yup, this is correct. To make certain we're all on the same page: Schema objects that are separate files, or are inline schema objects or subschemas with their own $id, are separate resources. If they are separate files, then the default $schema value is determined by the JSON Schema Core / Validation specs. Which means that it is https://json-schema.org/draft/2019-09/schema.

If they are inline schema objects or subschemas that have an $id but not a $schema, the $schema value is inherited from the enclosing resource (the OAS document or standalone schema file).

[philsturgeon]

Had a go at this in #2266.