OAS schema dialect clarifications

[MikeRalphson]

This PR covers a few outstanding issues from the 3.1-Final milestone:

• Clarify the relationship between OpenAPI 3.1 and the JSON Schema version #2363 mechanism for use of different JSON Schema drafts
• Create an OpenAPI JSON Schema vocabulary #2364 add the default $schema id before our extension vocabulary
• Provide guidance on how to override the implicit OpenAPI vocabulary #2365 overriding the $schema dialect
• Where is the schema instance for 3.1? #2371 change the previous suggested $schema value to avoid confusion with schema instances

It also changes a few JSON Schema Draft specification links either to use the https scheme or to make them more specific.

The mechanism for setting a default value for $schema for schemaObjects within the OAS document could be modelled in several different ways. And while not wanting to over-engineer anything, or provide extension points which may never be needed, it felt right to use a containing defaults object with a schema defaults object within it. This does allow us to have a place to put other things like global default parameters, responses in the future, which have been asked for repeatedly, and over a long period.

Alternatively to having one $schema field which is REQUIRED in that schema defaults object, we could possibly allow other JSON Schema core vocabulary keywords using patterned fields (^\$.*), but I could not make a definite case for this. I don't think it's a good idea to make the schema defaults object a schemaObject itself as the JSON Schema folk have taught us that merging schemas is hard.

In reviews, I would especially like a check that my terminology (especially around $schema, "schema resoources", "dialects", "meta-schemas" and "vocabularies") is correct and clear. @Relequestual could you also weigh in? Wording / sentence ordering changes also welcome of course.

[philsturgeon]

I think "schema resoources" is the Canadian spelling, right @darrelmiller?

The defaults object feels a little concerning, but if its just for setting default $schema that might not be so bad. Can we discuss this on the call today?

[MikeRalphson]

I think "schema resoources" is the Canadian spelling, right @darrelmiller?

😆 At least it's only in the notes and not in the diff!

The defaults object feels a little concerning, but if its just for setting default $schema that might not be so bad. Can we discuss this on the call today?

Yep, I was in several minds how to achieve the default for $schema. It's on the agenda, so hopefully wise people will have thoughts.

[handrews]

I'm very concerned about a default schema object, as it changes the meaning of schemas in a very non-obvious way. There are exactly two schema keywords for which it is valid to externally specify default behavior, which is different from setting default values.

The initial base URI, normally set by the top-level $id, can be determined in the absence of such an $id by external means per RFC 3986. We have specifically updated JSON Schema in the next draft to state that when JSON Schema is embedded in another media type, then in the absence of $id it takes on the base URI set by the enclosing document, according to that media type's rules. Which of course, ultimately default to the document's retrieval URI if the document does not set it explicitly.

The meta-schema, normally set by $schema, is the other keyword where implementations must make an assumption if it is not explicitly present. Here is what the forthcoming draft says about it:

                     The "$schema" keyword SHOULD be used in the document root schema object,
                     and MAY be used in the root schema objects of embedded schema resources.
                     It MUST NOT appear in non-resource root schema objects.  If absent from
                     the document root schema, the resulting behavior is implementation-defined.

                     If multiple schema resources are present in a single document, then
                     schema resources which do not have a "$schema" keyword in their root
                     schema object MUST be processed as if "$schema" were present with the
                     same value as for the immediately enclosing resource.

The "implementation" here can include whatever implementation of an OAS processor is being used to read the OAS document and hand off the JSON Schema processing to a JSON Schema implementation (which might be embedded in the OAS implementation or might be separate). Therefore, if OAS has a special rule for determining that value, like "check the schemaDialect field of the defaults object", then by the time it logically reaches the JSON Schema implementation it knows what to tell that implementation to use in the absence of $schema.

Many if not most implementations today actually require you to set this externally anyway even if you do have $schema present. For implementations that do not have such a capability and actually rely on $schema, yes you can inject a $schema into the JSON structure before handing it off. But conceptually, that is a workaround and not a real defaulting behavior. The distinction is important.

Having a schema object in a defaults produces an implicit non-local change in behavior, that I assume applies recursively to nested schemas? So now everything has to be processed by some sort of recursive schema walker in order to produce something that could be handed off to a JSON Schema implementation, which means that the schema you get is not the schema it looks like you have. Which is true even if the defaults are not set recursively.

Please don't do this. Please consider specifying the default meta-schema as its own thing, not as a default schema object. Applications are allowed to externally determine the processing meta-schema specifically because $schema, like $id, is needed to bootstrap that processing. That should not be taken as a reason to offer broader external defaulting.

We can clarify the language in the JSON Schema spec by adding something like:

                     If a schema object is embedded in a document of another media type, then
                     that document MAY specify the meta-schema by which embedded schema
                     objects that lack a "$schema" in the root object are to be processed.

This would parallel the language currently used with the initial base URI and $id, and avoid the problems that would arise with non-local defaulting.

[MikeRalphson]

@handrews the schema object within the defaults object is not a schemaObject type. It is a strictly limited object which only has one required field: $schema, which is a default value to use in place of the implicit OAS 3.1 schemaObject dialect id.

Does this assuage any of your concern, or am I misunderstanding?

I explicitly said above

I don't think it's a good idea to make the schema defaults object a schemaObject itself as the JSON Schema folk have taught us that merging schemas is hard.

[MikeRalphson]

If a schema object is embedded in a document of another media type, then that document MAY specify the meta-schema by which embedded schema objects that lack a "$schema" in the root object are to be processed.

If I understand this correctly, this is exactly what this PR aims to enable.

[handrews]

@MikeRalphson

I explicitly said above

I don't think it's a good idea to make the schema defaults object a schemaObject itself as the JSON Schema folk have taught us that merging schemas is hard.

I saw this at 6AM unable to go back to sleep and saw that it was on the agenda for this morning's meeting so I replied. I searched for "merge" trying to find this but didn't think to search for "merg" or "merging."

Does this assuage any of your concern, or am I misunderstanding?

Yeah, it works. I think it's a bad idea to make something that looks too much like a schema object, but isn't. I'd just have a metaSchema field or something.

But I'm not going to fight over it, I only answered at all because you specifically assigned me as a reviewer. So, yes it should work. It's my opinion that it will cause problems because people will treat a field called "schema" in a section called "defaults" as a default schema object, but if y'all want to deal with that confusion that's fine.

[MikeRalphson]

Henry makes very good points, and they were design decisions I did hum and hah about. Happy to rework to avoid the schema vs schemaObject confusion.

[Relequestual]

Left a few comments.

I may not see any follow up comments. I tend to rely on slack notifying me of github activity for specific projects... or rather just JSON Schema.
You know where to find me =]

[darrelmiller]

Proposed @OAI/tsc change:

jsonSchemaDialect: http://json-schema.org/draft-2019/schema#

[karenetheridge]

jsonSchemaDialect: http://json-schema.org/draft-2019/schema#

with no trailing '#', please. The most recent drafts are stricter about where empty fragments are allowed/preferred.

[darrelmiller]

I don't know why I added that trailing #. Wasn't intentional!

[handrews]

Something that just occurred to me due to the ongoing discussion about discriminator, particularly if the selectBy idea is adopted: It might be better to put discriminator in its own vocabulary since it is complex to implement and (particularly if selectBy were added) not strictly necessary.

Putting it in its own vocabulary allows implementations to choose whether to implement it separately from the other OAS extension keywords, and could allow for dialect configurations that require the other OAS extension keywords but leave discriminator as optional. idk if that's really a great idea or not but thought I would mention it. We moved unevaluatedProperties and unevaluatedItems into their own vocabulary for similar reasons- they are complex and potentially expensive so some use cases might prefer not to deal with them (streaming implementations, for example, are much more costly with these).

[darrelmiller]

We believe this issue is one that requires further conversation but don't believe it should hold up merging the current PR that introduces the dialect property.