Authority/specification governing $ref in Schema Object

[watuwo]

Which specification is responsible for describing the processing of $ref in Schema Objects?

• OpenAPI 3.1 (ultimate authority)
• JSON Schema 2020-12 (any authority?)

If you believe that $ref in Schema Objects is processed according to the JSON Schema specification, could you tell me why examples like this are guaranteed to work:

"$ref": "#/components/schemas/Address"

Note that I can see how I can make such a reference work. The question is which rules mandate and guarantee the behavior assumed/intended by the author of this example (and many similar examples). If an implementation did not support this behavior, which rules would it break? How would we argue in a hypothetical court case where good will and common sense are weak arguments?

If you believe that $ref in Schema Objects is not processed according to the JSON Schema specification, where is the JSON Schema behavior overruled (why would $ref not be subject to: "Unless stated otherwise, the property definitions follow those of JSON Schema and do not add any additional semantics.")? Also, if you believe that $ref is not processed according to the JSON Schema specification, please explain this examples from the OpenAPI specification (notice type as sibling of $ref):

items:
  type: object
  $ref: '#/components/schemas/Address'

[flaksp]

There are no violations of JSON Schema in these examples.

could you tell me why examples like this are guaranteed to work

I can't understand what problem do you see in spec examples. Are you talking about the Address schema definition is missing in those examples? Imagine it's defined there because spec authors omitted #/components section in YAML to make examples simpler and shorter. These examples are only slices of the entire imaginary OpenAPI document.

notice type as sibling of $ref

It's valid. $ref may be used with any other keywords. Schema instance should pass validation if it's valid against all keywords in a schema. So it will validate an instance against type keyword first, then validate it against $ref. Any other JSON Schema keyword may be added to this schema.

[watuwo]

Are you talking about the Address schema definition is missing in those examples?

Creative guess but this is not the problem that I am having. I could have formulated this part of the question differently. If you believe the $ref usage is specified by the JSON Schema specification:

• Why is the intended target schema identified by / associated to the URI someBase.rfc3986Resolve("#/components/schemas/Address")? components is not $defs and components is not addressed in the JSON Schema specification. Also note this comment from the JSON Schema 2020-12 core specification:

  Note that documents that embed schemas in another format will not have a root schema resource in this sense. Exactly how such usages fit with the JSON Schema document and resource concepts will be clarified in a future draft.

  Source: https://json-schema.org/draft/2020-12/json-schema-core.html#section-4.3.5-2

• What is the process leading from the string "#/components/schemas/Address" to the intended target schema? Is an application of a JSON pointer involved? Is some concept of URI/IRI-equivalence involved?

If I just want to use $refs in that way ("#/components/schemas/Address"), I can come up with answers to these questions myself.

I want to know what the specifications demand and guarantee. I am looking for answers to these questions that can be found in a specification. I want to know whether/where the intended behavior is formally/explicitly specified. I want to know whether the responsibilities are properly distributed between OpenAPI and JSON Schema. This is a question about "regulation" (I am not looking for a tutorial).

If I did not want to support this kind of $ref ("#/components/schemas/Address"), what rules would such a decision break?

It's valid.

I do not claim that any of the examples are invalid or bad. I included the example containing type alongside $ref because it indicates that $refs inside Schema Objects might not be meant to be processed as in other parts of OpenAPI documents.

Note also that AsyncAPI explicitly does not process $refs according to the JSON Schema specification:

It is appropriate to clarify that the $ref keyword MUST follow the behavior described by Reference Object instead of the one in JSON Schema definition.

Source: https://www.asyncapi.com/docs/reference/specification/v2.6.0

So for AsyncAPI everything is clear. I wish it were that clear for OpenAPI as well.

[jdesrosiers]

components is not $defs and components is not addressed in the JSON Schema specification.

In JSON Schema, a reference can point to anywhere [1]. It isn't limited to $defs or to JSON Schema keywords. Therefore, it doesn't matter that components isn't defined in JSON Schema. Because the document referenced is an OpenAPI document, it's OpenAPI's responsibility to define how fragments are resolved [2]. As long as the resolved value is a JSON Schema, JSON Schema can handle it.

[watuwo]

In JSON Schema, a reference can point to anywhere

This is strong.

• The 2020-12 version of JSON Schema is not stable (in the sense of forward compatible). Control over the dereferencing of $ref IRIs can be reclaimed (by the JSON Schema specification team). If it is the goal to eventually define a stable (forward compatible) version of JSON Schema, reclaiming control over the dereferencing of $ref might be more problematic. Allowing implementations / users / higher levels to freely associate IRIs with schemas can be a strong commitment.
• The dereferencing of $refs is one aspect of schema identification. The schema "resource" concept is another one. It would seem this influences whether or not a $dynamicAnchor is found in a dynamic scope during schema evaluation. In addition to the forward compatibility challenge, there is this TODO comment in the JSON Schema specification about how "the JSON Schema document and resource concepts will be clarified in a future draft".
• Is there a consensus an "anywhere"?

Because the document referenced is an OpenAPI document [...]

So we have to know what the reference points to in order to determine what the reference points to? The same way JSON schemas are embedded in OpenAPI, the OpenAPI tree could also be embedded in something else (and so on). I would say the system of nested embeddings works because of the good will and "reasonable" behavior of all participants rather then an explicit specification. This would be OK for now. I guess my biggest problem is that it feels to me (totally subjectively) as if people wanted to hide the vaguenesses and edge cases (and I believe there are others in JSON Schema, e.g. related to regular expressions, floating point numbers, IRI-percent-encoding, ultimate base URI, the role of "configuration", ...).

Thank you for saying it so clearly: "In JSON Schema, a reference can point to anywhere". Maybe I should stop worrying and start thinking about it like this as well.

[jdesrosiers]

So we have to know what the reference points to in order to determine what the reference points to?

JSON Schema works the same way as the Web. In a web browser, you don't have any guarantee what you're going to get when you follow a link. The browser makes a request for the resource identified by the URI and the response tells the browser what it is via the Content-Type header. The browser then decides if the response is something it understands and proceeds accordingly.

JSON Schema references work the same way. You retrieve the resource the URI identifies and the implementation needs to decide if that's something it understands and can work with. So, in order for a JSON Schema implementation to work with OpenAPI, it needs to understand the OpenAPI media type, specifically, where schemas are located within the document. OpenAPI needs to specify that and a JSON Schema implementation that wants to support OpenAPI documents needs to build that support into their implementation. It's like if a new image format is introduced, then browsers need to be updated to support that image format.

Allowing implementations / users / higher levels to freely associate IRIs with schemas can be a strong commitment.

Just like the Web, allowing users to freely associate URIs with resources (in this case schemas) is feature, not a bug.

I guess my biggest problem is that it feels to me (totally subjectively) as if people wanted to hide the vaguenesses and edge cases

I don't see what you think is vague or an edge case in this situation.

[watuwo]

You have shown new views and perspectives that I would not have considered otherwise. I thank you for that.

[hkosova]

Which specification is responsible for describing the processing of $ref in Schema Objects?

• OpenAPI 3.1 (ultimate authority)
• JSON Schema 2020-12 (any authority?)

(This answer is specifically about OAS 3.1. Earlier versions are slightly different in this regard.)

There are two types of $refs:

1. Within schemas (aka Schema Objects), the $refs are processed according to the JSON Schema rules. For example, given a schema definition based JSON Schema 2020-12 (the default version used by OAS 3.1), this is the relevant section of the JSON Schema spec.

2. Everywhere else (e.g. $refs to parameter/response/requestBody definitions) the $refs are the so-called "Reference Objects" and are processed according to this section of the OpenAPI 3.1 Spec.

[watuwo]

This is very interesting.

In the JSON Schema specification, right after the paragraph you quoted (thank you very much for that!) there is a sentence I don't think I fully understand:

Unless the "$id" keyword described in an earlier section is present in the root schema, this base URI SHOULD be considered the canonical URI of the schema document's root schema resource.

What does it mean? Is this applied here in the $ref: '#/components/schemas/Address' example? Or why else would we be able to refer to a schema by its "initial base URI"?

[handrews]

The "initial" base URI is simply the base URI as determined by RFC 3986 §5.1.2-5.1.4, prior to taking into account any base URI set within the document by "$id" in accordance with §5.1.1. As shown in the diagram in RFC 3986 §5.1, base URIs set within the document take precedence. However, if the value of "$id" is itself relative, then you need to resolve it against a base URI determined by one of the mechanisms in §5.1.2-5.1.4. This base URI is "initial" in the sense that once you resolve the relative "$id" against it, you will get the proper "final" base URI of the schema.

However, "$id" is not required, and if it is absent, that "initial" base URI is used as the "final" base URI, which is also the only URI available for the document overall. This makes the word "initial" a bit confusing, and IIRC that wording might get clarified. But again, this is no different than in HTML: if an HTML document does not identify itself, then the retrieval URI (used as the base URI per RFC 3986 §5.1.3) is considered the HTML document's URI.

[watuwo]

If I can read...

However, "$id" is not required, and if it is absent, that "initial" base URI is used as the "final" base URI

... as ...

However, "$id" is not required, and if it is absent, that "initial" base URI is used as the "final" base URI [and an identifier that can be used with $ref]

... that would be the missing link. It is not the only option, e.g. the "freedom of URI-schema-association feature" could have been invoked.

IIRC that wording might get clarified

Nice... looking forward to this. I guess the same goes for "canonical"/"document"/"resource" (I am saying this because I suspect that this identifier obtained through the initial base URI is not supposed to "shield" "$dynamicAnchor"s the way "$id" does but I might be wrong about that).

I guess the HTML example is an analogy/metaphor that does not have to be 100% accurate in every respect. Just for fun: I would not expect to be able retrieve an HTML document through the URI used in a <base...> tag.

It feels like issues starting here are related (but I am absolutely not familiar with the internals of jschon and might be completely wrong about this).

Another impression: The answers I get are more diverse than I would have expected.

In the future the topic might resurface .... maybe under a different heading ... maybe one of those:

• How would we feel about {"properties": {"conditionallyAllowedProperty": {"$ref": "any://boolean/anywhere#not/originally/intended/for/use/as/schema"}}}?

• Should we promote the "$schema": ..., "$id": ... prologue on schema branch roots or does this feel too much like XML ceremony? Should there be a "default" mechanism for $id too?

• After the discussion "What is a (un-)known keyword and how do we recognize it?" there could also be the discussion "What is a JSON schema and how do we recognize it?".

• What is the progress on that:

  Exactly how such usages fit with the JSON Schema document and resource concepts will be clarified in a future draft.

• What kind of guidance should be offered to implementers and authors of other specifications on the topic of "discovery, identification, storage, retrieval and publication of JSON schemas"?

• ...

[handrews]

... that would be the missing link. It is not the only option, e.g. the "freedom of URI-schema-association feature" could have been invoked.

Yes, I only left that off because $id always associates a URI with the schema object that contains it. Many people find the base URI behavior confusing but the URI association is straightforward.

All of your other questions should be raised at the JSON Schema repo, as they are not at all specific to OpenAPI.

[jdesrosiers]

I guess the HTML example is an analogy/metaphor that does not have to be 100% accurate in every respect. Just for fun: I would not expect to be able retrieve an HTML document through the URI used in a <base...> tag.

There isn't really a good analogy for $id in HTML (or the web in general, which I think makes it awkward in JSON Schema). <base> is close, but not quite because $id identifies the schema rather than just effecting the base URI like <base>. $id is awkward. I'd rather see it gone (or replaced with a true <base> analog), but it's probably something we'll be stuck with for historical reasons.

However, when $id is used in a sub-schema, I think it's a pretty good analogy for <iframe>. It's the identifier for a schema embedded in another schema while being distinct from the parent.

[handrews]

I've marked the correct answer to the original question. The rest of the discussion is more about JSON Schema than OpenAPI, so I'm going to close this as resolved (as far as OpenAPI is concerned).