Fix description vs document terminology

[mikekistler]

This PR attempts to fully distinguish "description" and "document" by using "description" consistently for the concept of an API description in the OpenAPI format and "document" to only refer to structural features.

I also revised the "OpenAPI Description Structure" section a bit in part to address the above and also in an attempt to simplify and clarify that section.

Some specifics:

• Every instance of "OpenAPI document" and "OpenAPI description document" has been eliminated. Most are changed to "OpenAPI Description".
• All occurrences of "OpenAPI Description" capitalize the "D" in "Description".
• I changed some instances of "OpenAPI Description" to "OAD", but there are still 33 occurrences of "OpenAPI Description" and at least some of these might be good candidates to change to "OAD".
• the previous text defined "self-contained OpenAPI document" -- I changed that "single-document OpenAPI Description" which I think is clearer and simpler.
• the previous text defined "syntactically complete OpenAPI document" -- I removed that as I believe that other wording changes I made made it unnecessary.

I think there are some good changes here but I'm definitely open to feedback / suggestions on how make it even better.

[ralfhandl]

In general ok, minor nits.

[ralfhandl]

This now breaks the "As ..." pattern, but I have no suggestion for avoiding this.

[handrews]

@ralfhandl

• As the root object of the entry document, which is always interpreted as an OpenAPI Object

[handrews]

@mikekistler I really appreciate your work here- I am surprised at how many things it has flushed out, and however we resolve those differing views the spec will be much stronger!

I feel a bit bad that I did not advise you to start with 3.1.1, which is what I did for every document parsing and referencing change (all of the others started in 3.0.4). 3.1 is much more complex, so I worked out what made sense there, and then backported the results to 3.0.

This is because 3.0 is, as I think @darrelmiller described, the "uncanny valley" between the 2.0 paradigm (the OAD should function as a single JSON/YAML document even if it is not) and the 3.1 paradigm (it's not, in general, possible to correctly parse only part of a document, and shared documents are expected to be "syntactically complete" but just have components and not paths or webhooks).

Most of the "this needs to be 'document'" stuff is related to supporting components-only "syntactically complete" documents in 3.1. We then want the wording to be as consistent as possible between 3.0 and 3.1, without accidentally imposing 3.1 requirements on 3.0. This is tricky, and is why some of the PRs in this area got re-written multiple times.

I think I'd recommend attempting a PR on 3.1.1 before revisiting this. I think it will clarify a lot of things for you. And me — clearly there is more work to do here than I realized, and I'm really glad you are surfacing it!

[handrews]

This is about a document, not an entire OAD. If all documents within an OAD are what I called "syntactically complete", then this will be true for each of those documents. (We don't address what happens if different documents in the OAD have different minor versions but I don't think anyone could reasonably expect that to work- different patch versions are fine).

[darrelmiller]

+1 on the bit about it being about a document.

As far as the second part. Our tooling always normalizes OpenAPI documents to the latest version, so having an OpenAPI Description that consists of documents with different major versions is actually possible. Assuming the version transformation was successful on reading.

[handrews]

so having an OpenAPI Description that consists of documents with different major versions is actually possible. Assuming the version transformation was successful on reading.

But from the point of view of the tools, there is no difference- it only tries to link them up post-version-migration. So I think that approach is outside of the scope of the OAS. From the OAS point of view, you're just working with all-3.1 documents (at least right now as 3.1 is latesst).

[handrews]

This is also about the document, not the OAD. As noted in my overall comment for this review, it could be about the OAD in 2.0, but not in 3.1, and we want to emphasize 3.0 -> 3.1 continuity as much as we can without actually invalidating existing correct 3.0 tooling.

[ralfhandl]

So is this about "An OpenAPI entry document that conforms"?

Because technically referenced documents could have a top-level array that contains to-be-referenced objects, arrays, or primitive values.

Suggested change

	An OpenAPI Description that conforms to the OpenAPI Specification is itself a JSON object, which may be represented either in JSON or YAML format.
	An OpenAPI [entry document](#openapi-description-structure) that conforms to the OpenAPI Specification consists of a JSON object, which may be represented either in JSON or YAML format.

Although that is some

[lornajane]

The original text "OpenAPI Description" is correct IMO. We already did a lot of work on not using "document" when we mean "description"

[handrews]

@lornajane but we do mean "document" here, not "Description." It is not always possible to represent an entire OAD as a single JSON/YAML object,

@ralfhandl

So is this about "An OpenAPI entry document that conforms"?

it's not just about entry documents. This gets into the distinction I waas trying to make with "syntactically complete" documents.

Because technically referenced documents could have a top-level array that contains to-be-referenced objects, arrays, or primitive values.

No, such a document does not, as a document conform to the OAS. Nowhere in this specification are such documents mentioned, much less deemed compliant with the OAS syntax.

You can make a plausible argument that a document with an Object other than an OpenAPI Object at its root is "compliant" with the OAS, although the 3.0 specification never directly says so one way or the other. I'm not sure 2.0 even says that, although it strongly implies it.

So there are really three cases here:

1. Syntactically complete documents (with an OpenAPI Object at the root) — these are unquestionably in compliance with the OAS, regardless of version
2. Documents with some other Object at the root — these are never explicitly mentioned in 3.x, and the proposed OAS media types do not support them as currently written, but they definitely work in 3.0. 3.1.1 lists the conditions under which they can work in 3.1. I think of these as fragments (OAD fragments? OAS fragments? compatible with fragments of the OAS?)
3. JSON/YAML documents in arbitrary format where some subset of the document (e.g. each element in a root array) is compliant with some or all of the OAS format. This is not addressed anywhere in the spec AFAICT. It can work for 3.0. But 3.1.1 explicitly states that this sort of behavior is undefined, because you cannot correctly process only part of a 3.1 document (there's a whole section about exactly this in 3.1.1)

[darrelmiller]

Personally I have never made a distinction between 2 and 3 before. I have referred to the chunks of these documents that do conform to some OAS object as OpenAPI fragments.
So, in my head there are documents that are OpenAPI Documents and documents that are JSON/YAML documents that can contain OpenAPI fragments.

[handrews]

or composed from elements distributed across multiple documents at the discretion of the author.

I'm not a fan of this wording as it suggests the 2.0 approach of references just being a way to distribute a structure that could be a single document. That is not the paradigm in 3.x.

I don't think the prior wording was great either. I need to think on what might be better than either.

I have a similar concern with:

are used to identify the documents containing the referenced elements.

This suggests the paradigm of extracting random bits of JSON or YAML from a document without parsing the rest of the document, which can't be done in 3.1 (and is dubious in 3.0, really- the way this can go "wrong" is pretty obscure but I an explain it if needed).

(we should also add the Discriminator Object's mapping field to the list of fields that can bring in another document- it can do this with its URI-reference syntax, just like operationRef)

[ralfhandl]

Compromise, because I like the addition to the last sentence:

Suggested change

An OpenAPI Description (OAD) MAY be structured as a single JSON or YAML document or composed from elements distributed across multiple documents at the discretion of the author. In the latter case, [Reference Object](#reference-object) and [Path Item Object](#path-item-object) `$ref` keywords, as well as the [Link Object](#link-object) `operationRef` keyword, are used to identify the documents containing the referenced elements.

An OpenAPI Description (OAD) MAY be made up of a single JSON or YAML document or be divided into multiple, connected parts at the discretion of the author. In the latter case, [Reference Object](#reference-object) and [Path Item Object](#path-item-object) `$ref` keywords, as well as the [Link Object](#link-object) `operationRef` keyword, are used to identify the documents containing the referenced elements.

[handrews]

@ralfhandl I still dislike the implications of that phrasing, as I expanded on in more detail in my last reply to another thread in this PR.

[handrews]

@ralfhandl

• As the root object of the entry document, which is always interpreted as an OpenAPI Object

[handrews]

This is another one where it's about the document. "parent Object within the OAD" is ambiguous- do you mean the literal parent in the JSON/YAML structure or the logical parent which might be a reference depending on how you reached it? This section was intended to clarify that distinction and the problems it can cause.

Here we are talking about the JSON/YAML structure parent.

While the presence of the next bullet point implies that this one has to be about something other than referencing, I would prefer to emphasize the document-oriented nature of the parent/child context.

[handrews]

@mikekistler this is my favorite change so far because I don't actually know which I prefer!

Your version is definitely correct, as any OAD MUST be rooted in a specific OpenAPI Object. But it is also true that the OpenAPI Object is the root object of any "syntactically complete" document, and that's important. Particularly in 3.1.

[handrews]

I think this is actually document-specific, which also gets into what I mentioned before about mixing documents with different openapi versions (OK with patch, not OK across minor, although we don't explain this anywhere).

See the next comment about the Info Object's version field.

[ralfhandl]

We need to replace "OpenAPI document" because we nowhere define this term.

[handrews]

Not anymore, but I had been using it to make the OAD vs document distinctions I've raised throughout this PR- admittedly, the distinction was not clear the way I wrote it. But we could easily make our usage of "document" clear along with our usage of OAD. There are many things that are document-specific, and not scoped to the whole OAD.

[handrews]

I think this might also be document. It doesn't matter for Server Objects under the OpenAPI Object's servers field, and if there are no such Server Objects, then we're basically "pretending" that there is one with a url based on the entry document's location.

But this gets weird when you have a Path Item Object in a referenced document that defines its own servers, or that has Operation Objects that define their own servers (or a Link Object that defines server). I think those ought to be relevant to the location of the document in which they appear.

In "Resolving Implicit Connections", I hand-waved the Servers problem on the grounds that only the entry document's Paths Object is relevant. But with a shared Path Item Object under the Components Object of a components-only "syntactically complete" document, do you look up the default by current document or by entry document? That might actually need clarification.

[miqui]

In "Resolving Implicit Connections", I hand-waved the Servers problem on the grounds that only the entry document's Paths Object is relevant. But with a shared Path Item Object under the Components Object of a components-only "syntactically complete" document, do you look up the default by current document or by entry document? That might actually need clarification.

It would seem logical to me then the lookup happens on the entry document.

[handrews]

I wanted to make a point that these are resolved like this in the entry document even if the OAD is multi-document. I don't feel strongly about this one, but I did have a purpose with that wording.

[handrews]

We have debated whether this is the version of the OAD or the version of the document, and I thought we decided on the document.

Consider a multi-document OAD that uses a shared components-only "syntactically complete" document. That document is maintained separately from any of the entry documents that reference it, so it needs its own version.

I think the version field of the entry document is suitable for use as the overall version of the OAD, although I don't think we discussed that.

[ralfhandl]

I don't want to go to different version numbers for different parts of an OAD and think Mike's change is fine as we don't define "OpenAPI document".

[handrews]

@ralfhandl then how do you havdle the case of a shared components document? This is a valid OAS 3.0.3 multi-document OAD:

Entry document openapi.yaml:

openapi: 3.0.3
info:
  version: 1.0.0
  description: The actual API
paths:
  /foo:
    get:
      requestBody:
        $ref: "shared.yaml#/components/requestBodies/foo"
      responses:
        "200":
          description: OK

Referenced document shared.yaml

openapi: 3.0.3
info:
  version: 2.4.0
  # It's really annoying that 'version' is a metadata field but there is no 'description' that describes the OAD, only one for the API
  description: Not an API at all
paths: {}  # the Paths Object MAY be empty in 3.0
components:
  requestBodies:
    foo:
      ...

[miqui]

I think the version field of the entry document is suitable for use as the overall version of the OAD, although I don't think we discussed that.

AFAIK, we have not. I agree with this statement, but mindful of @ralfhandl comment about different versions spread across different OADs. Maybe we should include additional semantics whereby the version of the root OAD is the overall version of the OAD when you do not specify a version on a shared component OAD.

[darrelmiller]

@miqui We have discussed this and we rejected it. Consider @handrews 's example above where you have an OpenAPI Document explicitly intended to provide shared components. If the entry document's version was the OAD version, then technically if the shared component OpenApi Document version was increased then all the the OAD versions that used the shared components would need to be bumped also. This would be a a pain to manage. We chose to scope the version field to the OpenAPI Document because that is easy for a developer to manage.

[ralfhandl]

To me the comments from @miqui and @darrelmiller seem to say the same:

• The version field of the entry document is the one and only version of a multi-document OpenAPI Description.
• If non-entry documents are reached via reference from a 2.0 or 3.0 entry document, it does not matter whether they have a version field; it is never interpreted and references are resolved on the in-memory representation of a "JavaScript" model resulting from plain JSON/YAML parsing of the text documents.
• If the entry document is 3.1, things apparently are different and I leave the parsing to @handrews 😎

[mikekistler]

I think this interpretation is a valid interpretation for 3.0 OADs. I think there may be other interpretations, including ones that are more consistent with 3.1. So we should let this be "implementation defined" in 3.0 with enough explanation to allow any of these valid interpretations while ruling out any others.

[ralfhandl]

How about

Suggested change

	An OpenAPI Description compatible with OAS 3.\*.\* contains a required [`openapi`](#oas-version) field which designates the version of the OAS that it uses.
	An OpenAPI [entry document](#openapi-description-structure) compatible with OAS 3.\*.\* contains a required [`openapi`](#oas-version) field which designates the version of the OAS that it uses.

[miqui]

@ralfhandl line 92: Sounds reasonably good to me.

[handrews]

Same objection to restricting this to entry documents as in the other thread.

[ralfhandl]

Suggested change

	* The root object of the entry document is interpreted as an OpenAPI Object
	* As the root object of the [entry document](#openapi-description-structure), which is always interpreted as an OpenAPI Object

[ralfhandl]

We don't define "OpenAPI document" and have to replace it with either "OpenAPI Description" or "OpenAPI entry document", whichever is meant.

And I prefer not to define "OpenAPI document" because it is halfway between "OpenAPI Description" and "OpenAPI entry document" and a larger "Hamming distance" between two terms of the same specification reduces confusion.

[ralfhandl]

We need to replace "OpenAPI document" because we nowhere define this term.

[ralfhandl]

I don't want to go to different version numbers for different parts of an OAD and think Mike's change is fine as we don't define "OpenAPI document".

[ralfhandl]

How about

Suggested change

	For these examples, assume all schemas are in a single-document OpenAPI Description; for handling of `discriminator` in referenced documents see [Resolving Implicit Connections](#resolving-implicit-connections).
	For these examples, assume all schemas are in the [entry document](#openapi-description-structure); for handling of `discriminator` in referenced documents see [Resolving Implicit Connections](#resolving-implicit-connections).

[handrews]

That's fine, I guess it's less words than what I had :-)

[mikekistler]

Folks, I had a discussion with Darrel Miller on this topic yesterday and wanted to recap some of that here.

@darrelmiller please correct me if I captured any of this incorrectly.

At the beginning of the discussion, Darrel felt that there was significance to the term "OpenAPI Document" and was surprised that it was not defined in the spec (we were looking mostly at the 3.0 spec). Darrel proposed this definition for "OpenAPI Document":

An OpenAPI document is any document that follows the syntax and semantics of the OpenAPI Object of the OpenAPI specification.

So an OAD's entry document (as we define this term in the spec) clearly must be an "OpenAPI Document" by Darrel's definition, but other documents referenced from the entry document may also be "OpenAPI Documents" by Darrel's definition. And from a tooling perspective, Darrel explained the importance of this as follows:

If a reference goes to a fragment in an 10 MB file, that fragment might be just 10 lines, but the tooling needs to read / parse most of the file to find it. For efficiency, the tooling will probably parse the whole file and keep the results in memory so that any other references into that file are handled efficiently.

So that makes sense but then begs the question of how the referenced document is parsed. Is it parsed as a JSON document, or an OpenAPI Document, or a JSON Schema document, or ??. I think Darrel was implying that if the file had "openapi" at the root, it would be parsed as an OpenAPI document.

But now we come to this question: Suppose an entry document contains this $ref:

"$ref": "./CommonTypes.json#/components/schemas/ResponseBody"

If the "CommonTypes.json" file happens to have "openapi" at the root, does that mean that it is parsed as an OpenAPI Document? If so, what happens if the version of OpenAPI indicated in the "openapi" field doesn't match the version in the entry document? What happens if the version in the "openapi" field is bogus, e.g. "42"? What happens if parts of the document other than the referenced fragment fail to parse correctly as an OpenAPI document?

But the real issue comes when we get to 3.1 and schemas can have $id fields. In that world, a $ref can contain a URI. To support this, there is an "out of band" process for tooling to "locate" schemas which are collected into a registry keyed by their "id" so that URI's can then be resolved when parsing the OAD.

So suppose in the "out of band process" there is a set of schemas loaded from an OpenAPI 3.1 Document or JSON Schema document. Parsing of those documents should follow the URI resolution rules, which means they need to resolve relative URIs according to RFC 3986 Section 5. In particular, the Base URI for a schema may come from the "Encapsulating Entity", and relative URIs within this schema are then resolved according to this Base URI.

So far, so good. The schema is entered into the registry and any URI with its "id" gets that schema. BUT, suppose an OAD contains a $ref to this schema by id and in another place it also contains a $ref to this schema by location? Now there can be a problem, because references by location don't necessarily consider the "context" of the fragment -- meaning that the base URI in the encapsulating entity might not be considered, and relative URIs in the schema may be resolved differently than when the schema was referenced by id. If the schema contains a "$id" field, then this is broken, because the same id represents two different schemas.

Maybe everyone else understood this and it is just me that is now coming to this realization.

I'll stop writing here as this is already too long, but I hope this is helpful to moving this discussion along.

[ralfhandl]

We really should add Darrel's definition of "OpenAPI document" to the Definitions section, and then work through the PR from there.

And start in parallel on a similar PR for 3.1.1 so we can see the necessary differences simultaneously.

@mikekistler Could you please create the second PR? You seem to have dug rather deep into this topic.

[mikekistler]

One new thing to add here. I am far from being an expert on JSON Schema, particularly the latest versions, so I was studying the 2020-12 draft looking for some better understanding, and found Section 7.1 Lexical and Dynamic Scope. I think this is essential reading as it describes the problem we are wrestling with and may offer a way out. I'll snip out here some of the key pieces:

Note that some keywords, such as "$schema", apply to the lexical scope of the entire schema resource ..

Other keywords may take into account the dynamic scope that exists during the evaluation of a schema, typically together with an instance document.

Lexical and dynamic scopes align until a reference keyword is encountered ... A keyword on the far side of that reference that resolves information through the dynamic scope will consider the originating side of the reference to be their dynamic parent, rather than examining the local lexically enclosing parent.

it may be possible to revisit the same lexical scope repeatedly with different dynamic scopes.

Now, it was not clear (to me anyway) whether "$ref" is a keyword that is resolved in lexical scope vs dynamic scope. But if we consider "$ref" to be resolved in dynamic scope, then I think we can treat "$id" as identifying the "lexical scope" of the schema -- which might be resolved to different things in different dynamic scopes. So there is only one schema with this id, but it resolves to different things in different dynamic scopes.

What do folks think of this idea?

[handrews]

@mikekistler First, I'm really glad you're getting into this, and that you and @darrelmiller had a great discussion about it. Darrel has heard all of the frustrations I encountered on this during OASComply, and we compared notes about the challenges of ipmlementing 3.1 at some point.

Maybe everyone else understood this and it is just me that is now coming to this realization.

No, I've been driving myself absolutely crazy(-er 🤪 ) trying to get people to understand this for the past several years, but it's really hard to explain until you really try to follow it all through to implement it. Congratulations on being one of the few who have now dug deep enough to get it :-) (I mean that sincerely- it's a lot of obscure detail but it's really important, and I wish I'd been able to communicate it more clearly. I have more than enough data to know that it's not easy to figure out, or else more people would have by now).

An OpenAPI document is any document that follows the syntax and semantics of the OpenAPI Object of the OpenAPI specification.

So an OAD's entry document (as we define this term in the spec) clearly must be an "OpenAPI Document" by Darrel's definition, but other documents referenced from the entry document may also be "OpenAPI Documents" by Darrel's definition.

I would definitely support this, as it defines "OpenAPI Document" the same as what I was calling a "syntactically complete" document, grasping for a way to distinguish it from fragmentary documents (that are not JSON Schema documents).

If the "CommonTypes.json" file happens to have "openapi" at the root, does that mean that it is parsed as an OpenAPI Document? If so, what happens if the version of OpenAPI indicated in the "openapi" field doesn't match the version in the entry document? What happens if the version in the "openapi" field is bogus, e.g. "42"? What happens if parts of the document other than the referenced fragment fail to parse correctly as an OpenAPI document?

I hit every single one of these problems trying to write oascomply. The only reason I haven't pushed harder on getting all of them resolved is that it was hard enough to get attention on the more obvious problems, and until you (Mike) started digging into this it just seemed like people were getting burnt out on my efforts. And so was I, tbh. Some parts of this have been answered, more on that in a bit. Most of the rest could be answered by keeping things analogous to the parts that have been answered.

But the real issue comes when we get to 3.1 and schemas can have $id fields. In that world, a $ref can contain a URI. To support this, there is an "out of band" process for tooling to "locate" schemas which are collected into a registry keyed by their "id" so that URI's can then be resolved when parsing the OAD.

This is all 100% completely addressed by 3.1.1 §4.3.1 "Parsing Documents." This is actually a thoroughly understood and solved problem in JSON Schema, and you can find many implementations of it in the wild.

It does not necessarily have to be out-of-band: 3.1.1 states that if it looks like an OpenAPI Document or a JSON Schema document, you can treat it like one. It also allows for parsing a small-d document based on a $ref to its root, as that provides the type of the whole docment (let's call it an OpenAPI Fragment, for example a document consisting of a Path Item Object). 3.1.1 explicitly notes that tools can be configured to know where a URI is located- this is, again, common in JSON Schema and has been for a long time (technically since well before draft-04, but I don't think we got the explanation clear enough until draft-07-ish. It's much more explicit in 2019-09 and 2020-12 where we added a whole section on loading schemas.

So far, so good. The schema is entered into the registry and any URI with its "id" gets that schema. BUT, suppose an OAD contains a $ref to this schema by id and in another place it also contains a $ref to this schema by location?

This is also all well-understood in JSON Schema-land, and is why both JSON Schema and OAS 3.1 require full-document parsing. I'll come back to it after addressing the next few statements:

Now there can be a problem, because references by location don't necessarily consider the "context" of the fragment

This is not true. There is no difference between resolving a fragment in a document based on identity vs location, it works exactly the same way. RFC3986, in fact, only cares about identity. Treating a URI as a URL is just a handy default way to locate an identified document. Whether a given URI can reasonably be treated as a URL is up to the application involved.

meaning that the base URI in the encapsulating entity might not be considered, and relative URIs in the schema may be resolved differently than when the schema was referenced by id. If the schema contains a "$id" field, then this is broken, because the same id represents two different schemas.

It's not broken, it works exactly as intended. This whole "ignore the format of the document" is a thing I've never seen anywhere else that has no relationship to how URIs, media types, and resources are supposed to work, which is for their behavior to be keyed by media type (in the absence of a media type, something like a file extension or content sniffing can be used as a heursitic).

AFAICT, this "ignore the context" thing is completely unique to OAS. Although I mght well be wrong, of course. I'm pretty sure it is not something that comes from JSON Reference which itself does not appear to understand how URIs, media types, and fragments are supposed to work (because it mandates behavior counter to RFC9601 regarding JSON pointers as URI fragments).

---

Regarding addressing a JSON Schema document (for simplicity- OAS doesn't change things much as so far it doesn't have an $id-equivalent, but now you are starting to see why it needs one in 3.1), there are a few cases:

• Reference the document by location, and it doesn't contain any $id, so this just works as you'd think
• Ref by location, $id in the root: In this case, both the location (retrieval URL) and $id are valid URIs for the schema. It's a little unclear if the retrieval URL should be retained for future reference or discarded ,so I'd always reference by identity- the whole point is to make your $ref values stable even if the target moves around. Repeatedly ref-ing by location might lose the benefits of the cache.
• Ref by location, $id in a subschema:
  • As long as your JSON pointer doesn't cross into the $id subschema, you are fine
  • If your JSON Pointer crosses into a subschema of the subschema containing $id, the behavior is implementation-defined. Don't do this. We were forced to make this implementation-defined because some people didn't want to have to check for an error. I wanted it to be an error
  • If your JSON Pointer points to the subschema containing the $Id, this is essentially the same as the retrieval URL vs identity case- the spec doesn't nail it down, but I'd expect it to work (the spec only says that crossing the $id is implemenation-defined). But still, I'd always reference by identity.

(that's enough for this comment, I'll probably make a few follow-ups)

[handrews]

@mikekistler

Now, it was not clear (to me anyway) whether "$ref" is a keyword that is resolved in lexical scope vs dynamic scope. But if we consider "$ref" to be resolved in dynamic scope, then I think we can treat "$id" as identifying the "lexical scope" of the schema -- which might be resolved to different things in different dynamic scopes. So there is only one schema with this id, but it resolves to different things in different dynamic scopes.

What do folks think of this idea?

The way $id, $anchor, and $ref (any of the $refs) work is 100% governed by RFC 3986 and there is no wiggle room at all in that. URIs are Universal: there aren't any scopes. There are, however, base URIs, which are well-specified (although the OAS kind of munges it by skipping some of the steps- I tried to fix that by not over-specifying it in 3.0.4 and 3.1.1). The only thing a media type can really impact is whether it sets the base URI within the document contents. The other three possible base URI sources are not under the media type's control.

The whole dynamic scope thing has to do with how $dynamicAnchor and $dynamicRef work. TL;DR: $dynamicRef searches the resources (not objects or documents) in the dynamic scope for the most distant (first-parsed that is still in scope) $dynamicAnchor, and sets the base URI of the $dynamicRef value to the absolute URI of that resource. If no distant $dynamicAnchor is found, then the base URI doesn't change and $dynamicRef behaves exactly like $ref.

I've never been entirely satisfied with this mechanism. It basically inserts some steps before handing off to normal RFC3986 behavior.

The "scope" language has more to do with how certain JSON Schema keywords communicate up and down the dynamic scope, which is how unevaluatedProperties works.

[handrews]

I guess the overarching theme here is: All of the URI-based parts are well-defined in 3.1, and are now (hopefully) actually well-explained in 3.1.1. 3.0 is both easier because it is location-only, and harder because location-only isn't very practical in a lot of environments, which is why some tools already allow you to load a document from an alternate location.

[mikekistler]

There are a number of things I'm struggling with here. You say

The way $id, $anchor, and $ref (any of the $refs) work is 100% governed by RFC 3986 and there is no wiggle room at all in that. URIs are Universal: there aren't any scopes.

and later

The whole dynamic scope thing has to do with how $dynamicAnchor and $dynamicRef work.

But the 2020-12 JSON Schema spec says:

The value of the "$dynamicRef" property MUST be a string which is a URI-Reference.

So if $dynamicRef is a URI-Reference, why doesn't RFC 3986 disallow scopes for this as well?

Also, when you say "there is no wiggle room" ... I'm guessing you are referring to Section 5 of RFC 3986 describing the rules for reference resolution, and maybe more specifically to 5.1.2 Base URI from the Encapsulating Entity:

If no base URI is embedded, the base URI is defined by the representation's retrieval context. For a document that is enclosed within another entity, such as a message or archive, the retrieval context is that entity. Thus, the default base URI of a representation is the base URI of the entity in which the representation is encapsulated.

What confuses me about this language is that it seems to imply that "the enclosing entity" is unique -- that "the entity" is well defined, but isn't it possible, at least for schemas, that a schema might be nested arbitrarily deep in other schemas and as a result there are arbitrarily many possible "enclosing entities". The RFC seems not to consider/allow this.

This is all new territory for me, so I appreciate any help in clearing up my confusion on these things.

[handrews]

@mikekistler

So if $dynamicRef is a URI-Reference, why doesn't RFC 3986 disallow scopes for this as well?

I'll direct you to my statement:

've never been entirely satisfied with [the $dynamicRef] mechanism. It basically inserts some steps before handing off to normal RFC3986 behavior.

By "no wiggle room" for $ref I mean that it's just defined as a URI-reference with no special rules. With $dynamicRef, it's defined as a two-step process: Do this weird thing to find the base URI and then follow RFC3986. In retrospect, I don't think this was a good idea. I should have come up with a different syntax that more clearly separated the non-RFC3986 and RFC3986 behaviors.

So, $dynamicRef was defined as a two-step process that inserted non-RFC3986 behavior (wiggle room) in the first step. None of the $refs are defined in that way.

Also, it's not clear to me what you're' trying to solve with "scopes", as the $refs, $id, and $anchor all already work correctly as far as I'm concerned. Is it the nested $id aspect? I'm guessing yes because you ask:

What confuses me about this language is that it seems to imply that "the enclosing entity" is unique -- that "the entity" is well defined, but isn't it possible, at least for schemas, that a schema might be nested arbitrarily deep in other schemas and as a result there are arbitrarily many possible "enclosing entities". The RFC seems not to consider/allow this.

I can see how this looks weird, but it's actually not. The enclosing entity is unique... for each individual URI-reference. Each schema object that contains a $id is the root of a distinct primary resource (in RFC3986 terms, a resource identified by an absolute-URI, without a fragment). Multiple resources can be nested in a document this way. In this sense, $id does behave somewhat like a scope ,as it sets the base URI for all URI-references in that resource.

A schema object A with an $id that is a subscehma of another object B that also has a $id has B as its enclosing enitity. If B also has a parent schema object C with $id, then C is B's enclosing entity. If there are no more parents with $id but this schema is nested in an OpenAPI Document, then the OpenAPI Document is the enclosing entity. And there might be another beyond that, such as if the OpenAPI Document is one part in a multipart/related that sets the base URI using Content-Location.

Alternatively, you could consider the URI-with-JSON Pointer-fragment in the enclosing schema that points to the enclosed schema as the enclosed schema's retrieval URL, and treat the $id nesting under RFC3986 §51.3 instead of §5.1.2, I've never been sure which is more correct. Either way, it also explains why relative $ids resolve against their parent $id or, failing that, the retrieval URL.

This is all new territory for me, so I appreciate any help in clearing up my confusion on these things.

It's exceptionally confusing territory, especially $dynamicRef. In my defense, at least $dynamicRef works as it was intended, unlike its draft 2019-09 predecessor, $recursiveRef. But it still should not have been written to take a URI-reference.

[handrews]

@mikekistler you might find the "JSF Part 2: A Processing Model" presentation from my abandoned effort to turn JSON Schema into a truly extensible keyword framework to be helpful. It talks about how to process JSON documents and resources and gives examples of how regular and dynamic references work.

[mikekistler]

The picture is getting clearer, but I'm trying to really make sure I understand.

It seems the the crux of this whole matter comes down to RFC 3986 Section 5.1.2:


If no base URI is embedded, the base URI is defined by the representation's retrieval context. For a document that is enclosed within another entity, such as a message or archive, the retrieval context is that entity. Thus, the default base URI of a representation is the base URI of the entity in which the representation is encapsulated.

so I want to make sure I fully understand this. But there are a couple things I find curious

• The term "entity" is not defined in RFC 3986, so I guess it could mean just about anything. It only occurs 11 times in the RFC, most of which are in section 5.1. Examples of an entity in the RFC are a message or archive. It is particularly curious that the term "entity" is used rather than "resource", which is already a very general term

  the term "resource" is used in a general sense for whatever might be identified by a URI.

  The RFC explicitly cites message and archive as two possible entity types. Does this mean that tools that process URIs must understand and properly process resources encapsulated in messages and archives? What other types of entities must be properly handled?


• The term "encapsulating" is also not defined in RFC 3986, and appears only 8 times, most of which are in Section 5.1. And "enclosed" occurs exactly once in the RFC, in the paragraph quoted above. It is particularly interesting that the definition of "Fragment" in Section 3.5 uses "primary" and "secondary" resource and not "encapsulating" and "encapsulated" resource.

Maybe I am being too pedantic about these details, but they bother me. In particular, if someone else were to point these things out to me, I don't know how I could explain them.

[ralfhandl]

• The term "entity" is not defined in RFC 3986

It was defined by HTTP/1.0 (RFC1945) and still present in HTTP/1.1 (RFC2616). Today it only survives in the term "entity tag (ETag)", which is easier to understand if one knows what an "entity" used to be 😎.

The difference to "message body" is rather slim:

The entity-body is obtained from the message-body by decoding any Transfer-Encoding that might have been applied to ensure safe and proper transfer of the message.

RFC9110 uses the term "content" instead of "entity body".

[mikekistler]

Very curious. If the term "entity" was intentionally used over "resource", then it seems that it was intended to mean the entirety of the body retrieved rather than some subset of it that might be a "resource". Do you read it that way?

[handrews]

@mikekistler

the term "resource" is used in a general sense for whatever might be identified by a URI.

The RFC explicitly cites message and archive as two possible entity types.

Do email messages have URIs? Not in general, to my knowledge. Therefore, they are not "resources", but are still something that has to be processed and handled.

Very curious. If the term "entity" was intentionally used over "resource", then it seems that it was intended to mean the entirety of the body retrieved rather than some subset of it that might be a "resource". Do you read it that way?

I think it just means you don't need the encapsulating/enclosing thing to have its own URI, just that it needs to provide context in some sense. For example, a multipart/related HTML page archive, a format intended to be used for an email attachment, uses a per-part Content-Location header to set the base URI (this format could easily be used as a bundling format for OADs, at least if people found multipart to be easy to deal with, which I rather suspect most people don't).

It is particularly interesting that the definition of "Fragment" in Section 3.5 uses "primary" and "secondary" resource and not "encapsulating" and "encapsulated" resource.

Because only primary resources have absolute-URIs and therefore define base URIs. A fragment/secondary resource cannot have a distinct base URI from its primary resource. Fragments, by definition, use the primary resource's URI as the base URI.

[mikekistler]

I think it just means you don't need the encapsulating/enclosing thing to have its own URI

Hmmm. Well the RFC seems to suggest that it does have a base URI, since it says

Thus, the default base URI of a representation is the base URI of the entity in which the representation is encapsulated.

Does it make sense for something to have a base URI without having a URI? How does that happen?

[handrews]

Does it make sense for something to have a base URI without having a URI? How does that happen?

This isn't about the encapsulating entity having a base URI, it's about one providing a base URI. I forgot to link the RFC 2557 for the multipart/related example, where the entity need not have a URI but provides a URI (and base URI) to each part through the Content-Location header.

---

I realized a few more things:

There is language in 3.1.1 using "embedding" / "embedded" that should probably use "encapsulating" / "encapsulated"

I mentioned being unsure about whether to use RFC3986 §5.1.2 or 5.1.3 in the nested encapsulated JSON Schema case - I think it ends up being the same. They are both about determining the base URI from the retrieval context - 5.1.2 works when there is no direct retrieval URI for the encapsulated resource, but one is provided by the encapsulating entity (e.g. the Content-Location header per-multipart/related part in an email attachment), while 5.1.3 works if there is a retrieval URI.

Since encapsulated JSON Schema resource (whether encapsulated in JSON Schema or OAS) can be addressed by the encapsulating resource's URI plus a JSON Pointer fragment, you can treat that directly as a retrieval URI under 5.1.3. Or you can look at the encapsulating resource more abstractly and just use its base URI under 5.1.2 - it's a philosophical difference without practical impact because either way, you are using the encapsulating resource's URI/base URI.

But if your encapsulating entity is a multipart/related email attachment or freestanding resource, you don't use its URI (if it has one), you just use the per-part Content-Location header. So in that case, 5.1.2 vs 5.1.3 has a practical difference (and only 5.1.2 works in the email attachment context since there isn't a retrieval URI for 5.1.3 in that case).