Fix description vs document terminology

[mikekistler]

This PR attempts to fully distinguish "description" and "document" by using "description" consistently for the concept of an API description in the OpenAPI format and "document" to only refer to structural features.

I also revised the "OpenAPI Description Structure" section a bit in part to address the above and also in an attempt to simplify and clarify that section.

Some specifics:

• Every instance of "OpenAPI document" and "OpenAPI description document" has been eliminated. Most are changed to "OpenAPI Description".
• All occurrences of "OpenAPI Description" capitalize the "D" in "Description".
• I changed some instances of "OpenAPI Description" to "OAD", but there are still 33 occurrences of "OpenAPI Description" and at least some of these might be good candidates to change to "OAD".
• the previous text defined "self-contained OpenAPI document" -- I changed that "single-document OpenAPI Description" which I think is clearer and simpler.
• the previous text defined "syntactically complete OpenAPI document" -- I removed that as I believe that other wording changes I made made it unnecessary.

I think there are some good changes here but I'm definitely open to feedback / suggestions on how make it even better.

[ralfhandl]

In general ok, minor nits.

[handrews]

@mikekistler I really appreciate your work here- I am surprised at how many things it has flushed out, and however we resolve those differing views the spec will be much stronger!

I feel a bit bad that I did not advise you to start with 3.1.1, which is what I did for every document parsing and referencing change (all of the others started in 3.0.4). 3.1 is much more complex, so I worked out what made sense there, and then backported the results to 3.0.

This is because 3.0 is, as I think @darrelmiller described, the "uncanny valley" between the 2.0 paradigm (the OAD should function as a single JSON/YAML document even if it is not) and the 3.1 paradigm (it's not, in general, possible to correctly parse only part of a document, and shared documents are expected to be "syntactically complete" but just have components and not paths or webhooks).

Most of the "this needs to be 'document'" stuff is related to supporting components-only "syntactically complete" documents in 3.1. We then want the wording to be as consistent as possible between 3.0 and 3.1, without accidentally imposing 3.1 requirements on 3.0. This is tricky, and is why some of the PRs in this area got re-written multiple times.

I think I'd recommend attempting a PR on 3.1.1 before revisiting this. I think it will clarify a lot of things for you. And me — clearly there is more work to do here than I realized, and I'm really glad you are surfacing it!

[ralfhandl]

We don't define "OpenAPI document" and have to replace it with either "OpenAPI Description" or "OpenAPI entry document", whichever is meant.

And I prefer not to define "OpenAPI document" because it is halfway between "OpenAPI Description" and "OpenAPI entry document" and a larger "Hamming distance" between two terms of the same specification reduces confusion.

[mikekistler]

Folks, I had a discussion with Darrel Miller on this topic yesterday and wanted to recap some of that here.

@darrelmiller please correct me if I captured any of this incorrectly.

At the beginning of the discussion, Darrel felt that there was significance to the term "OpenAPI Document" and was surprised that it was not defined in the spec (we were looking mostly at the 3.0 spec). Darrel proposed this definition for "OpenAPI Document":

An OpenAPI document is any document that follows the syntax and semantics of the OpenAPI Object of the OpenAPI specification.

So an OAD's entry document (as we define this term in the spec) clearly must be an "OpenAPI Document" by Darrel's definition, but other documents referenced from the entry document may also be "OpenAPI Documents" by Darrel's definition. And from a tooling perspective, Darrel explained the importance of this as follows:

If a reference goes to a fragment in an 10 MB file, that fragment might be just 10 lines, but the tooling needs to read / parse most of the file to find it. For efficiency, the tooling will probably parse the whole file and keep the results in memory so that any other references into that file are handled efficiently.

So that makes sense but then begs the question of how the referenced document is parsed. Is it parsed as a JSON document, or an OpenAPI Document, or a JSON Schema document, or ??. I think Darrel was implying that if the file had "openapi" at the root, it would be parsed as an OpenAPI document.

But now we come to this question: Suppose an entry document contains this $ref:

"$ref": "./CommonTypes.json#/components/schemas/ResponseBody"

If the "CommonTypes.json" file happens to have "openapi" at the root, does that mean that it is parsed as an OpenAPI Document? If so, what happens if the version of OpenAPI indicated in the "openapi" field doesn't match the version in the entry document? What happens if the version in the "openapi" field is bogus, e.g. "42"? What happens if parts of the document other than the referenced fragment fail to parse correctly as an OpenAPI document?

But the real issue comes when we get to 3.1 and schemas can have $id fields. In that world, a $ref can contain a URI. To support this, there is an "out of band" process for tooling to "locate" schemas which are collected into a registry keyed by their "id" so that URI's can then be resolved when parsing the OAD.

So suppose in the "out of band process" there is a set of schemas loaded from an OpenAPI 3.1 Document or JSON Schema document. Parsing of those documents should follow the URI resolution rules, which means they need to resolve relative URIs according to RFC 3986 Section 5. In particular, the Base URI for a schema may come from the "Encapsulating Entity", and relative URIs within this schema are then resolved according to this Base URI.

So far, so good. The schema is entered into the registry and any URI with its "id" gets that schema. BUT, suppose an OAD contains a $ref to this schema by id and in another place it also contains a $ref to this schema by location? Now there can be a problem, because references by location don't necessarily consider the "context" of the fragment -- meaning that the base URI in the encapsulating entity might not be considered, and relative URIs in the schema may be resolved differently than when the schema was referenced by id. If the schema contains a "$id" field, then this is broken, because the same id represents two different schemas.

Maybe everyone else understood this and it is just me that is now coming to this realization.

I'll stop writing here as this is already too long, but I hope this is helpful to moving this discussion along.

[ralfhandl]

We really should add Darrel's definition of "OpenAPI document" to the Definitions section, and then work through the PR from there.

And start in parallel on a similar PR for 3.1.1 so we can see the necessary differences simultaneously.

@mikekistler Could you please create the second PR? You seem to have dug rather deep into this topic.

[mikekistler]

One new thing to add here. I am far from being an expert on JSON Schema, particularly the latest versions, so I was studying the 2020-12 draft looking for some better understanding, and found Section 7.1 Lexical and Dynamic Scope. I think this is essential reading as it describes the problem we are wrestling with and may offer a way out. I'll snip out here some of the key pieces:

Note that some keywords, such as "$schema", apply to the lexical scope of the entire schema resource ..

Other keywords may take into account the dynamic scope that exists during the evaluation of a schema, typically together with an instance document.

Lexical and dynamic scopes align until a reference keyword is encountered ... A keyword on the far side of that reference that resolves information through the dynamic scope will consider the originating side of the reference to be their dynamic parent, rather than examining the local lexically enclosing parent.

it may be possible to revisit the same lexical scope repeatedly with different dynamic scopes.

Now, it was not clear (to me anyway) whether "$ref" is a keyword that is resolved in lexical scope vs dynamic scope. But if we consider "$ref" to be resolved in dynamic scope, then I think we can treat "$id" as identifying the "lexical scope" of the schema -- which might be resolved to different things in different dynamic scopes. So there is only one schema with this id, but it resolves to different things in different dynamic scopes.

What do folks think of this idea?

[handrews]

@mikekistler First, I'm really glad you're getting into this, and that you and @darrelmiller had a great discussion about it. Darrel has heard all of the frustrations I encountered on this during OASComply, and we compared notes about the challenges of ipmlementing 3.1 at some point.

Maybe everyone else understood this and it is just me that is now coming to this realization.

No, I've been driving myself absolutely crazy(-er 🤪 ) trying to get people to understand this for the past several years, but it's really hard to explain until you really try to follow it all through to implement it. Congratulations on being one of the few who have now dug deep enough to get it :-) (I mean that sincerely- it's a lot of obscure detail but it's really important, and I wish I'd been able to communicate it more clearly. I have more than enough data to know that it's not easy to figure out, or else more people would have by now).

An OpenAPI document is any document that follows the syntax and semantics of the OpenAPI Object of the OpenAPI specification.

So an OAD's entry document (as we define this term in the spec) clearly must be an "OpenAPI Document" by Darrel's definition, but other documents referenced from the entry document may also be "OpenAPI Documents" by Darrel's definition.

I would definitely support this, as it defines "OpenAPI Document" the same as what I was calling a "syntactically complete" document, grasping for a way to distinguish it from fragmentary documents (that are not JSON Schema documents).

If the "CommonTypes.json" file happens to have "openapi" at the root, does that mean that it is parsed as an OpenAPI Document? If so, what happens if the version of OpenAPI indicated in the "openapi" field doesn't match the version in the entry document? What happens if the version in the "openapi" field is bogus, e.g. "42"? What happens if parts of the document other than the referenced fragment fail to parse correctly as an OpenAPI document?

I hit every single one of these problems trying to write oascomply. The only reason I haven't pushed harder on getting all of them resolved is that it was hard enough to get attention on the more obvious problems, and until you (Mike) started digging into this it just seemed like people were getting burnt out on my efforts. And so was I, tbh. Some parts of this have been answered, more on that in a bit. Most of the rest could be answered by keeping things analogous to the parts that have been answered.

But the real issue comes when we get to 3.1 and schemas can have $id fields. In that world, a $ref can contain a URI. To support this, there is an "out of band" process for tooling to "locate" schemas which are collected into a registry keyed by their "id" so that URI's can then be resolved when parsing the OAD.

This is all 100% completely addressed by 3.1.1 §4.3.1 "Parsing Documents." This is actually a thoroughly understood and solved problem in JSON Schema, and you can find many implementations of it in the wild.

It does not necessarily have to be out-of-band: 3.1.1 states that if it looks like an OpenAPI Document or a JSON Schema document, you can treat it like one. It also allows for parsing a small-d document based on a $ref to its root, as that provides the type of the whole docment (let's call it an OpenAPI Fragment, for example a document consisting of a Path Item Object). 3.1.1 explicitly notes that tools can be configured to know where a URI is located- this is, again, common in JSON Schema and has been for a long time (technically since well before draft-04, but I don't think we got the explanation clear enough until draft-07-ish. It's much more explicit in 2019-09 and 2020-12 where we added a whole section on loading schemas.

So far, so good. The schema is entered into the registry and any URI with its "id" gets that schema. BUT, suppose an OAD contains a $ref to this schema by id and in another place it also contains a $ref to this schema by location?

This is also all well-understood in JSON Schema-land, and is why both JSON Schema and OAS 3.1 require full-document parsing. I'll come back to it after addressing the next few statements:

Now there can be a problem, because references by location don't necessarily consider the "context" of the fragment

This is not true. There is no difference between resolving a fragment in a document based on identity vs location, it works exactly the same way. RFC3986, in fact, only cares about identity. Treating a URI as a URL is just a handy default way to locate an identified document. Whether a given URI can reasonably be treated as a URL is up to the application involved.

meaning that the base URI in the encapsulating entity might not be considered, and relative URIs in the schema may be resolved differently than when the schema was referenced by id. If the schema contains a "$id" field, then this is broken, because the same id represents two different schemas.

It's not broken, it works exactly as intended. This whole "ignore the format of the document" is a thing I've never seen anywhere else that has no relationship to how URIs, media types, and resources are supposed to work, which is for their behavior to be keyed by media type (in the absence of a media type, something like a file extension or content sniffing can be used as a heursitic).

AFAICT, this "ignore the context" thing is completely unique to OAS. Although I mght well be wrong, of course. I'm pretty sure it is not something that comes from JSON Reference which itself does not appear to understand how URIs, media types, and fragments are supposed to work (because it mandates behavior counter to RFC9601 regarding JSON pointers as URI fragments).

---

Regarding addressing a JSON Schema document (for simplicity- OAS doesn't change things much as so far it doesn't have an $id-equivalent, but now you are starting to see why it needs one in 3.1), there are a few cases:

• Reference the document by location, and it doesn't contain any $id, so this just works as you'd think
• Ref by location, $id in the root: In this case, both the location (retrieval URL) and $id are valid URIs for the schema. It's a little unclear if the retrieval URL should be retained for future reference or discarded ,so I'd always reference by identity- the whole point is to make your $ref values stable even if the target moves around. Repeatedly ref-ing by location might lose the benefits of the cache.
• Ref by location, $id in a subschema:
  • As long as your JSON pointer doesn't cross into the $id subschema, you are fine
  • If your JSON Pointer crosses into a subschema of the subschema containing $id, the behavior is implementation-defined. Don't do this. We were forced to make this implementation-defined because some people didn't want to have to check for an error. I wanted it to be an error
  • If your JSON Pointer points to the subschema containing the $Id, this is essentially the same as the retrieval URL vs identity case- the spec doesn't nail it down, but I'd expect it to work (the spec only says that crossing the $id is implemenation-defined). But still, I'd always reference by identity.

(that's enough for this comment, I'll probably make a few follow-ups)

[handrews]

@mikekistler

Now, it was not clear (to me anyway) whether "$ref" is a keyword that is resolved in lexical scope vs dynamic scope. But if we consider "$ref" to be resolved in dynamic scope, then I think we can treat "$id" as identifying the "lexical scope" of the schema -- which might be resolved to different things in different dynamic scopes. So there is only one schema with this id, but it resolves to different things in different dynamic scopes.

What do folks think of this idea?

The way $id, $anchor, and $ref (any of the $refs) work is 100% governed by RFC 3986 and there is no wiggle room at all in that. URIs are Universal: there aren't any scopes. There are, however, base URIs, which are well-specified (although the OAS kind of munges it by skipping some of the steps- I tried to fix that by not over-specifying it in 3.0.4 and 3.1.1). The only thing a media type can really impact is whether it sets the base URI within the document contents. The other three possible base URI sources are not under the media type's control.

The whole dynamic scope thing has to do with how $dynamicAnchor and $dynamicRef work. TL;DR: $dynamicRef searches the resources (not objects or documents) in the dynamic scope for the most distant (first-parsed that is still in scope) $dynamicAnchor, and sets the base URI of the $dynamicRef value to the absolute URI of that resource. If no distant $dynamicAnchor is found, then the base URI doesn't change and $dynamicRef behaves exactly like $ref.

I've never been entirely satisfied with this mechanism. It basically inserts some steps before handing off to normal RFC3986 behavior.

The "scope" language has more to do with how certain JSON Schema keywords communicate up and down the dynamic scope, which is how unevaluatedProperties works.

[handrews]

I guess the overarching theme here is: All of the URI-based parts are well-defined in 3.1, and are now (hopefully) actually well-explained in 3.1.1. 3.0 is both easier because it is location-only, and harder because location-only isn't very practical in a lot of environments, which is why some tools already allow you to load a document from an alternate location.

[mikekistler]

There are a number of things I'm struggling with here. You say

The way $id, $anchor, and $ref (any of the $refs) work is 100% governed by RFC 3986 and there is no wiggle room at all in that. URIs are Universal: there aren't any scopes.

and later

The whole dynamic scope thing has to do with how $dynamicAnchor and $dynamicRef work.

But the 2020-12 JSON Schema spec says:

The value of the "$dynamicRef" property MUST be a string which is a URI-Reference.

So if $dynamicRef is a URI-Reference, why doesn't RFC 3986 disallow scopes for this as well?

Also, when you say "there is no wiggle room" ... I'm guessing you are referring to Section 5 of RFC 3986 describing the rules for reference resolution, and maybe more specifically to 5.1.2 Base URI from the Encapsulating Entity:

If no base URI is embedded, the base URI is defined by the representation's retrieval context. For a document that is enclosed within another entity, such as a message or archive, the retrieval context is that entity. Thus, the default base URI of a representation is the base URI of the entity in which the representation is encapsulated.

What confuses me about this language is that it seems to imply that "the enclosing entity" is unique -- that "the entity" is well defined, but isn't it possible, at least for schemas, that a schema might be nested arbitrarily deep in other schemas and as a result there are arbitrarily many possible "enclosing entities". The RFC seems not to consider/allow this.

This is all new territory for me, so I appreciate any help in clearing up my confusion on these things.

[handrews]

@mikekistler

So if $dynamicRef is a URI-Reference, why doesn't RFC 3986 disallow scopes for this as well?

I'll direct you to my statement:

've never been entirely satisfied with [the $dynamicRef] mechanism. It basically inserts some steps before handing off to normal RFC3986 behavior.

By "no wiggle room" for $ref I mean that it's just defined as a URI-reference with no special rules. With $dynamicRef, it's defined as a two-step process: Do this weird thing to find the base URI and then follow RFC3986. In retrospect, I don't think this was a good idea. I should have come up with a different syntax that more clearly separated the non-RFC3986 and RFC3986 behaviors.

So, $dynamicRef was defined as a two-step process that inserted non-RFC3986 behavior (wiggle room) in the first step. None of the $refs are defined in that way.

Also, it's not clear to me what you're' trying to solve with "scopes", as the $refs, $id, and $anchor all already work correctly as far as I'm concerned. Is it the nested $id aspect? I'm guessing yes because you ask:

What confuses me about this language is that it seems to imply that "the enclosing entity" is unique -- that "the entity" is well defined, but isn't it possible, at least for schemas, that a schema might be nested arbitrarily deep in other schemas and as a result there are arbitrarily many possible "enclosing entities". The RFC seems not to consider/allow this.

I can see how this looks weird, but it's actually not. The enclosing entity is unique... for each individual URI-reference. Each schema object that contains a $id is the root of a distinct primary resource (in RFC3986 terms, a resource identified by an absolute-URI, without a fragment). Multiple resources can be nested in a document this way. In this sense, $id does behave somewhat like a scope ,as it sets the base URI for all URI-references in that resource.

A schema object A with an $id that is a subscehma of another object B that also has a $id has B as its enclosing enitity. If B also has a parent schema object C with $id, then C is B's enclosing entity. If there are no more parents with $id but this schema is nested in an OpenAPI Document, then the OpenAPI Document is the enclosing entity. And there might be another beyond that, such as if the OpenAPI Document is one part in a multipart/related that sets the base URI using Content-Location.

Alternatively, you could consider the URI-with-JSON Pointer-fragment in the enclosing schema that points to the enclosed schema as the enclosed schema's retrieval URL, and treat the $id nesting under RFC3986 §51.3 instead of §5.1.2, I've never been sure which is more correct. Either way, it also explains why relative $ids resolve against their parent $id or, failing that, the retrieval URL.

This is all new territory for me, so I appreciate any help in clearing up my confusion on these things.

It's exceptionally confusing territory, especially $dynamicRef. In my defense, at least $dynamicRef works as it was intended, unlike its draft 2019-09 predecessor, $recursiveRef. But it still should not have been written to take a URI-reference.

[handrews]

@mikekistler you might find the "JSF Part 2: A Processing Model" presentation from my abandoned effort to turn JSON Schema into a truly extensible keyword framework to be helpful. It talks about how to process JSON documents and resources and gives examples of how regular and dynamic references work.

[mikekistler]

The picture is getting clearer, but I'm trying to really make sure I understand.

It seems the the crux of this whole matter comes down to RFC 3986 Section 5.1.2:


If no base URI is embedded, the base URI is defined by the representation's retrieval context. For a document that is enclosed within another entity, such as a message or archive, the retrieval context is that entity. Thus, the default base URI of a representation is the base URI of the entity in which the representation is encapsulated.

so I want to make sure I fully understand this. But there are a couple things I find curious

• The term "entity" is not defined in RFC 3986, so I guess it could mean just about anything. It only occurs 11 times in the RFC, most of which are in section 5.1. Examples of an entity in the RFC are a message or archive. It is particularly curious that the term "entity" is used rather than "resource", which is already a very general term

  the term "resource" is used in a general sense for whatever might be identified by a URI.

  The RFC explicitly cites message and archive as two possible entity types. Does this mean that tools that process URIs must understand and properly process resources encapsulated in messages and archives? What other types of entities must be properly handled?


• The term "encapsulating" is also not defined in RFC 3986, and appears only 8 times, most of which are in Section 5.1. And "enclosed" occurs exactly once in the RFC, in the paragraph quoted above. It is particularly interesting that the definition of "Fragment" in Section 3.5 uses "primary" and "secondary" resource and not "encapsulating" and "encapsulated" resource.

Maybe I am being too pedantic about these details, but they bother me. In particular, if someone else were to point these things out to me, I don't know how I could explain them.

[ralfhandl]

• The term "entity" is not defined in RFC 3986

It was defined by HTTP/1.0 (RFC1945) and still present in HTTP/1.1 (RFC2616). Today it only survives in the term "entity tag (ETag)", which is easier to understand if one knows what an "entity" used to be 😎.

The difference to "message body" is rather slim:

The entity-body is obtained from the message-body by decoding any Transfer-Encoding that might have been applied to ensure safe and proper transfer of the message.

RFC9110 uses the term "content" instead of "entity body".

[handrews]

This is getting very, very close! Thanks for all of the continued hard work on this @mikekistler and everyone else involved!

[handrews]

Partial follow-up to the TDC call and subsequent discussions with @darrelmiller .

The fix related to the big table of how to resolve relative URL-references in different fields will have to be a separate PR that I will submit shortly as GitHub does not allow making "suggestions" on lines that were not already changed by the PR.

Assuming these changes are accepted.

[handrews]

See #4130 for the follow-on change that I could not make a "suggestion" here because those lines were not changed in this PR.

[ralfhandl]

Minor nits

[handrews]

I think that's all the concerns resolved! 🎉