Proposal: Bundled streams w/ self-identification

[handrews]

[NOTE: Please prioritize the open 3.0.4 and 3.1.1 PRs]

This adds a proposal for bundling both OAS (including full 3.1 support) and Arazzo using YAML native streams or any of several JSON document stream formats. It includes self-identification of documents to ensure that all references within the stream can be resolved without looking outside of the stream (except for External Documents).

Also paging @frankkilcommins for the Arazzo aspect (GitHub won't let me make you a reviewer, for who knows what reason).

[ralfhandl]

+1, minor nits

[mikekistler]

There is no way to combine all of the OAD and Arazzo documents involved into a single document

I'd like to understand why this is the case. What about just OAD documents?

[kevinswiber]

Objects in descriptions are scoped to the document in which they originate. We don't really have semantics for merging multiple documents. In some cases, it would involve renaming unique identifiers like map keys. Tools that have done this create an inconsistent experience, and to reverse the process requires a source map of some kind. Maintaining document identity in a multi-document scenario obviates the need for that.

[handrews]

@mikekistler basically what @kevinswiber said, but I'll add two things:

I chatted with someone on some Slack a while ago who was having a lot of problems because his "merge to a single document" tools that he needed in order to get something (AWS gateway, maybe?) to accept the OAD was messing things up with what it though were "safe" transformations. As Kevin pointed out, there are a lot of choices that tools doing this have to make, and those can be surprising and sometimes breaking. When you start with a single document, split it, and then use the same toolchain to re-combine the pieces, that works quite well because the split and combine tool are the same and make the same assumptions. But mixing toolchains doesn't work well here.

Another way to look at it is to look at how JSON Schema bundling works. It relies heavily on "$id" to preserve not just the referencing behavior but the literal reference values. This way there isn't any "rewriting" of the documents, and no need to merge them. They are simply placed in the standard location ("$defs") and used in a way that the name under "$defs" is irrelevant.

As briefly mentioned in the call this morning, if we want to support "$id" (or an equivalent named differently to reduce confusion with JSON Schema) in every Object type in the OAS and Arazzo, we could absolutely implement single-document bundling. But nested "$id"s make it much harder to determine the base URI, which is something that tooling vendors seem to struggle with (or are just indifferent to) as it is. Using a document stream minimizes the changes and implementation work involved.

I'm trying to get the maximum improvement for the minimum effort with this proposal.

[kevinswiber]

Could we make composite keys using a combination of the identifier for the entry OAD and identifiers for dependency documents?

[handrews]

@kevinswiber Arazzo identifying targets in OADs is not a problem, if that's what you mean. I'm just kind-of assuming that since multi-document support has been a challenge (or just not viewed as cost-effective) for our community, publishing a spec that depends on reading and using multiple documents seems a touch risky.

[kevinswiber]

For interleaving multiple OAD streams, we could also include an optional field to identify the bundle, perhaps using the entry document ID. Ideally, it would allow multiple bundles to prevent duplicating documents in certain scenarios, but I'm usually the odd one out for being cool with arrays.

[handrews]

@kevinswiber I think this would be a good further direction, but I worry about making the initial thing too complicated. Really, the only thing you need to solve for having multiple OADs in a stream is how to identify each entry document. This proposal just says "the first document in the stream is the entry document", which means there can only be one clear OAD in the stream. But there's no reason you couldn't get a stream, and then select (by URI) each entry document you want to try to parse from the stream. As long as all needed documents are in the stream, it would work fine.

[handrews]

Per yesterday's TDC call, I am going to split this into two proposals, as the self field met with wide acceptance, while there are still questions and concerns about bundling. Ironically, these two were only together because I though the bundling piece would be enthusiastically embraced, and would be the only way to sell folks on the self field 🙃