Imports proposal

[handrews]

NOTE: Credit for the imports object and import-using syntax goes to Darrel Miller. I just wrote it up and added pieces around how imports get resolved.

I feel that it's important to get imports nailed down early on, for several reasons:

1. We want this to be a reliably implemented part of the spec, without the problems that have plagued referencing.
2. Understanding how we will do imports will make it a lot easier to think about what can and cannot be backported from 4.x to 3.x.

This proposal attempts to codify what @darrelmiller has already used in the deployments proposal, with gaps filled in based on my experiences writing oascomply and dealing with the inconsistent support for resolving and loading referenced schemas in JSON Schema-land.

I can provide use cases for anything that I intentionally added beyond what the deployments proposal uses.

This is written more-or-less like specification text, but I'm assuming the wording will go through a lot of changes.

NOTE: This proposal does not address the interaction between Moonwalk importing an JSON Schema referencing. See discussion #73 for that aspect of the proposal.

---

OpenAPI Description Structure

An OpenAPI Description (OAD) MAY be made up of a single document, or divided into multiple documents connected by imports.

This specification defines the OpenAPI Description Document (OAD Document) format. OADs can incorporate other formats, such as JSON Schema, as stated for specific keywords in this specification or extensions to it.

Relative References in Description IRIs

The base IRI for relative IRI-references that identify OAD resources MUST be determined in accordance with RFC 3987 §6.5 which delegates the process to RFC 3986 §5.

The self field in the OpenAPI Object, if it is present, MUST be used as the base IRI in accordance with RFC 3986 §5.1.1.

As the full environment necessary to determine a base IRI might not be visible an OAS implementation, implementations MUST allow the user to specify an external base IRI for each OAD Document, which is assumed to be in accordance with RFC 3986 §5.1.2 – §5.1.4.

If self is not present and no external base IRI is provided, implementations MAY define and document a default base IRI in accordance with RFC 3986 §5.1.4. Otherwise, if relative IRI-references are present, implementations MUST raise an error.

Embedding OAD Documents in Other Formats

NOTE: This section added 11 Feb 2024.

Other formats MAY embed OAD Documents. Embedded OAD Documents SHOULD define a self import. If no self import is present, the behavior is undefined by this specification.

It is RECOMMENDED that formats embedding OADs require and enforce a self import that uses an absolute-IRI in order to be interoperable with tools that parse and use the embedded OAD Document format without understanding the containing format. The behavior of OAS-specific tools with embedded OADs that do not define such a self field is implementation-defined.

TODO: should implementations be required to support embedded OAD Documents with an absolute-IRI self field, or is that a MAY feature only? Presumably, parsing out the embedded document so it can be treated as if it were standalone would happen before anything else. It would complicate using a URL to import things as it would presumably require a fragment, and I am highly reluctant to require handling such things directly.

Relative References in API URLs

Relative URL-references in the described API are resolved using the base URL defined in the relevant Deployment Object. If no Deployment Object is defined or relevant, API URL-references MUST be resolved as described for IRIs in the previous section.

Schema

OpenAPI Object

Fixed Fields

Field Name	Type	Description
self	IRI-reference (without a fragment)	Sets the IRI of this document, which also serves as its base IRI in accordance with RFC 3986 §5.1.1; the value MUST NOT be the empty string
imports	[Import Object]	A list of resources to import, from which all references within this document MUST be resolved

NOTE: The following paragraph was added 19 Feb 2024.

The value of self is an identifier, and not necessarily a locator. The value need not be usable as a URL, and even if it is, the document need not be accessible at that location.

Import Object

An object associating a resource, identified by an absolute-IRI, with a namespace.

Fixed Fields

TODO: namespace syntax, which MUST allow for full use of unicode in human-readable components

Field Name	Type	Description
namespace	string	A name, unique to the imports in this document, that is used to reference the imported document; this name MUST NOT be 'self'
href:	IRI-reference (without a fragment)	The IRI of the document to import; this IRI is not necessarily usable as a URL.

The namespace self is reserved as the namespace for the current document.

Using Imported Names

NOTE: This syntax comes directly from the deployments examples. It does involve values that are either objects (inline) or strings (imported names) which might be of concern to developers in strictly typed languages.

Imported names are used in specific contexts that require a particular type of Object. The syntax for using a name is:

  <fieldName>: <namespace>:<componentName>

where <namespace> is replaced by the namespace from an Import Object in the current document, and <componentName> MUST but the name of an Object in the imported document's Components Object, under the appropriate semantic type section. Therefore:

openapi: 4.0.0
self: https://example.com/fooComponents
components:
  pathItems:
    Foos: {...}
  schemas:
    Foos: {...}

openapi: 4.0.0
self: https://example.com/fooPaths
components:
  pathItems:
    Bars: {...}
imports:
- namespace: foo
  href: fooComponents
paths:
  /foos: foo:Foos
  /bars: self:Bars

In this example, the import href "fooComponents" is an IRI-reference resolved against self as a base IRI, producing https://example.com/fooComponents. Since the foo:Foos is found where a Path Item Object is expected, the "Foos" entry under pathItems is used rather than the identically-named Schema Object under schemas.

The /bars entry shows how to use a component from the current document with the reserved self namespace.

Referencing a Complete Document

NOTE: This section added 11 Feb 2024.

Some fields require a document-level reference, such as connecting a deployment to its shape. In that case, only the namespace is used, without the component name separator:

  <fieldName>: <namespace>

This form MUST only be used when it makes sense to talk about the effect of the complete document, rather than components within the document.

Locating and Loading Imported Resources

NOTE: The requirements in this section are phrased to allow, but not require, compliant implementations to delegate all I/O to applications. It should also be possible to implement resolvers separate from the core OAS support, somewhat like reference resolvers/removers but with much better UX.

Implementations MUST support two methods of resolving imports:

• matching the import IRI to the resource's self-assigned IRI
• treating the import IRI as a retrieval URL

Note that interacting with local files is equivalent to retrieval using file: IRIs ([RFC 8089])(https://datatracker.ietf.org/doc/html/rfc8089).

As implementations cannot determine a self-assigned IRI without parsing the resource, they MUST support pre-loading resources or otherwise configuring some mechanism to load likely candidates on demand.

Implementations MAY directly support URL retrieval. Those that do SHOULD support URI schemes most relevant to their environment, such as file: for local access and https: for network access.

Implementations that do not support URL retrieval MUST support associating a retrieval URL with each resource.

Security Considerations for URL Retrieval

TODO

[ralfhandl]

Will this be our general pattern?

Everywhere one can "inline" an object one can also place a string with a reference to an imported or locally defined component object

[handrews]

@ralfhandl I don't know whether "everywhere" means literally everywhere (probably not- importing strings would be ambiguous, for example), every named object in the spec (possibly- I don't really see why not) or some subset of objects the way we limit Reference Objects now.

I also avoided getting into ways to group and reference groups of objects, which is a topic that has come up at times, and also avoided any question of modifying an imported object at point of use. I think it's best to get the general sense of things settled first. Those questions may impact the exact name-using syntax (particularly if we want to support modifying imports), but I think that's fine. It's more the questions of whether this approach to specifying imports and resolving them is correct right now.

[ralfhandl]

modifying an imported object at point of use

Oh yes, that idea has been bounced around for some time, for example a "parameterizable path items object" for describing a "pattern-based" API without having to spell out every instantiation of that pattern.

Better save that for later...

[handrews]

@ralfhandl JSON Schema in OAS 3.1 effectively has that sort of parameterization with $dynamicAnchor and $dynamicRef. OAS processing outside of schemas is simpler (there is no instance on which to make conditional decisions) so a simplified form of that might not be unreasonable. But yes, let's do that after we get the basics done!

[handrews]

I am particularly interested in whether folks think the self field (which only changes base IRI/URI determination in that you have to check it first) is straightforward enough to backport to 3.2. Persistent, stable, location-independent identifiers are things I've had people ask for in 3.x, and it's only possible with Schema Objects that use $id right now.

[handrews]

I also want to call attention to the requirements on what implementations MUST accept. As noted in the proposal, these requirements assume an implementation that does not do any I/O, and either accepts JSON/YAML text or just parsed JSON/YAML in whatever data structure is typical for the environment.

• MUST support two ways to resolve an import IRI: as a document's self IRI or as its retrieval URL
• For each document:
  • MUST accept an external (RFC 3986 §5.1.2-5.1.4) base IRI, which is assumed to be correct according to those sections of the RFC
  • MUST allow pre-loading to determine the self IRI, or otherwise offer a mechanism to map imported IRIs to documents that might have a matching self IRI
  • MUST accept a retrieval URL (if embedded in an application that does URL retrieval, this just means providing the URL to the "core" OAS implementation)

Note that:

1. While implementations MUST accept this info, not all of it will always be provided. For single-document OADs, you can get away without providing any of the three IRIs (self, external base, or retrieival).
2. While the retrieval URL is one way to determine a base IRI (RFC 3986 §5.1.3), it may or may not match the provided external base IRI as §5.1.2 might pre-empt it; we just trust that the user/application is giving us the right one in each case

I assume many implementations will do their own parsing and local filesystem and/or network I/O! For those tools, it's useful to think about the internals as the OAS implementation, and the part that does parsing and I/O as an "application". This allows tools to handle that I/O and parsing however they think is best while still being compliant.

[handrews]

Note that I just added two sections to the proposal above:

• Embedding OAD Documents in Other Formats
• Referencing a Complete Document

[SensibleWood]

It would complicate using a URL to import things as it would presumably require a fragment, and I am highly reluctant to require handling such things directly.

I agree. There'd be a great deal of complexity involved in defining this at specification-level that would make adoption harder not easier IMHO. Allowing the referencing to happen concisely seems pragmatic to me.

[pavelkornev]

without the problems that have plagued referencing

Could we clearly document what exact problems this proposal is trying to solve comparing to the current referencing approach using $ref? Could be under "Motivation" subsection.

[handrews]

(also, if there's not anyone else that can explain the motivation, that's probably a sign to reconsider something)

[miqui]

@handrews ---> "Alternatively, I already wrote a detailed report with a brief summary that hardly anyone read," <--- buddy, I read your stuff as fast as I can. But, given the technical nature (like RFCs) I have to read it many times before it sinks in.

thanks for sharing and putting this together.

[handrews]

@miqui I guess I should say more accurately that not that many people commented. I imagine more read it than said anything about it!

[handrews]

@pavelkornev I've been thinking about this more since yesterday's meeting, and I guess there are two starting points.

No tools fully and correctly implement referencing across all versions

This is the primary reason to change from "referencing" to "imports."

It almost doesn't matter what else is going on... the point of a standard is to enable interoperability. If an aspect of a standard is not consistently implemented (or is often left un-implemented), then the standard has failed in that fundamental goal.

I'm making a pretty bold claim here to say that there are no such implementations, but it's based on a few observations:

• No one has ever been able to convincingly tell me exactly what the full requirements for referencing support even are. There are tons of ambiguities in the spec, and very little proper normative language
• As documented in the report linked above, if you resolve those ambiguities based on what the OAS/Swagger 2.0 text implies, you will get one behavior. If you resolve those ambiguities based on how JSON Schema behaves (because OAS 3.1 requires supporting that), you will get contradictory behavior.
• It's theoretically possible (again as acknowledged in reports linked above) to use JSON Schema 2020-12 behavior only in 3.1 Schema Objects and 2.0 behavior everywhere else, but I've never heard anyone talk about this. It could be out there, and if anyone knows of such a thing please tell me about it
• It's not just about $ref, it's also (again, from the reports) about how $ref interacts with other features

If anyone thinks they know a tool that fully and correctly implements referencing across at least two versions, I would very much like to see it. Be aware that I will pick at it and challenge whether certain things are "correct" or just assumptions in the face of ambiguity. Assumptions are generally not interoperable, and show a failure of the specification.

The part where I'm really burnt out is convincing people of all of these things that aren't implemented, or that their "but of course it's done this way" is not actually stated in the spec, and therefore can't be relied upon by users.

There are substantial use cases for identity separate from location

This is what motivates the self field, and the rules for what information needs to be accepted to resolve imports and properly set base URIs. Part of this comes from long experience with analogous behavior in JSON Schema. But the larger OAS-specific part comes from a real-world use case from a client of mine (who are aware that I am citing the for this).

I honestly hesitate to bring this up because of how often it's been dismissed, not on the merits, but because critics have never personally seen the need for such a thing. I hope anyone who objects to this will focus on costs and benefits, and not their own personal assessment that "no one" will want this. Separation of location and identity is a feature of many systems using URI-identified resources.

Lessons learned from JSON Schema

Part of that is just directly applying the experience of coming into JSON Schema post-draft-04, seeing that support for $ref and id (later split into $id and $anchor) was an absolute dumpster fire of inconsistency, when it was supported at all (sound familiar?)

It took numerous drafts to find what things we needed to say in the spec to get support to be consistent and reliable. It was particularly challenging to find the right wording to explain what is necessary to support $id being an identifier and not just a locator.

At this point, most JSON Schema implementations support references in a predictable way, but with enough discretion left to implementations to not back folks into a corner. I want to have the same situation with OAS tools.

The OGC's Building Blocks

The API specifications defined by the Open Geospatial Consortium (OGC) are an example of complex, multi-layered systems, incorporating deployable schema components from multiple publishers. These components are often important to identify at the point of use, because they represent standards that can be processed by 3rd-party tools or libraries once parsed out of the API payload.

The OGC defines general purpose API components, expecting these to be extended to meet application needs. These re-usable components correspond to industry and other standards that its members need to incorporate into their APIs. These components include but are not limited to schemas.

These "Building Blocks" also build on other standards, such as GeoJSON, which need to be recognizable within the building blocks. The Building Blocks are often containers that are further customized to hold specific kinds of data. However, the containers need to be recognizable within that re-use.

The result is a complex layered system of at least three different publishers: Low-level standards such as GeoJSON, the OGC's own standards work, and the specifics of each individual API provider. Each layer has tools that can work with it, if the layer can be recognized.

While it is theoretically possible to determine if two JSON Schemas accept the same data set, this can be complex in practice. Even if implemented, false positives can happen with schemas that just happen to look like a standard data type.

All of these components need recognizable identifiers that remain in the published OADs. Removing them (by "de-referencing") or changing the reference from a stable URI to a custom URL (for packaging/deployment) destroys the information necessary for the functioning of the geospatial API ecosystem.

[handrews]

Here is another use case for separating identity and location, this one based on network authentication problems:

• Support resolution of refs which require authentication OpenAPI-Specification#3270