Forward compatibility with must-understand keywords

[awwright]

When submitting JSON Schema as an Internet media type (e.g. application/schema+json), one of the potential problems in JSON Schema is: how does it support forward compatibility?

This would be easy to implement if there were some standard way to distinguish between must-understand keywords and may-understand keywords. This way, a validator could understand when there's a keyword that should be performing validation, but that is not implemented.

This example uses keywords that contain a . to denote keywords that are may-implement, and other keywords are mandatory, must-understand:

{
   ".title": "Price",
   "type": "number"
}

Or using TitleCamelCase, which may be friendlier when encoding schemas as ECMAScript objects (afaik, we don't have any keywords that begin with a capital letter):

{
   Title: "Price",
   type: "number",
}

The other way to do this would be to indicate the must-understand keywords, for example, a ! suffix:

{
   "title": "Price",
   "type!": "number"
}

[Relequestual]

I'm confused. This use case is fully supported by vocabularies and dialects. This is exactly what they are for. Are you wanting another method on top of that?

[awwright]

This allows the default vocabulary (when there's no $ keywords) to evolve, and not have to be written in stone, so to speak.

[Relequestual]

The default dialect (and therefore vocabularies) are implementation defined.

If [$schema is] absent from the document root schema, the resulting behavior is implementation-defined.

Without a dialect being defined, how would an implementation know WHICH interpritation of any given keyword to use? "Must understand" based on whos definitons?

Could you elaborate on the value proposition here, and how you would expect it to work please?

[karenetheridge]

Without a dialect being defined, how would an implementation know WHICH interpritation of any given keyword to use?

The spec says (in the very line quoted above!) that in the absence of any other direction, behaviour is implementation-defined. Therefore the implementation can do whatever it wants (but please document it).

In my implementation, I provide a constructor argument for the specification version. It defaults to the latest, 'draft2020-12', when not provided. So, if you pass a schema without any $schema keyword, you get vanilla draft2020-12 behaviour. You can pass a schema without a $schema keyword, but with specification_version => "draft7", to get vanilla draft7 behaviour.

[Relequestual]

One thing we know we want is predictable interoperability.
"implementation-defined" is like saying "you're colouring outside the lines so you're on your own, bud."
If I were to write an implementation, I would throw an error without any indication (via $schema or param as per @karenetheridge 's example) as to the dialect. But I don't have an implementation! 😅

[awwright]

I only saw your reply just now as I link to it @Relequestual—A few points here

This phrasing is incompatible with earlier drafts of JSON Schema, which specifies definitions for every keyword, regardless of the presence or value of $schema.

I think it should be a feature that $schema should be able to change the vocabulary in use, and the addition of this feature is compatible with pre-existing schemas in the wild. However, no forward compatibility is defined: Validators will ignore an unknown value of $schema if present, when perhaps they should error when encountering an unknown value. (Ignoring unknown values is proscribed in older drafts, and undefined in the current draft.)

And also, the idea that the entire document would be completely implementation-defined is somewhat incompatible with the purpose of a media type. Schemas like {"type": "string"} have a definite, global meaning, not an implementation-defined one.

[Relequestual]

Replying to your comments at #69 (comment)

That is: New features can continue to evolve, but people implementing against that release should be guaranteed forward compatibility.

How is this associated with an RFC?
RFCs can be superseeded. Does doing so require or guarantee any forwards compatability?

In #119 I asked: Is there a way we can guarantee that older validators reading schemas with newer features don't report a false positive? I think if we can answer this, we can publish a stable RFC, and get our media type registration.

Doesn't this support the idea of writing schemas without defining a dialect ($schema)?
We always want people to define a dialect IMHO.

I think your whole proposition rests on the idea that once we announce a "stable" version or publish a "fully fledged RFC", people will recognise and use that over older versions, and use implementations which implement it over existing other implementations. I think this is a false presumption, and the horse has already bolted/ship has already sailed. This is one of the drivers for dialects and vocabularies: drive forward adoption with the promise of standardised, reliable, interoperable, predicatable behaviour based on requirements.

[awwright]

RFCs can be superseded, but as a general rule, new RFCs cannot break the functionality of the document they are replacing. Observe how you can still deploy an HTTP/1.0 server coded against RFC 1945 and it will work perfectly well in a browser implementing RFC 7230.

Suppose we publish a specification with a new validation keyword, say, editDistance. It would be the case that an older tool would ignore the keyword and report a false positive, while a newer one would would correctly reject certain instances.

Whether or not this is technically breaking of reverse compatibility, I don't think very many people would expect this behavior.

[awwright]

Afaik, this point also applies to vocabularies. It seems like a good feature to allow individual vocabularies to evolve without needing a new vocabulary identifier.

[karenetheridge]

If you are concerned about new keywords being ignored when evaluated by older spec versions, you could use a version of the metaschema that basically did

{
  "$dynamicAnchor": "meta",
  "$ref": "https://metaschema-uri",
  "unevaluatedProperties": false,
}

Actually, it wouldn't be a bad idea at all if we published that ourselves, and gave it an official name like ".../strictschema".

[Relequestual]

Actually, it wouldn't be a bad idea at all if we published that ourselves, and gave it an official name like ".../strictschema".

100% agree. It's been on my mental to-do list for a while. I delayed because I wanted to see if there was any other "strict" things we could pick out from ajv's new "strict mode" in addition to preventing additional properties.

[Relequestual]

Afaik, this point also applies to vocabularies. It seems like a good feature to allow individual vocabularies to evolve without needing a new vocabulary identifier.

Can you explain why you think this would be a good idea? Surley having interoperable assurance is important, and not possible without defining what dialect (and therefore vocabulary) you require for the schema to be understood as intended?

Suppose we publish a specification with a new validation keyword, say, editDistance. It would be the case that an older tool would ignore the keyword and report a false positive, while a newer one would would correctly reject certain instances.

This would only be the case if the schema which uses that new keyword doesn't declare the dialect they are using.

I'd like to modify the specification so the a missing dialect defintion SHOULD cause an error to be thrown.

[jdesrosiers]

This section I wrote recently for UJS is relevant to this discussion. It's guidelines for using custom keywords and pitfalls to look out for. https://json-schema.org/understanding-json-schema/reference/schema.html#vocabularies

As for submitting to IETF. I strongly believe that the only keyword we should be defining for IETF is $schema. That value defers definition of all the other semantics to whatever that URI identifies. It should be a very simple affair. How we want to support forward or backward compatibility as we release new versions is orthogonal to IETF.

[awwright]

@handrews Correct, I didn't mean to imply that no validators will ever work this way, it's a really cool feature. What I mean is, I don't think it's realistic to expect all validators to work this way. Consider, for example, the validator running on an embedded platform, or in a database engine. There needs to be a way for these validators to gracefully fail (with an indeterminate result).

[karenetheridge]

Asking all implementations to support pluggable vocabularies and keywords might be too much, but it's not too much to require supporting the $schema keyword. For example, it's required for parsing OpenAPI documents, and OpenAPI is one of our major usecases of JSON Schema (which also adds an additional vocabulary on top of draft2020-12).

[handrews]

@awwright an implementation that intentionally understands only a fixed set is a different case entirely from an extensible implementation. A fixed set implementation will automatically fail anything it doesn't recognize, and that's fine (if it doesn't already, the spec should probably say something like "if you both don't recognize and can't access the meta-schema to figure out what vocabularies it requires, you SHOULD refuse to process the schema").

I don't think that case should constrain our general approach. I'm also not entirely convinced that such an implementation would not support pluggable vocabularies. in jschon the "built-in" keywords are just extensions that are automatically available. And in any event, a really constrained environment would probably be using CBOR or something.

[handrews]

I'm also going to dispute the idea that extensibility is something that most people won't care about or think about. Most people will use the $schema value that their API's documentation or their tech lead told them to use, so in that sense they won't be thinking about it much.
But it will be there. This has nothing to do with whether it runs inside a database engine - that's not a general purpose implementation! Of course it won't do arbitrary things. It's not running in that sort of environment.

But there is, and has been for years, a steady stream of people asking for new keywords (or new format values, which should be new keywords instead). A major reason to add vocabulary support was to be able to tell the vast majority of those people "no, that's not worth putting in the main spec, make a vocabulary." We should actually be seeing a lot of that new keyword stream offloaded to vocabularies.

I also think y'all are substantially underestimating the range of potential use cases, especially involving annotations. These use cases in particular would never fit in the main JSON Schema specifications.

If you look at JSON Schema as a thing that does what JSON Schema currently does, then I can see why you might consider this a nice-to-have. But you need to look at what could be done, quite easily, with elegant implementations that already exist and work great.

And from a social engineering perspective, I'm concerned about the impact of much of the JSON Schema team categorizing this stuff as inessential and basically encouraging people to not support it. Again, I'm not talking about constrained or super-application-specific implementations, but the general case. If we're like "hey we have a cool feature but most people won't need it", of course that's not going to get implemented, and that's a bit distressing.

[jdesrosiers]

I don't think it's true that most implementation don't or won't support meta-schemas. I can't even name one that I'm familiar with that doesn't. Even plugable vocabulary support is looking very common. I can name more implementations that support it than don't including mine, @karenetheridge's, and @gregsdennis's.

[jdesrosiers]

I don't think what you are proposing is actually forward compatibility. Here's a definition of forward compatibility from Wikipedia.

A standard supports forward compatibility if a product that complies with earlier versions can "gracefully" process input designed for later versions of the standard, ignoring new parts which it does not understand.

As soon as you introduce a new must-understand keyword, you've broken forward compatibility. The old implementation would throw an error given a schema from the new standard, not gracefully degrade by ignoring keywords it doesn't understand.

Forward compatibility doesn't mean the older implementation can never produce different results than the newer implementation, it just means that it must accept the newer schema and do the best it can.

As far as the specification is concerned, I think JSON Schema satisfies forward compatibility by ignoring unknown keywords. Beyond that, we would need to make a commitment to not making changes to keywords that aren't forward compatible.

From the UJS Vocabularies examples, we see that ignoring keywords you don't understand is more "graceful" is some cases (isEven) than it is in others (requiredProperties). In the requiredProperties example, the schema becomes almost completely useless when you ignore the requiredProperties keyword. This is where we would have to be careful about what keywords we add. With forward compatibility in mind, The requireAllExcept keyword is a more forward compatible alternative to requiredProperties because older implementations would only be missing required property assertions and not all the property schemas.

We would have to come up with criteria for what level of degradation we consider "graceful" when considering new keywords. I think that false positives are graceful while false negatives are not. In the UJS examples, we talk about the schemas being used on the front-end and on the back-end. If the validator on the front-end skips an assertion, it's not the end of the world because the error will be caught on the back-end. However, if you enter valid data and get a false negative on the front-end, that's a serious problem. It would block the user from proceeding.

It wouldn't be unreasonable for JSON Schema to make a commitment to forward and backward compatibility and drop version identifiers going forward like HTML. There would be some consequences that we would have to think through carefully and be ok with a bit of a change of direction.

I'm not a fan of the proposal, but I do think it's worth discussing whether or not we want to commit to forward and backward compatibility going forward.

[awwright]

I don't think what you are proposing is actually forward compatibility

This is a good point that I have anticipated :-]

Since there is only one correct behavior, we have to "gracefully degrade" by returning an indeterminate result. This is better than a false positive.

So yes, it's not "forward compatibility" in the strict sense (like how a computer program might fall back to an older API), but it's some kind of graceful degradation, and I think it's fair to call this "forward compatibility" so long as we're aware of the point you make.

[jdesrosiers]

I don't think saying that there is only one correct behavior is a useful argument. We could say the same about HTML. There is only one correct way to render an HTML document, but rendering as many tags as the implementation understands is more useful than rendering nothing. It's the same for JSON Schema. Validating 4 out of 5 keywords is more useful than validating nothing. I don't think this could be considered even a loose interpretation of forward compatibility.

But, setting aside the semantic argument about forward compatibility, I'm sure you're not asking for forward compatibility for forward compatibility's sake. I think what you want is for older implementations to be able to support schemas written for newer releases as long as the schema doesn't use any newer keywords it doesn't understand. Correct me if that's not right or I'm missing something.

Assuming I understand the goal correctly, the current system doesn't support that goal. The way things are now, a validator should refuse to process a schema written for 2020-12 if it only understands draft-07 even if the only keywords used are compatible with draft-07.

That's a perfectly reasonable goal for us to aim for and it's in line with something I've brought up in a couple different contexts. I proposed that every keyword should be identified by a URI that doesn't change from release to release unless it's semantics change. A vocabulary/dialect would be a map of keyword names to keyword URIs. An implementation would be able to support any future or custom dialect as long as it has support for the keywords used in the schema and reject the schema otherwise.

At one point we were looking into applying some of those ideas to the current vocabulary system, but that work has been on the back burner for a while. We're no where close to something like that being a viable proposal, but I think it's a better direction than a keyword naming convention.

[awwright]

We could say the same about HTML.

I don't believe so. HTML is specifically designed so it can be rendered different ways depending on the needs or desires of the user, or the media it's being formatted for. Even in ways not predicted by the author.

Validating 4 out of 5 keywords is more useful than validating nothing. I don't think this could be considered even a loose interpretation of forward compatibility.

I'm not proposing anything to the contrary.

I think what you want is for older implementations to be able to support schemas written for newer releases as long as the schema doesn't use any newer keywords it doesn't understand.

Yes.

That's a perfectly reasonable goal

Not for the whole spectrum of applications.

For internal usage, using a $schema identifier and even a custom meta-schema, and a library that supports these features, is perfectly reasonable. All of the work on this is reasonable.

However for widespread Internet usage, if I publish a schema as part of an evolving API, upgrading the $schema identifier is a big deal. Publishing a meta-schema that validators can read won't be feasible for quite a while, if ever. To support the widest variety of validators, I would have to determine what is the oldest meta-schema that supports all the functionality that I'm looking for.

Even the fact that I have to embed a $schema keyword is a big step for people. I personally like how XML and RDF uses URI namespaces, but it's just not popular.

[jdesrosiers]

To support the widest variety of validators, I would have to determine what is the oldest meta-schema that supports all the functionality that I'm looking for.

Actually, that won't necessarily help. Our releases are not backwards compatible. You can't validate a schema written for draft-04 using an implementation that only supports draft-07. Newer implementations usually don't bother to support older releases. That just further illustrates you point that pinning a schema to one dialect with no backward or forward compatibility guarantees is not good for evolution.

I personally like how XML and RDF uses URI namespaces, but it's just not popular.

If I understand you correctly, that's not far from what I was proposing.

[awwright]

Our releases are not backwards compatible

Partly, I was trying to simplify things for the sake of the argument, also I forgot this is technically possible.

that's not far from what I was proposing

Yeah. And again, I'm not against using namespaces in some capacity (it's the only way to do support 3rd party extensions).

But we should have a "default" keyword set that doesn't require any namespaces. {} should always mean the same thing, and {"type":"string"} should always mean the same thing, etc.

To put it this way, if I were standardizing application/xhtml+xml, I would define an implicit default namespace, which can be extended or overridden.

[jdesrosiers]

After some discussion, I think we are generally on the same page. The following are things that we might want to support in JSON Schema. The first two are not currently supported, while the second two are supported. The first step is to get agreement on which properties we want JSON Schema to have. Then we can entertain proposals to make it happen.

1. An implementation written for an older release should be able to process a schema written for a newer release if the only keywords used are compatible with the older release.
   Implications: Will require vocabulary system changes
2. An implementation written for a newer release should be able to process a schema written for an older release.
   Implications: Keywords can't be modified or removed once released, but they could be deprecated.
3. A schema should ignore keywords it doesn't understand resulting in a best-effort validation result.
4. A schema should be able declare which keywords must be understood in order to process the schema.

[awwright]

(1/2) Is it possible to combine these together? e.g. "The behavior of keywords must not change once defined (and/or adopted in the wild)".

(3/4) These make sense individually, but they seem contradictory. I opened this discussion thinking there should be three validation results: valid/invalid/indeterminate. Instead, perhaps "schema errors" should be a separate feature from "validation errors". This way, an application can use the "valid, determinate" result and also be confident that there cannot be a false positive, or see a "valid, indeterminate (1 unknown keyword error)" result and fail (or ignore) accordingly.

While JSON Schema is not supposed to be a general-purpose programming language, consider how defining keywords is similar to the function library of a programming language: There is a standard library of functions (validation keywords), and this library can grow over time, but the existing standard library will generally not change (unless there's a really good reason). Calling an undefined function is usually an error. Finally, you can import functionality, if the standard library is insufficient.

[karenetheridge]

"The behavior of keywords must not change once defined (and/or adopted in the wild)".

I could see possibly being able to make this assertion in a (future) non-draft specification publication. But keep in mind that much of the changes between drafts has not been in changing keyword semantics specifically, but things more wide-ranging to how the document itself is treated, e.g. what a dialect means, what a schema resource means, how reference resolution is intended to work -- and that's the trickier stuff that I don't think we've quite solidified into a "stable" state yet. (maybe we're close? but we seem to have a lot of things still being debated that may change yet again for draft-next.)

[awwright]

how the document itself is treated, e.g. what a dialect means, what a schema resource means, how reference resolution is intended to work

Yeah, I don't mind if we have to tweak how these sort of things work; especially if it's to align the behavior with another standard, or if different implementations already have diverging behavior.

[jdesrosiers]

(1/2) Is it possible to combine these together? e.g. "The behavior of keywords must not change once defined (and/or adopted in the wild)".

It would have to be "must not be changed or removed", but yeah committing to that constraint would give us the properties expressed in (1) and (2). I wrote it that way because I want us to be on the same page about what properties we want before we discuss what constraints we can impose to get those properties.

(3/4) These make sense individually, but they seem contradictory.

I don't think they are contradictory, but they do give schema authors two different paths they can choose to go down. This is an area where we need to flesh out more what we want. I think some situations call for a best effort validation (3) while others need to be more strict (4). You can even have both situations with the same schema use in different contexts. Your example of the validator responding with information about unknown keywords could be one solution. That information can be returned in the standard output format. Then the user has the information and can do with it what makes sense for the situation. They can choose to accept the result, or do something else if they don't trust it. But, in addition to that, it might still make sense for the schema author to be able to explicitly declare keywords as "must understand" and implementations would be expected to refuse to validate if they don't understand those keywords.

keep in mind that much of the changes between drafts has [been] things more wide-ranging to how the document itself is treated, [...] and that's the trickier stuff that I don't think we've quite solidified into a "stable" state yet.

I wasn't going to bring that up yet, but I agree 100%. We need to make sure we're really happy with the fundamental core workings before we say we aren't going to to change things anymore, only add and deprecate. I for one am still not quite happy with where some of the core keywords are at the moment.

[Relequestual]

I think I agree... I see a lot of frustration in the community in terms of wanting a "stable" version of JSON Schema.

I feel like there are a lot of things to think about here, and lots that needs to be defined to make such a thing viable.

I agree, we need to assess if this is really what the community want. I think it could be benficial.

We need a better handle on what things we are unahppy with in terms of the current spec that we feel need to be resolved before we can call it stable.

[Relequestual]

Far off consideration, but what do ya'll think about versioning here? If we released what we call a "stable" or forwards compatible specification, should we use semantic versioning for releases? If we want to send those signals, semver is a good way to make promises.

@jdesrosiers this seems to be opposed to your notion of a living standard. Have you been convinced otherwise or am I missing something?

[jdesrosiers]

If we released what we call a "stable" or forwards compatible specification

Don't forget backward compatible as well. That's even more important to a "stable" release than forwards compatibility.

should we use semantic versioning for releases?

I don't think semantic versioning makes sense in this case. If we commit to backwards compatibility, the major version number will never change. The patch version would only change for clarifications and fixing errors. So, the only thing that tells you anything about what version you are using is the minor version number. So, if it's just one number that has any real meaning, we might as well stick with dates.

this seems to be opposed to your notion of a living standard. Have you been convinced otherwise or am I missing something?

I'm not sure what you mean. Do you mean versioning? Versioning isn't incompatible with the concept of a living standard. EcmaScript is a living standard and they release new versions every year. That's the kind of model I think makes the most sense for JSON Schema. In my view, the properties we are discussing are practically required if we want to get serious about JSON Schema following a living standard model. So, I'm not sure what opposition you're referring to.

[karenetheridge]

I don't think we're at a place where we can promise no breaking changes, either forward or backwards. But we can make some commitments, which we can put in the spec so they're solid (some of these are new proposals, others are not):

• keywords starting with '$' are reserved for the specification itself and implementations should not seek to define their own (or conflicts may occur)
• we will never define keywords starting with 'x-', nor allow formal vocabularies to be published which define them: therefore this namespace is reserved for internal/private use and can contain anything the schema author wishes, within the constraints of their own application
• the '$schema' keyword will always be used going forward to describe the URI of the metaschema for the schema. (IMO we could improve the section of the specification that describes this keyword, but that's out of scope) - i.e. we won't change the name of this keyword
• the '$id' keyword will always be used going forward to define the canonical URI of the schema resource (either relative to a parent resource or absolute) - i.e. we won't change the name of this keyword

I think with this foundation, any implementation should be able to bootstrap any schema created now or using a future specification version, and either be able to process it or be able to generate a sane error that it can't process it (and why).

e.g. consider an implementation that only understands up to draft2020-12, and we've released 2022-06 which totally changes how the '$vocabulary' keyword works. It is given this schema to process:

{
  "$id": "http://acmecorp.com/my/application/schema",
  "$schema": "http://acmecorp.com/furble/metaschema",
  ... other keywords here...,
}

Because we've guaranteed the consistency of "$id" and "$schema", it knows how to get started: first it needs to determine the dialect that this schema is written in. It doesn't know this metaschema, but it is capable of loading schemas from the network, so it goes to fetch http://acmecorp.com/furble/metaschema, which is:

{
  "$id": "http://acmecorp.com/furble/metaschema",
  "$schema": "https://json-schema.org/release/2022-06",
  "$vocabulary": {
    ...,
  },
  ...
}

Here's another $schema URI that it doesn't know, so it has to load this one too. It is:

{
  "$id":  "https://json-schema.org/release/2022-06",
  "$schema": "https://json-schema.org/release/2022-06",
  "$vocabulary": {
    ...
  },
 ...
}

Now we've got a $schema keyword that is the same as the document we're already trying to process -- so are unable to process further, and must return an error. [Humans reading this will recognize that this means we are looking at a specification metaschema, but an implementation doesn't need to make this distinction -- it can just recognize that a schema has been defined in terms of itself, and therefore it can go no further.]

This is all possible because we've guaranteed the consistency and integrity of the "$id" and "$schema" keywords. If in a future version we'd (for example) renamed "$id" to "$identifier" or "$schema" to "$schema-uri", an older implementation might think there is no identifier or schema declared in the schema at all, and proceed with assuming sensible defaults, which might result in interesting and unpredictable errors as it tried to apply older logic to newer keywords.

[jdesrosiers]

I agree with all of this. We don't have to commit to no breaking changes all at once. These commitments would go a long way without limiting us where we aren't ready yet.