An alternate approach to meta-schemas

[jdesrosiers]

Since there's been some discussion about $recusriveAnchor/$recursiveRef lately, it might be a good time to share one of the experimental things I've tried out with Hyperjump Validation. It requires a big change to the concept of meta-schemas, but it eliminates the need for dynamic scope keywords and you basically get vocabulary support for free. It's not perfect and the syntax could certainly use some cleanup, but it has some pretty interesting properties.

In Hyperjump Validation, instead of having a meta-schema that describes a schema, it uses a separate meta-schema for each keyword.

Here's how it works... Keywords are identified by URLs. Each keyword the schema can use is declared using the $meta keyword. $meta works in a very similar way to @context in JSON-LD mapping a plain name to a URL.

{
  "$meta": {
    "type": "https://validation.hyperjump.io/common/type",
    "properties": "https://validation.hyperjump.io/common/properties"
  },
  "type": "object",
  "properties": {
    "foo": { "type": "string" }
  }
}

The URL given to a keyword should be dereferenceable to a schema that validates that keyword. Therefore, a validator can determine if a keyword is valid even if it doesn't know how to evaluate the constraint that keyword expresses.

Declaring keywords this way is like in any programming language where you would import a
dependency before you use it. Any keyword that appears in the schema that is not declared in $meta is an error. Many people have wanted to include "additionalProperties": false in the JSON Schema meta-schema. This constraint provides the protection people want from typos in keyword names without constraining extensibility.

Most people wouldn't want to have to declare every keyword they use this way, so most schemas will likely choose to link to a common $meta that declares all the keywords they use. This list of keywords is effectively a vocabulary. This example uses the standard Hyperjump Validation vocabulary.

{
  "$meta": { "$href": "https://validation.hyperjump.io/common" },
  "type": "object",
  "properties": {
    "foo": { "type": "string" }
  }
}

A vocabulary can be constructed by creating a new $meta document that declares all the keywords for the vocabulary. This could include new keywords mixed with standard keywords or keywords from other vocabularies. When constructing a vocabulary, you can change the name of keywords (like definitions to $defs) just by changing the name of the keyword in the $meta document. Another reason to change the name of something might be when combining two vocabularies that each have a keyword with the same name.

That's all great, but we still need something like the old meta-schema so keyword meta-schemas that include schemas (like properties) can declare that something is a schema. To solve this problem, I created a new keyword called validation (or schema in JSON Schema terms) that is a flag that indicates that the value should be a schema. It doesn't matter what vocabulary of the schema is. The schema will declare it's own vocabulary. This keyword only indicates that it's a schema of some kind.

{
  $meta: { $href: "https://validation.hyperjump.io/common" },
  validation: true
}

I don't imagine that anyone wants to make a change of this magnitude to JSON Schema, but hopefully it inspires people to look at the problem from a different angle and maybe come up with some new ideas that could be applied in a way that wouldn't completely change the way meta-schemas work.

[handrews]

@jdesrosiers I think there are good ideas in here that would fit with the plan to have a machine-readable vocabulary file, which would handle things like "is this a validation assertion", "is this an in-place applicator", etc. #602 talks about this, but it's kind of a rambling mess and I should re-file it.

This might not be as dramatic of a change, but might be a way to accomplish some of what you are talking about in way that fits a little more smoothly. Or by the time of the next draft (after this one), we might have more feedback on vocabularies etc. to justify a larger change. Thanks for filing this!

[gregsdennis]

I really like this. Nice job @jdesrosiers.

we still need something like the old meta-schema so keyword meta-schemas that include schemas (like properties) can declare that something is a schema. To solve this problem, I created a new keyword called validation

I don't think we need this (expanded upon below). When a meta-schema validates itself, it's not aware that this is what's going on. It's just validating an instance. It just so happens that the instance is the JSON representation of itself.

---

@handrews in line with your ideas of machine-readable vocabs, the dereferenced schema would need to contain additional data about the keyword. The properties that define this metadata would only have meaning when dereferenced within the context of the $meta keyword.

{
  "$keywordClass": "applicator",
  ... // more metadata
  ...  // schema definition of the keyword
}

Outside of the $meta context, these additional keywords could be ignored (hence not needing a specific validation keyword).

---

Lastly, just as with $vocab, it's important to note that just because the keyword can be semantically validated by an implementation does not mean that the implementation supports it. The implementation still needs to contain the logic for the intended operation of the keyword.

Also, @handrews, I appreciate your openness to this idea. I remember you had asked for other approaches before we published draft 2019-09, and the response was underwhelming. This, I think, counts toward that.

[handrews]

@gregsdennis my intention would be to use the URI from $vocabulary as the URI for the semantic description file, so I don't think a new $meta keyword would be needed in the meta-schema (or schema).

Caveat: I have not delved into this issue deeply enough due to other distractions, just skimmed it, so this might not be right, I just wanted to get a general "yes there's stuff worth considering" comment on the record ASAP.

[gregsdennis]

I see where you're going with that, but I hesitate to reuse a keyword, even if it's relatively new and lightly adopted. This is quite a shift in design, even though the purpose is similar. I'd prefer to make the hard break and use $meta (or whatever).

[handrews]

@gregsdennis OK I'll have to put this on hold because I'm 800% overloaded right now, am farther behind on the current draft than I was a week ago, and seem to be missing something.

But the explicit purpose of setting up $vocabulary the way it was set up was for it to eventually identify a machine-readable semantic, and possibly partially syntactic, description of the vocabulary. That has always been the point. My initial interest is seeing what from this proposal can fit with that long-standing intention.

We didn't create that semantic description file specifically in oder to see what people came back with after implementing, which @jdesrosiers has done.

I might end up sold on the more dramatic reworking, but where I'd like to start is lining up this proposal with the (very vaguely articulated) original direction and seeing if that produces a good balance between disruption (keep in mind we're putting draft 2020-0? out with OAS 3.1/4.0 so unlike 2019-09 it will immediately have a huge user base) and functionality.

[gregsdennis]

where I'd like to start is lining up this proposal with the (very vaguely articulated) original direction

That's a fair goal, but I think that this achieves the same end goals in a cleaner way.

I imagine we'll continue to discuss. Join back in when you can.

[handrews]

That's a fair goal, but I think that this achieves the same end goals in a cleaner way.

I agree that this could do that in the sense of not needing $recursive* specifically for meta-schemas, but while meta-schemas are the main rationale for those keywords, they are not the only recursive schemas. So I'm not sold that we would drop the keywords even if we did this (and I still don't see why we'd need to use $meta instead of $vocabulary).

If y'all think we can/should dump $recursive* entirely (and also decline $extends) and could do that in the next two weeks, then I'm interested in that discussion.

But this feels to me more like a solution that overlaps rather than entirely replaces.

[jdesrosiers]

@handrews

while meta-schemas are the main rationale for those keywords, they are not the only recursive schemas.

That's true. Moving to keyword-level meta-schemas doesn't make $recursive* completely redundant. But, other than meta-schemas, extending recursive schemas is extremely rare. It could be argued that it's rare enough that it isn't worth including. If it wasn't used for meta-schemas, it certainly wouldn't exist and we wouldn't be losing sleep over it.

If y'all think we can/should dump $recursive* entirely (and also decline $extends) and could do that in the next two weeks

I definitely don't think that's a reasonable timeline.

[jdesrosiers]

@gregsdennis
The validation keyword is my least favorite part of how this ended up working, but I couldn't come up with any way around it (or something like it). Let's look at the meta-schema for the properties keyword.

{
  "$meta": { "$href": "https://validation.hyperjump.io/common" },
  "type": "object",
  "patternProperties": {
    "": { "validation": true }
  }
}

If I don't use validation here, what can I do? Normally, we would use { "$ref": "#" }, but then # would be the keyword meta-schema, not a full schema.

[gregsdennis]

2019-09 uses $recursive* for that. The difference is that you're splitting on keyword, and the current spec is splitting on "vocabulary."

I guess that's the heart of the issue here. Maybe that can lead us to a more unified solution.

[thinking out loud] If we break down the vocabularies further to their constituent keywords, we're pretty close to your idea with $meta.

Each vocabulary meta-schema would merely be a collection of keyword meta-schemas, and they're all specified as required.

Current implementation of applicator meta-schema

{
  "$schema": "https://json-schema.org/draft/2019-09/schema",
  "$id": "https://json-schema.org/draft/2019-09/meta/applicator",
  "$vocabulary": {
    "https://json-schema.org/draft/2019-09/vocab/applicator": true
  },
  ...
}

The idea here is that https://json-schema.org/draft/2019-09/meta/applicator and https://json-schema.org/draft/2019-09/vocab/applicator are separate documents. The original concept was that the vocab one would eventually point to a differently-formatted document that a validator could use to assert some things about the keywords that need to be supported.

However, the meta-schema goes on to declare each keyword under properties.

I think these can be combined using the $meta idea.

Thinking out loud model

// applicator meta-schema
{
  "$schema": "https://json-schema.org/draft/2020-XX/schema",
  "$id": "https://json-schema.org/draft/2020-XX/meta/applicator",
  "$vocabulary": {
    "https://json-schema.org/draft/2020-XX/vocab/applicator/multipleOf": true,
    "https://json-schema.org/draft/2020-XX/vocab/applicator/maximum": true,
    "https://json-schema.org/draft/2020-XX/vocab/applicator/properties": true,
    ....
  },
  "$recursiveAnchor": true // [edit: added]
}

// properties meta-schema
{
  "$schema": "https://json-schema.org/draft/2020-XX/schema"
  "$id": "https://json-schema.org/draft/2020-XX/vocab/applicator/properties",
  "$vocabulary": {
    "https://json-schema.org/draft/2020-XX/vocab/keywordMeta": true,
    ... // do we need to list others?  core is included implicitly...
  },
  "type": "object",
  "additionalProperties": { "$recursiveRef": "#" }, // this
  "default": {},
  ...
}

This new document would also contain additional metadata about the keyword, whatever we need.

• keyword type
• annotation dependencies (e.g. what other keywords are required to be processed first)
• etc.

The key here is that $vocabulary now assumes a new "import" type role where it not only aggregates other vocabularies, but it also aggregates specific keywords, and each keyword is defined independently.

tl;dr This uses the ideas from $meta while expanding on the existing $vocabulary.

[gregsdennis]

It's worth mentioning that one thing that $vocabulary brings that $meta doesn't address is the ability to reference a vocab without it being required (by using a false value with the declaration).

[jdesrosiers]

@gregsdennis I've been really busy today and haven't had time to look at your last comment in detail, but I wanted to point this out ...

If I don't use validation here, what can I do? Normally, we would use { "$ref": "#" }, but then # would be the keyword meta-schema, not a full schema.

2019-09 uses $recursive* for that.

$recursiveRef doesn't work either. Where does the $recursiveAnchor go? Without a dialect level schema, there's nothing for the $recursiveRef to point to.

[gregsdennis]

$recursiveAnchor is still at the root schema and each meta-schema. (I'll edit the comment.) I don't see how the $recursive* keywords wouldn't work the same with this.

[handrews]

Regarding validation, my original plan (which may or may not be written down anywhere) involved specifying the classification of each keyword.

• assertion: this keyword can produce a false boolean result
• annotation: this keyword produces a result beyond the boolean result
• applicator: this keyword applies one or more subschemas, so a plugin would need to call back into the main validator implementation
  • in-place: this is important b/c keywords like unevaluated* require that they are run after all in-place applicators
  • child: not sure if it's useful to say how the subschemas are mapped to child instances?
  • by-reference: the value of this keyword needs to be resolved to get the actual subschema
  • inline: opposite of by-reference
  • automatic: $schema and contentSchema are weird applicators that are not automatically applied
• identifier: these keywords assign or modify (e.g. set base URIs) identifiers, and probably get processed in some way on schema load to build the URI-to-schema-object map
• reserved locations: $comment and $defs
  • schema location: the other thing (than applicator) that is relevant for finding schemas
  • non-schema location: everything else

Some of those very clearly map to either the behavior of the generic implementation or the behavior of the plugin. Others are not as clear and maybe are not necessary. One version of the idea allowed keywords to specify on which annotations they depended, and whether they depended on them from adjacent keywords only, or also from subschemas of adjacent in-place applicators. I decided that was probably complicating things.

This is just a brain dump so y'all can pick over the ideas and see if any are helpful.

[handrews]

Stepping back a bit, @awwright 's $extends leverages OO inheritance ideas, this proposal feels more like composition, and $recurse* is more of a "here's how we're using goto/setjmp/longjmp under the hood, please don't pay too much attention" sort of thing. $recurse* kind of tried to do an inheritance-ish thing without calling it inheritance, which is part of why it's confusing.

Philosophically, I think composition fits JSON Schema better than inheritance.

[gregsdennis]

Thanks for bringing the brain dump here, @handrews. That'll help.

[jdesrosiers]

@gregsdennis I'm finally getting around to reviewing how you have applied $meta to JSON Schema. Here's my questions/comments.

It looks like you've kept the concept of a the vocabulary-level meta-schema, which is why the $recursive* keywords still make sense. If there are still vocabulary-level meta-schemas, it seems there would then be two ways to define a meta-schema: at the vocabulary-level and at the keyword level. That's a bit complicated, but it could also be useful. For example, someone could create a vocabulary that uses the type keyword, but restricts it to not allow the array form using the dialect-level meta-schema.

I don't see a way to declare the keyword name for a keyword in your example. Did you mean to use the keyword-name => keyword-id mapping like in $meta?

Your example shows a vocabulary-level meta-schema, but I'm not sure what a dialect-level meta-schema would look like.

[gregsdennis]

Did you mean to use the keyword-name => keyword-id mapping like in $meta?

Yeah, that's the idea. The dialect schema (I think this means "validates an individual keyword") is this bit

// properties meta-schema
{
  "$schema": "https://json-schema.org/draft/2020-XX/schema"
  "$id": "https://json-schema.org/draft/2020-XX/vocab/applicator/properties",
  "$vocabulary": {
    "https://json-schema.org/draft/2020-XX/vocab/keywordMeta": true,
    ... // do we need to list others?  core is included implicitly...
  },
  "type": "object",
  "additionalProperties": { "$recursiveRef": "#" },
  "default": {},
  "keywordType": "applicator" // from keywordMeta vocab
}

This validates the concept of the properties keyword. But it's not called "properties" until included in the vocabulary schema. The $id supports the idea of calling it properties, but that's not a hard requirement; the vocabulary could call it $props.

Then multiple vocabularies can be combined to achieve a full meta-schema. Finally the meta-schema is used to create schemas.

[jdesrosiers]

@gregsdennis By dialect (@handrews's word) I mean the top level schema. Draft 2019-09 and OAS 3.1 are two examples of dialects. At the vocabulary-schema level, $vocabulary is used to declare keywords. At the dialect-schema level, it needs to declare vocabularies, right? How does that work.

[gregsdennis]

Ah... I see.

It would work exactly as it does now: by pulling in multiple vocabularies. The hierarchy is

• Meta-schemas use $vocabulary to collect vocabularies.
• Vocabularies use $vocabulary to collect individual keywords.
• Keywords don't strictly need a $vocabulary (Core is implicit), but we'd probably want a keyword metadata vocabulary.

Each is represented by its own file, but the same format for all.

You'd need the keyword metadata vocabulary to validate a schema because you need to validate keywords, but you shouldn't need it when validating an instance.

---

With this setup, you could rename all of the keywords, and a validator should be completely fine with it because the keywords should be recognized by their $id rather than their name. (E.g. https://json-schema.org/draft/2020-XX/vocab/applicator/properties is the keyword, not properties.) The "keyword" (e.g. properties) becomes merely an alias, defined by the vocabulary.

NOTE This would require a fairly hefty rewrite of my library, so I don't propose it lightly.

[jdesrosiers]

Did you mean to use the keyword-name => keyword-id mapping like in $meta?

Yeah, that's the idea.

I'm not sure what a dialect-level meta-schema would look like.

It would work exactly as it does now

Each is represented by its own file, but the same format for all.

These seem like contradictory statements to me.

How can $vocabulary use keyword-name => keyword-id mapping and work the same way it does now?

If it uses keyword-name => keyword-id mapping, how do you do that on the dialect level where there are no keywords, just vocabularies?

If it works as it does now, how are keyword names assigned to keyword-ids?

An example of a meta-schema from each level (dialect, vocabulary, keyword) would probably be the best way to clear this up.

[gregsdennis]

I see what you mean. I'll work up a top-down example.

[gregsdennis]

This is the best I could come with. I had some other options, but they all lacked something or duplicated/muddled ideas.

Meta-schema
{
  "$schema": "https://json-schema.org/draft/2020-XX/schema",
  "$id": "https://json-schema.org/draft/2020-XX/schema",
  "$vocabulary": {
    "https://json-schema.org/draft/2020-XX/vocab/applicator": true,
    ...
  },
  "$recursiveAnchor": true,
  ...
}

Vocabulary
{
  "$schema": "https://json-schema.org/draft/2020-XX/schema",
  "$id": "https://json-schema.org/draft/2020-XX/vocab/applicator",
  "$vocabulary": {
    "https://json-schema.org/draft/2020-XX/vocab/keywordMeta": true,
    ...
  },
  "$keywords": {  // defined in keywordMeta  (a bit more explicit than "$meta")
    // define aliases here
    "multipleOf": "https://json-schema.org/draft/2020-XX/vocab/applicator/multipleOf",
    "maximum": "https://json-schema.org/draft/2020-XX/vocab/applicator/maximum",
    "properties": "https://json-schema.org/draft/2020-XX/vocab/applicator/properties",
    ...
  }
}

Keyword
{
  "$schema": "https://json-schema.org/draft/2020-XX/schema"
  "$id": "https://json-schema.org/draft/2020-XX/vocab/applicator/properties",
  "$vocabulary": {
    "https://json-schema.org/draft/2020-XX/vocab/keywordMeta": true,
    ...
  },
  "type": "object",
  "additionalProperties": { "$recursiveRef": "#" },
  "default": {},
  "$keywordClass": "applicator",  // defined in keywordMeta
  ...
}

KeywordMeta
{
  "$schema": "https://json-schema.org/draft/2020-XX/schema",
  "$id": "https://json-schema.org/draft/2020-XX/vocab/keywordMeta",
  "$vocabulary": {
    "https://json-schema.org/draft/2020-XX/vocab/keywordMeta": true,
    ...
  },
  "$keywords": {
    "keywords": "https://json-schema.org/draft/2020-XX/vocab/keywordMeta/keywords",
    "keywordClass": "https://json-schema.org/draft/2020-XX/vocab/keywordMeta/keywordClass",
    ...
  }
}

This solution defines a new vocabulary: keywordMeta. This vocabulary would contain all of the keyword metadata keywords (e.g. $keywordClass to identify "applicator", "assertion", etc.) as well as a keyword to be used to express a collection of keywords (i.e. $keywords).

The other thing this does it make the vocab URIs deferenceable. The URI points to a resource that is itself a meta-schema that gathers keywords.

An interesting side effect from this is that meta-schema authors can rename keywords. This is especially helpful when they want to reference two vocabularies with the same keyword but different meanings for them.

As I stated before, an implementation wouldn't want to identify keywords based on the keyword name with this, which is what mine does (and probably most others). Instead it would need to use the ID for the keyword-meta-schema to identify whether it supports that keyword.

[Relequestual]

Having only skimmed the issue, I'm keen to explore this... after draft-8-patch-1

[jdesrosiers]

@gregsdennis I like it. Allow me to iterate a bit.

Meta-schema

No changes here, but I'll point out that even tho the keywords are providing their own meta-schemas, additional constraints could be added here. For example, assume you want a standard JSON Schema, but you want to forbid the array form of type. You can add your constraint in the meta-schema. Otherwise, the meta-schema just declares vocabularies. This is similar to what you do in 2019-09 except with one important difference. In 2019-09 you replace your custom meta-schema with the standard one. In this version, you can't replace the existing one, but you can add another meta-schema to further constrain a keyword. I like this approach better.

{
  "$schema": "https://json-schema.org/draft/2020-XX/schema",
  "$id": "https://json-schema.org/draft/2020-XX/schema",
  "$vocabulary": {
    "https://json-schema.org/draft/2020-XX/vocab/applicator": true,
    ...
  },
  "$recursiveAnchor": true
}

Vocabulary

This one doesn't quite work. The $vocabulary keyword is only taken into account at the top-level meta-schema, so declaring the keywordMeta vocabulary doesn't have any effect here. I suggest this not be a full schema, but rather something else that describes a vocabulary. It's not a meta-schema, so $vocabulary doesn't belong.

{
  "$schema": "https://json-schema.org/draft/2020-XX/vocabulary",
  "$id": "https://json-schema.org/draft/2020-XX/vocab/applicator",
  "$keywords": {
    "multipleOf": "https://json-schema.org/draft/2020-XX/vocab/applicator/multipleOf",
    "maximum": "https://json-schema.org/draft/2020-XX/vocab/applicator/maximum",
    "properties": "https://json-schema.org/draft/2020-XX/vocab/applicator/properties",
    ...
  }
}

Keyword

Again, I don't think $vocabulary is right here. This creates a new kind of schema that's just a standard schema with the keywordMeta vocabulary added. However, it would probably be better to include the keywordClass data in the vocabulary schema some how instead and make this just a standard schema.

{
  "$schema": "https://json-schema.org/draft/2020-XX/meta-schema",
  "$id": "https://json-schema.org/draft/2020-XX/vocab/applicator/properties",
  "type": "object",
  "additionalProperties": { "$recursiveRef": "#" },
  "default": {},
  "$keywordClass": "applicator", 
  ...
}

The problem here is with the $recursiveRef. This schema is never $refed, so it doesn't have any dynamic scope to inherit. It would have to be be defined how that works. That shouldn't be too hard I think, but the complexity seems to be growing.

[gregsdennis]

Vocabulary - I suggest this not be a full schema, but rather something else that describes a vocabulary.

Keyword - I don't think $vocabulary is right here.

I think the idea I had going was that it was schema objects all the way down. That way we don't have to define a new document type. Additionally $vocabulary can be at the root of any schema. In 2019-09 it's only meaningful in meta-schemas, but as working on the next draft has shown, these things are somewhat fluid.

@handrews also conceived of defining a new document type for vocabs, but I rather like the idea of having the same document type describe all levels.

The problem here is with the $recursiveRef.

The reference goes back to the meta-schema root. I would say that if the target can't be found, the validator should fail (like if resolving $ref fails). This would prevent someone from using the vocab- or keyword-level meta-schemas independently, which is likely desired. (It also forces any meta-schema to define the anchor.)

[ldexterldesign]

A vocabulary can be constructed by creating a new $meta document that declares all the keywords for the vocabulary. This could include new keywords mixed with standard keywords or keywords from other vocabularies. When constructing a vocabulary, you can change the name of keywords (like definitions to $defs) just by changing the name of the keyword in the $meta document. Another reason to change the name of something might be when combining two vocabularies that each have a keyword with the same name.

I discovered this recently here (sorry, commit isn't as atomic as it should be) because of (possible) conflicts with description, specifically keeping description-1 (i.e. annotation keyword which I believe can be used anywhere) from json-schema.org and replacing description-2 (i.e. vocabulary property, which can be used anywhere) with disambiguatingDescription from schema.org

Hope this is relevant/useful

Sincerely