Proposal: New Spec Development Lifecycle Model

[jdesrosiers]

At this point we seem to have near unanimous agreement that we want to remove IETF from our process (at least for now). This gives us the opportunity to make changes to our process. The following is a proposal for a Specification Development Lifecycle (SDLC) process that I think will be a good fit for JSON Schema. This process is inspired by the TC39 Process used by ECMAScript (blog) with additional inspiration from the trunk-based development paradigm.

Spec documents would remain in essentially the same format they are now, but would be converted from XML to Github flavored markdown and be hosted on the json-schema.org website. This will be a "stable" spec but will include features that are flagged as unstable. In this case, "stable" means no backward/forward incompatible changes. The spec will evolve by modification rather than producing a new spec every release. This is a reasonable approach only because we will be committing to strict compatibility requirements for stable features and clearly flagging unstable features and flagging stable features with the release they reached stability.

Feature Status Flags

Every feature in the spec has a release status. It's either stable, stage-1, stage-2, or deprecated. Flags are used to show the status a feature is in. If a feature doesn't have a flag, it's considered stable. If it has a year flag (such as 2023) it means the feature reached stability with the 2023 release. The other flags indicate a feature that is not yet stable or is deprecated.

• STAGE-0 - Consensus has been reached that a keyword is worth pursuing as a standard dialect keyword. A champion is identified with the responsibility of ticking the boxes for STAGE-1 eligibility.
• STAGE-1 - This feature is new and may change or be removed all together. Implementers are encouraged to implement these features, but are not expected to maintain support for previous versions when they change. Users who choose to use these features should be comfortable being on the cutting edge and with everything that comes with it.
• STAGE-2 - This feature is in the last stages of becoming stable. We don't expect any changes, but backward incompatible changes are still possible. At this point, users can be reasonably sure that the feature won't change, but can't expect all implementations to support the feature.
• {YEAR} - This feature is stable as of the year specified. Stable features can not be changed or removed, but may be deprecated.
• DEPRECATED-{YEAR} - This feature was deprecated as of the year specified. Implementations should still support this feature, but new schemas should not use them. They are retained for backwards compatibility only. Implementations may choose to drop support for deprecated features, but if they do, they must raise an error if they encounter a schema with a deprecated keyword. They must also clearly document that they don't support JSON Schema releases before the keyword was deprecated.

Release Process

Effective Jan 1 each year, we would update feature status flags to promote features to stable or STAGE-2. Implementations that express support for the 2023 release must support all features that are stable as of the 2023 release including previous releases. A blog post should be published describing the status changes for the release. A snapshot of the spec will be taken of the stable parts of the spec and made available.

Development Process

Bug fixes, clarifications, and other non-functional spec updates can be merged at any time even for stable features. Changes to STAGE-1 and STAGE-2 features can also be merged at any time. A change-log blog post should be prepared quarterly to give visibility to any changes that were merged in the last quarter.

STAGE-0 Criteria

STAGE-0 features are not added to the spec. They will be maintained through some other channel.

• General consensus among the core contributors for pursuing the feature
• The feature has a designated champion that will take responsibility to ensure the feature is progressing through the stages.

STAGE-1 Criteria

STAGE-1 features can be added at any time as long as it meets all criteria for STAGE-1.

• General consensus among the core contributors for adding the feature to the spec
• A PR for the spec that completely specifies the feature and includes the STAGE-1 flag
• Tests are available in the test suite
• Two implementations have implemented it and pass the tests suite
• A blog post is prepared introducing the new feature

STAGE-2 Criteria

A STAGE-1 feature can only be promoted to STAGE-2 on Jan 1. Generally a feature should stay in STAGE-2 for one year, but may stay longer if we don't see it used enough in the wild.

• There is general consensus that the feature has been proven to be a good addition to JSON Schema and is unlikely to change.
• We see the feature being used successfully in the wild and not generating a lot of community support issues that could indicate a problem.

Stable Criteria

A STAGE-2 feature can only be promoted to stable on Jan 1. Once stable, the status can not be changed back to an unstable status, but may at some point be changed to DEPRECATED.

• The feature hasn't changed for a significant amount of time (recommended is at least a year)
• There is general consensus that we don't expect the feature to change
• The feature is being used successfully in the wild

Backward Compatibility

We would commit to no backward incompatible changes on stable features. Most of the spec is pretty stable. It's mostly new concepts like the vocabulary system, annotations, output format, and dynamic references that are likely to change. Those features and anything else we think might not be stable can be flagged as STAGE-1 or STAGE-2 to start with.

Forward Compatibility

We would commit to not adding/modifying features that could be problematic for forward compatibility.

Forward compatibility can be an issue for unknown and unstable keywords, so if they are supported, they would need to be disabled by default. Users can opt-in to enabling these features with the understanding that they will lose forward compatibility guarantees.

If an implementation encounters a dialect it doesn't support, it should raise an error. If no dialect identifier is provided, the standard validation dialect should be used.

If an implementation encounters a keyword it doesn't support, it must raise an error.

Meta-Schemas

Because there is only one version of the spec, there would be only be one set of meta-schemas. We wouldn't publish a new set for each release, just update the existing ones as necessary. There would be one URI (to be determined later) we would use as the $id for the current standard dialect meta-schema. Because of backward and forward compatibility rules, all schemas written for a previous release or for a future release should work with whatever version of the meta-schema an implementation might be using.

Vocabulary System

The core vocabulary would no longer be an indicator of the version of JSON Schema because there is only one version and that version is always assumed. The vocabulary system would initially be considered unstable so any other changes we might want to make in this area can be decided later, even after the initial release under this model.

External Standardization

If we get to a point where there's nothing left in STAGE-1 or STAGE-2 and we don't foresee any major changes coming up, we can consider standardization with a recognized standards body such as IETF or W3C.

Media Type Registration

Although IETF wouldn't be part of the main spec development, we still want to register our media types with IANA and we would do that through IETF. The media type registration would follow the IETF process, not this one. That means that it should be stable and we should not expect frequent updates like with the main spec. Our new process gives us a stable spec that the media type registration can link to, but must also be compatible with the older "draft" releases. It would be best to get this done as soon as possible because the media types are already being used in the wild.

---

EDIT 09/06 - Added media type registration and clarified that we would not be issuing new documents for each release the way we do now.

EDIT 09/09 - Added "Stable Criteria" section. Moved section on dialect to support to the forward compatibility section and fixed an error that seemed to imply that vocabularies aren't a thing.

EDIT 09/22 - Update forward compatibility rules to make unknown and unstable keywords disabled by default. Rewrote "$schema" section and added "Meta-schemas" section based on what was made possible by forward compatibility changes.

EDIT 09/25 - Remove assuming the standard validation dialect by default. Reworded some things about dialects in attempt to address misunderstandings that have come up.

EDIT 09/28 - Add designation of a champion to STAGE-1. Remove suggested URI for current standard dialect meta-schema.

EDIT 10/12 - Add STAGE-0. Mention snapshots.

[jdesrosiers]

I know this is a massive paradigm shift for us, but I think it's worth the effort to get us to something like this. I think the proposal will: (1) provide a stable version of JSON Schema without hindering our ability to evolve the spec, (2) eliminate the burden of implementers having to support multiple versions, (3) allow us to get clarifications and bug fixes out to people when they happen, (4) allow us to iterate on new/unstable features more quickly while also putting less burden on implementers and getting features to a stable state sooner.

Other than process changes and changes already identified, we would need to determine which features are not stable, but JSON Schema wouldn't be practically usable without them. We would have to get any such features to a stable place before we could do our first release with the new process. Based on recent discussion, I think $id and $schema are the main things we want to make sure we're happy with.

Ideally, it would be great to have our first release under this model out for the Jan 1, 2023 release. That's only four months away, so that might be a tough goal to hit. If we aren't ready for Jan 1, 2023, I think we should just get it out ASAP and not wait for Jan 1, 2024 for the first release.

[jdesrosiers]

If we move to this model, I'm going to fork my implementation and remove all the baggage from previous (and future!) releases. It should be fairly trivial to provide a compatibility mode for draft-06, draft-07, and 2020-12 with minimal bloat to the implementation.

By compatibility mode, I mean that any schema written for draft-07 should evaluate correctly, but certain edge cases wouldn't necessarily pass the test suite. For example, $ref would allow siblings, but since there was no reason anyone would have a sibling keyword to $ref in draft-07 and expect it to do anything, it shouldn't be a problem. It wouldn't be safe to use the compatibility mode to develop a new schema (unless you really know what you're doing), but it should evaluate all your real-life, non-pathological, non-contrived, legacy draft-07 schemas.

[gregsdennis]

Should we (can we) recommend libary architecture consisting of a stable base library with stage1 and stage2 extensions? This would help with versioning, I think. (It would for me anyway. I'm not sure how libraries work in other languages. Maybe it's just a me thing.) Changing a Stage X feature would only bump version of the associated extension lib. This would help the "stable" lib appear stable.

If we move to this model, I'd probably also major-rev my library to indicate the breaking nature of the shift in schema-version paradigm.

Another option is to move to a non-semver versioning (e.g. year-based that is sync to the spec).

[jdesrosiers]

I'm always against recommending any implementation details. However, I'd be fine with you writing a blog post describing what you do and personally (not officially) encouraging others to do something similar.

However, I'm not sure it would always be possible for all unstable features to be implemented as extensions. It certainly makes sense for keywords, but I expect things like vocabularies and annotations to be initially flagged as unstable. I expect those things would be much more difficult to implement as annotations.

As far as versioning goes, I'll probably increment the minor version for any change to an unstable feature even if it's a non-backwards compatible change. Those features are specifically not guaranteed to be stable anyway, so it could be argued that it's ok.

[gregsdennis]

I think an opinion post on how I manage stages is a good idea.

[gregsdennis]

Stage 1

Tests are available in the test suite

Two implementations have implemented it and pass the tests suite

Do we have a plan for coordinating this since they are different repos?

It also sounds like we're requiring implementations to have some support for pre-release features. I've looked at doing this in my implementation in the past, and it comes with some complications that I won't get into now. Suffice it to say that doing so is hard.

Stage 2

if we don't see it used enough in the wild

Do we have a mechanism for identifying usages of features? I know I can't get any kind of metrics out of my library.

Forward compatibility

STAGE-1 and STAGE-2 features would also only be allowed if the $schema keyword is used.

$schema on its own might not be enough. If a schema author expects stage X features, how can the consumer of that schema know this? If the author only expect stable keywords, they can use only stable keywords. But if the author uses Stage X keywords and expects them to be used, how can the consumer of the schema know this?

Should the consumer have the right to disable Stage X features, even if a schema contains them?

Other

Are we going to retroactively apply this to pre-existing releases?

How does this affect MAY/SHOULD/MUST requirements? I don't think that it's practical for stable to imply MUST. E.g. there will inevitably be implementation-defined behaviors in stable.

[gregsdennis]

Do we have a plan for coordinating this since they are different repos?

I don't think coordination should be difficult. I think it's reasonable to not worry about this until we experience pain points and make adjustments to address the problems we experience.

So the sequence would need to be:

1. Discuss and design the feature.
2. Write some tests (requires a place to put the tests, I expect currentYear/stage1). (But if it requires implementations, is it really stage 1?)
3. Get a couple implementations to volunteer the feature, which presupposes that implementations that want to do this have some kind of "experimental feature" feature so that it doesn't interfere with daily operation.

Personally, on point 3, I foresee this creating a lot of version churn which could indicate to users that the implementation is unstable. That's not an impression I want to give. (This is one reason for suggesting extension libs.)

if the author uses Stage X keywords and expects them to be used, how can the consumer of the schema know this?

It's only safe to use unstable features in a closed system where you know what implementation(s) will be used to evaluate the schema.

You and I know this, but users won't. They'll see a keyword and want to use it. Then they'll publish the schema and wonder why so many people are submitting data that doesn't meet the requirements because the consumers' validators don't support the feature. Yeah, it's on them for using an unstable feature, but I expect we're going to get a lot of questions.

Should the consumer have the right to disable Stage X features, even if a schema contains them?

Yes. Unstable features are never required to be implemented. There are effectively no guarantees what you're going to get when you use an unstable feature. You have to rely on the implementation's document to know.

My thoughts on this were around requirements on implementations. If the consumer has the right to disable unstable features, the implementation needs a setting to do that. Additionally, we need to state whether they on on or off by default (probably off, but it should be stated).

[jdesrosiers]

So the sequence would need to be:

I would expect a step between 1 and 2 that is to have an approved PR for adding the STAGE-1 feature to the spec. And the last step would be preparing a blog post introducing the feature.

which presupposes that implementations that want to do this have some kind of "experimental feature" feature so that it doesn't interfere with daily operation.

New features shouldn't interfere with daily operations. At this point we've committed to no backward/forward incompatible changes. New features should only effect users if they use them. If we find that we need a flag to enable/disable new features, it means we've broken compatibility somewhere and the feature needs to go back to the drawing board or abandoned.

I foresee this creating a lot of version churn which could indicate to users that the implementation is unstable.

Personally, I see libraries with frequent updates as a good sign that the project is active and evolving, but I think there's ways to deal with this for those who might me concerned. For example, you might have a version 2023, that doesn't include unstable features and a version 2023-beta that includes unstable features.

They'll see a keyword and want to use it.

Mostly I expect they'll see the keyword either in the spec, documentation, or blog post in which case it would be clearly marked as unstable. But, it's possible that people might see the keyword in someone else's schema and try to use it for themselves not knowing it isn't stable. Hopefully at some point they look up the documentation for that keyword and see that it's unstable.

If the consumer has the right to disable unstable features, the implementation needs a setting to do that.

Implementations could do that. Implementations are allowed to provide whatever level of support they want for unstable features. They can choose not to implement them, they can choose to implement them always on, or they can choose to implement them but require a config to enable them. It's implementer's choice.

Additionally, we need to state whether they on on or off by default (probably off, but it should be stated).

My thought was that there should be no harm in them being on by default because compatibility commitments mean there should be no harm in doing so. However, given your point about people trying to use unstable features without realizing that they are unstable is a reason to make them off by default. Making users set a useUnstable config to turn unstable feature support on is another signal to users that they are using an unstable feature.

It should be noted, however, that not all features are easily turned off. For example, I expect the output format to initially be STAGE-1. Turning off the output results doesn't make much sense. The users need some kind of feedback even if it's unstable. Maybe the rule could be that unstable features SHOULD be turned off by default. That way there's room for leaving some things turned on by default in the cases where it makes sense.

[gregsdennis]

Implementations could do that. Implementations are allowed to provide whatever level of support they want for unstable features. They can choose not to implement them, they can choose to implement them always on, or they can choose to implement them but require a config to enable them. It's implementer's choice.

If the implementor has the choice, then the user can't have the right to disable unstable features. (Consider if the implementor decides to have them always on and doesn't provide a way to disable them.)

I'm not sure this would have an impact except where a schema consumer is separate from the schema author where the author has decided to use unstable features that the consumer's implementation happens to support but the consumer doesn't want to use them. It's an edge case, but I think it's worth considering.

I expect the output format to initially be STAGE-1. Turning off the output results doesn't make much sense.

I agree with this. I wouldn't be able to turn off output or revert to 2019-09/2020-12 output in my implementation. Being strongly typed, the structure is defined by the models, and the current way and the new way are incompatible.

[jdesrosiers]

where the author has decided to use unstable features that the consumer's implementation happens to support but the consumer doesn't want to use them.

So the consumer of the schema wants to override the intention of the schema author? Technically, it would be allowed since there are no guarantees when using unstable features, but that sounds like a bad enough idea that there's no reason to specifically enable it.

[jdesrosiers]

where the author has decided to use unstable features that the consumer's implementation happens to support but the consumer doesn't want to use them.

So the consumer of the schema wants to override the intention of the schema author?

@gregsdennis I realized why you're right that this should be a concern of schema consumers. When using an unstable keyword, a consumer can't be sure that the implementation they are using will give the same results as the implementation the schema author is using for development. The feature may have changed and one implementation may have not included the change while the other has. So, even if the unstable keyword is enabled, the consumer can't be sure they will get the behavior the schema author intended. If the consumer doesn't want to take this, they may choose to refuse to process schemas with unstable keywords.

[gregsdennis]

cc: @Julian

I would like to propose the following structure for the test suite:

- {year}
  - stage1
    - feature1.json
  - stage2
    - feature2.json
  - stable
    - feature3.json

Given that features progress through the stages, it makes sense to copy the latest year folder in its entirety for the next year (as we already do between drafts) and move tests around accordingly. This would allow implementors to retroactively test compliance. ("When was I last compliant?")

I'm not sure how this fits in with the current effort of reorganizing the folders into the MAY/SHOULD/MUST requirement levels.

[gregsdennis]

Like I said, it will be moved from the "stage-2" folder to the "2025" folder.

This is my point. It seems important that there be some sort of history that it was stage 2 in 2024 (someone will care about that). We can't have that information if we move the tests around. I'm less worried about implementors finding the tests and more about how they can report/document what they support.

Secondly, since the features are still unstable, the tests are expected to change over time. Do we guarantee that they won't change when moving between stages?

[jdesrosiers]

since the features are still unstable, the tests are expected to change over time. Do we guarantee that they won't change when moving between stages?

No. They will change as the feature evolves until it stabilizes. Just like implementations shouldn't keep around old versions of unstable features, the test suite shouldn't keep around old versions of tests for unstable features. If the feature changes, the test suite changes, and implementations should fail the test suite until they update to support the changes.

It seems important that there be some sort of history that it was stage 2 in 2024

I'm not entirely sure why this is important. I think there is probably some misunderstanding. I answered the other question first because I think it might inform this one. There's no such thing as the 2023 version of propertyDependencies vs the 2024 version of propertyDependencies. Any change completely replaces the previous version. It shouldn't matter what stage it was in what release. The second structure I proposed does preserve that history, but only as a convenience.

[gregsdennis]

Here's my scenario:

It's 2024, and I've been claiming that I'm compliant with 2023 + stage 2. But inevitably some features that were in 2023 stage 2 have been moved to 2024, perhaps even changed subtly in that move. How can I prove my compliance if the test suite no longer represents that state?

[jdesrosiers]

features that were in 2023 stage 2 have been moved to 2024, perhaps even changed subtly in that move.

That's not allowed. We can't change a feature and move it to stable in one step. If we make a change, then it's not stable and needs to stay in stage-2 and demonstrate that it's not going to change again. Moving a feature to stable is just a commitment to not make incompatible changes. It has the same behavior and tests that it did in stage-2 and hasn't changed for a significant period of time.

I've been claiming that I'm compliant with 2023 + stage 2

Claiming compliance with 2023 + stage 2 isn't a claim that makes sense in this model. Unstable features progress independently from stable releases. There's no such thing as a 2023 stage-2 feature. It's just a stage-2 feature independent of release. An implementation would claim support for certain features independent of what stage of standardization they are in.

Let's assume dynamic references are a stage-2 feature in 2023 and become stable in 2024. A 2023 implementation would claim support for 2023 + dynamic references. Before 2024, dynamic references might change. This is possible but unlikely if it's in stage-2, but let's say that it does. The spec would change, the tests would be updated, and the implementation would fail the tests until they update to support the changes. Remember that changes to the spec can happen at any time. Releases aren't when changes are released, it's just when we update stability flags. When 2024 comes around and dynamic references move to stable, nothing changes about the feature other than our commitments and what directory the test file is in.

Let's say that dynamic references stay in stage-2 in 2024 and change during that time. Your 2023 + dynamic references implementation is out of compliance if it doesn't make the changes that were added after Jan 1 2024. Those changes are not scoped to the 2024 release. If you claim support for dynamic references, you are responsible for updating that feature as far into the future as it takes to be stable. If it takes til 2028 to be stable, you need to include all of those changes even if you never update your implementation beyond 2023. This is a practically impossible scenario for a stage-2 feature, but could happen for a stage-1 feature.

[gregsdennis]

I think this aspect of the paradigm shift is going to be the hardest for implementors to adopt (as illustrated by my questions). I'm still trying to figure out how I'd realistically support this. My main concern (as mentioned in another thread) is giving the appearance of stability.

I'm open to giving this a go.

[handrews]

@jdesrosiers It's great to see a comprehensive proposal! I have not had time to catch up to this beyond a quick skim, so it will be a while before I have substantive comments. But I do want to note:

At this point we seem to have near unanimous agreement that we want to remove IETF from our process (at least for now).

Given that not everyone reading these discussions has the full context across all of the repos involved, especially those not under the JSON Schema Org, would you be willing to change this to note that the media type registration is continuing with the IETF? I think it's important that folks be aware of that ongoing connection when reading these proposals.

[jdesrosiers]

Thanks for the feedback. I've updated the proposal.

[jdesrosiers]

It turns out that the OpenTelemetry specification uses a process of mixed stable and unstable components similar to this proposal. https://opentelemetry.io/docs/reference/specification/versioning-and-stability/

[handrews]

I'm still working on digesting the whole proposal and responding to it holistically, but for this:

Contrary to today, best practice would be to not declare a dialect. Because of the backwards/forwards compatibility rules, past releases are automatically supported and future releases are automatically supported as long as you don't use new keywords. The only time you would want to use $schema is when using a custom dialect, unstable keywords, or unknown custom keywords.

I think it would make more sense to say something like (changes in bold italics):

Contrary to today, dialects would no longer be used to select a version of JSON Schema. Because of the backwards/forwards compatibility rules, past releases are automatically supported and future releases are automatically supported as long as you don't use new keywords. The only time you would want to use $schema is when using a custom dialect, unstable keywords, or unknown custom keywords.

[jdesrosiers]

First of all, I don't want to worry about wordsmithing the proposal. As long as people get the gist, it's doing it's job. A formal document will have to be written up later that's more precise. However, if something isn't clear, we can definitely clarify.

I think your edit says something important, but I don't think it replaces what was being said.
It's correct that under this model, dialects would no longer be responsible for selecting a version because there's only one version going forward. But, that's not what I was trying to say in that paragraph.

Although dialects wouldn't be used the select a version of JSON Schema, they could still be used to restrict usage to what was available at a specific release. If someone uses "$schema": "https://json-schema.org/2023/validation", they are declaring a dialect of the one JSON Schema version that includes only what was stable in the 2023 release (and possibly some unstable features). This paragraph is saying that we can move away from $schema being effectively required, to only being needed in special cases.

[handrews]

I see the distinction you're making. There's something here that I want to get at regarding the conflation of "dialect" and "version" but I think that's best addressed elsewhere.

[jdesrosiers]

I've been rethinking part of this proposal. Currently the forward-compatibility rules say that if you declare a dialect, you can use unknown keywords as annotations and unstable keywords are enabled. If you don't declare a dialect, these features are disabled. Although this works, it complicates things for implementations and is an awkward rule for users to remember.

So, here's what I was thinking instead. Unknown keywords are never allowed and unstable keywords are off by default and must be enabled by configuration provided by the implementation. This is easier for implementers, still preserves forward compatibility by default, and is a much easier concept for users to understand. Of course the downside is that users are forced to use the vocabulary system if they want to use unknown keywords, but that should be fine if we have easy to follow documentation on how to do that.

This simplification enables another big benefit. It means that you would only declare a dialect ($schema) if you are using a custom meta-schema or custom dialect, which means we don't need to publish a set of meta-schemas for each release. We only need to maintain one set of meta-schemas for the current release. Because of backward and forward compatibility rules, all schemas written for a previous version or for a future version should work with whatever version of the meta-schema you are using. The only caveat is that you loose compatibility guarantees if you enable unstable keywords, but that's already the tradeoff you accept if choose to use unstable keywords.

Note that the must-be-off-by-default rule only applies to unstable keywords, not necessarily any unstable feature. For example, the output format would initially be unstable, but it doesn't make sense for it to ever be off. It could also make sense to automatically turn on an unstable keyword in some cases. For example, dynamic references would be initially unstable, but they are required to validate the schema against the meta-schema. In this case it would make sense for implementations to turn on dynamic references temporarily for this step and reset it when that step is over.

If no one objects, I'll update the proposal in a few days.

[handrews]

In case it wasn't clear, when I talk about there being a single dialect, I don't mean that dialects are effectively going away. I just mean that each release isn't effectively a new dialect like it is now.

That was definitely muddy, given language like "only one validation dialect". The paragraph makes it sound like custom dialects are relegated to some weird exceptional case instead of expected to be common. Given the vast volume of requests for keywords and formats (I shunted a bunch over the vocabs repo this week), we should expect custom dialects to be common, and talk about them accordingly. And yes, I think that means many common "validation" dialects.

Regardless, I also do not want a 2023, 2024, etc. dialect. That's not what I'm arguing for at all (I'd have said versions if I meant versions).

The whole point of this proposal is to move to a single stable-yet-continuously-evolving spec rather than a string of irregularly released immutable specs.

Those are not the only two options. We can have stable-yet-continuously evolving things without stapling all of the vocabularies in the current two documents into a monolith. That's what I'm trying to get at. There is no need to focus on the keyword set as primary definition of "JSON Schema". We should be focusing on what I called the architecture and feature layers. While we need to maintain the "standard" keyword definitions, they're (mostly) outside of what's really important. Get the lower-level requirements correct and the keywords are trivial.

I think @Relequestual was the one who commented that perhaps we should think about keywords like how functions are imported from modules in programming languages. I was skeptical whenever it was that he said that, but I have been coming around to that view as a better granularity and flexibility than what we've been doing.

While JSON Schema is not a programming language, there are enough parallels that it's a useful analogy here. Specifically, a programming language and its standard library are separate things. The programming language is still the programming language without the standard library, and it is the programming language's requirements and capabilities that are critical and have the heaviest implementation burdens. The standard libraries are important and often non-trivial themselves, but they're a separate layer with different concerns.

To me, the questions of how we manage the stability and change of whatever we're producing is orthogonal to the question of whether we have a single "JSON Schema validation" dialect or not (regardless of whether custom vocabs/dialects are still allowed).

This is why I feel like we have a lot of agreement but you feel like we do not: You have put these two things together in one proposal and I honestly do not understand why.

Those sections describe one way to fix existing issues with $schema and meta-schemas that would no longer be an issue with the new model. What's presented in those sections are not necessarily the only way. We can talk about alternatives, but it's absolutely relevant to the proposal and needs to be discussed.

So why not just specify the problems and the requirements for the fixes so that we can talk about the alternatives separately from the process? We don't need to decide whether to accept this or an equally-suitable alternative in order to make a process decision.

[handrews]

Also, as package management and automated installation has become more common, the importance of the distinction between the standard library and 3rd-party packages has lessened. For example, requests is not part of the Python standard library, but the Python standard urllib.request library documentation links to requests and advises people to use it for most HTTP client needs (urllib.request is more low-level).

That's the model I have in mind. In that model, yes it does matter which vocabularies/keywords are "standard", but nowhere near as much. Some 3rd-party libraries/vocabularies may become more important than some standard libraries/vocabularies (e.g. the content vocabulary is pretty specialized and not that widely used, but a good date-time or currency 3rd-party vocabulary would probably be very widely adopted.

[jdesrosiers]

Ok, so it sounds like maybe most of our disagreement is actually misunderstanding.

we should expect custom dialects to be common, and talk about them accordingly. And yes, I think that means many common "validation" dialects.

Absolutely. All I'm saying is that we as the JSON Schema Org are only releasing one validation dialect and that would be the default dialect. I would expect and encourage many third-party dialects. Absolutely nothing about the proposal is discouraging this.

stapling all of the vocabularies in the current two documents into a monolith

This is very much not what I'm proposing. The proposal positions the vocabulary system as something we don't need to be stable to move forward, not as something to be ignored, removed, or minimized. Other than identifying it as an evolving feature, this proposal says nothing about the vocabulary system, only dialect identification. I assume keywords will still be grouped into vocabularies and we would define our standard validation dialect very similarly to how we do now, but those are a separate topic and this proposal takes no position.

Given these misunderstanding, I'm starting to understand why you think parts of the proposal are irrelevant. If the proposal was addressing these things, I would agree that it doesn't belong.

Those are not the only two options.

I know, that's why I was trying to start a discussion of what other options you are in favor of. I don't think anyone wants to maintain the status quo and my proposal isn't the only alternative. If you don't care for my proposal, I want to have some alternatives we can discuss.

It's interesting to hear your thoughts on the vocabulary system, but that's not what I was asking about alternatives for. I was asking about the development/release model. Currently we have immutable, incompatible, irregularly released spec documents. I don't think anyone wants to keep doing that. This proposal proposes a single, semi-mutable, continuously updated spec document. At this point I'm unsure about your position on this fundamental part of the proposal. Are you ok with that model? If not, what alternative would you propose?

[handrews]

@jdesrosiers thanks for the discussions elsewhere the other day. The most recent updates are helpful. I going to ask that you remove the proposed URI for the default meta-schema, as that is a detail that we can decide at any time. I disagree with your suggestion (at least at the moment), but debating it here would be a distraction. The proposal does not hinge on that URI choice in any way.

I still have numerous other concerns, most of which I would prefer were separate discussions as I still think this conflates too much into a single proposal, but I will abandon that point and just start more top-level threads here.

[handrews]

@jdesrosiers it will take a few more days, but I have settled on some feature vs keyword concepts that I am working out to clearly illustrate my concerns around which layer(s) we target regarding stability and forwards/backwards compatibility, which is where my primary concerns are at the moment. That and pushing for generative use case concerns to be addressed rather than remaining instance evaluation-centric. But I think the generative thing will be more clear after I explain the keyword/feature thing. I just want to make it clear that I'm actively working on this even if I'm quiet for the rest of this week.

[Relequestual]

Broadly, I love this concept. On reflection, I'd want to see a few alterations based on recent discussions.

In our 2022-09-26 Open Community Working Meeting, you conceeded (or agreed, doesn't really matter) that having a snapshot in the similar way that TC39 does with ECMAScript, has valid use cases.

Currently, the proposal as it stands suggests that all features, regardless of stage, will live in the spec document. It will be an activity to the reader to consider each feature or keyword and determine if a feature is stable or not by reading the associated stage tag. The spec document will be "live" in as much as things can be updated and people should always look at the latest version.

The discussion regarding snapshots got me thinking. We aren't going to want snapshots containing proposals at specific stages. You only want the snapshots to contain the current stable features, or it sort of defeats the point of not having to determine which parts of the spec doc you can and cannot use / rely on.

I went back and read over how TC39 (ECMAScript) works in this regard, and then it hit me. The significant difference in what you're proposing here vs how TC39 function is where the proposals live. I'm very much of the opinion that the proposals should NOT live in the spec document. Our consensus regarding snapshots of the spec aiding people wanting to know what they can use with a specific "editions" of the spec, supports this direction.

Having proposals in snapshots feels like a really bad idea.

If you wanted to counter by saying the living spec doc could still have proposals, you're creating some unnesecary additional gymnastics for creating the snapshot. Beyond automation, you'd have to make sure section references were correct, and the new stable parts of the spec didn't refrence proposals.

My suggestion is to follow the approach of TC39, and rather than proposals going into the spec document, they live outside the spec document until they are propoted to the appropriate stage which guarantees them entry into the spec. If we accept the notion of yearly snapshots, there's no reason promotion of features to "stable" couldn't happen on a more regular basis (quaterly? TC39 do 6x a year).

In summary: This is a really great start. I'd like to see it looking closer to TC39, which I think is justified if I'm on the same page in that we found consensus regarding the need for snapshots (or "editions") to be published.

As a note, which I think I've mentioned before (but not here), TC39 consider their spec "a living standard" with snapshots. Staged proposals have been used in production and replaced with stable versions when reached, so their approach on this seems to work fine.

[handrews]

@Relequestual by "proposals" do you mean anything that's STAGE-1 or STAGE-2? I can definitely see wanting STAGE-1 proposals elsewhere. I would also prefer that deprecated features get moved out (they can be linked in the main document, but shouldn't take up as much space). Maybe they get to stay in the main document for a year of transition and then they move out.

I'm not entirely sold on promoting to "stable" more than once a year, but I'm open to it. I think that if STAGE-2 proposals live in the same doc, then that gives us more flexibility.

The theme here for me is that the "main document" (whether that's literally one document, or two like we have now, or one for each vocabulary, or whatever) should have the things that we want people to use. STAGE-2 things are things we want to encourage people to implement and use enough to justify a stable designation, and having the outside the main doc(s) discourages that. Likewise, we want to discourage the use of deprecated features.

[jdesrosiers]

Thank you, this is great feedback and exactly the kind of details we need to be working out right now.

I was thinking more about the snapshot idea as well. It occurred to me that filters would solve the use cases the group brought up for wanting snapshots and it solves it in a way that doesn't have the problems we identified with using snapshots. The spec is moving to the website, which means we can easily do things like filter the content based on the stability flags. So if the spec is at 2025 and someone wants to view just what was stable in 2023, they can click a UI element of some kind and filter out anything from 2024, 2025, STAGE-1, or STAGE-2. This way they can focus on the content they want without losing the clarification and bugfix updates they would be missing if they were looking at a snapshot. The filter approach also solves the problem of unstable features in snapshots.

The significant difference in what you're proposing here vs how TC39 function is where the proposals live. I'm very much of the opinion that the proposals should NOT live in the spec document.

Yes, that is a significant difference. Just to be clear on terminology, I would not consider unstable features to be proposals. The life cycle of a feature would be to go from proposal (Github Issue or Discussion) to STAGE-1 (in the spec from here on) to STAGE-2 to stable and potentially to deprecated. There's actually a very high bar just to get to STAGE-1 status. Much higher than we have now to add a feature.

My thinking about including unstable features directly in the spec was in line with what @handrews said, that having them in the spec encourages people to implement them. If we aren't ready to encourage people to implement something, it's not ready for even STAGE-1. Therefore, I think a proposal doesn't belong in the spec, but STAGE-1 and STAGE-2 features do. They can always be filtered and hidden away as described above so people don't have to look at them if they explicitly can't use unstable features.

I would also prefer that deprecated features get moved out (they can be linked in the main document, but shouldn't take up as much space). Maybe they get to stay in the main document for a year of transition and then they move out.

Deprecated features still need to be implemented and can't be removed because of backward compatibility. I don't think you're suggesting removing them entirely from the spec, just hiding them away somewhere so people don't stumble on them and want to use them. I think I'd prefer that they were collapsed by default rather than somewhere else entirely. We want to hide them away from schema authors, but implementers still need to be able to find them.

If we accept the notion of yearly snapshots, there's no reason promotion of features to "stable" couldn't happen on a more regular basis (quaterly? TC39 do 6x a year).

The reason I had promotion only happen once a year is because the labels wouldn't make sense between releases. The problem would be the same whether we do snapshots or not. If we are in the middle of 2024 and make a feature stable, it would be making it stable for the 2025 release. But, there is no 2025 release yet. Someone might write their implementation in 2024 and claim support for 2025 because they implemented all the 2025 features that were in the spec, but more 2025 features were added after they released.

I think we can find a solution to that problem and I think it could be value in that it gives implementers advance notice about what will be stable in the next release. But, I think it's a tweak we can make later. I have no expectation that we're going to get this perfect on the first try. I expect this to be a baseline that we build upon. So, I say we keep it simple for now with promotion once a year and make adjustments later if necessary.

[handrews]

I think I'd prefer that they were collapsed by default rather than somewhere else entirely. We want to hide them away from schema authors, but implementers still need to be able to find them.

I'm not particularly hung up on the mechanism, I just want it to be very clear that they are deprecated even to the most casual skimming glances at the spec. Instead of, say, having them mixed in with the other keywords with just a little "deprecated" tag. In particular, I want it to be apparent from the table of contents. having a separate document obviously accomplishes that. Having a section structured such that the keywords in it don't get the same prominence as the non-deprecated ones would as well. At minimum, pushing them all to a clearly marked section at the end would be acceptable. There are probably other options.

[Relequestual]

It occurred to me that filters would solve the use cases the group brought up for wanting snapshots and it solves it in a way that doesn't have the problems we identified with using snapshots.

I hadn't considered that we might provide the spec as a dynamic artifact (as opposed to static content). I'm not super keen on this approach. Doing so would make moving to a standards org at a later date more difficult. (I guess if we decided to do such a thing, our process would change anyway.)

The significant difference in what you're proposing here vs how TC39 function is where the proposals live. I'm very much of the opinion that the proposals should NOT live in the spec document.

Yes, that is a significant difference. Just to be clear on terminology, I would not consider unstable features to be proposals. The life cycle of a feature would be to go from proposal (Github Issue or Discussion) to STAGE-1 (in the spec from here on) to STAGE-2 to stable and potentially to deprecated. There's actually a very high bar just to get to STAGE-1 status. Much higher than we have now to add a feature.

I'd like to see a formally declared "Stage-0", and to have Stage-1 broken up into two parts. I can see the bar is a LOT higher, and I'd want to break it up for the following reasons:

• It should be hard to get proposals to STAGE-1, but not so hard that it puts people off trying. Having a lower hanging easier stage might make it easier, and allow us to focus less on immature poroposals rather than trying to handhold people towards STAGE-1 (such as getting tests in the test suite)
• I feel we need a "this has legs, go do stuff" approval stage. The proposal currently requires someone make significant effort after simply presenting the idea, before the "core contributors" anoint it STAGE-1. I'd rather have a "fast failing" filter, which I feel would likely take the form of another stage.

(Apologies, they weren't as short as I'd have liked.)

I'd also like to see a champion being a requirement, and I think the process would benefit from being perscriptive about the order of things to be done in order to reach a Stage.

Having defined a few Spec DLCs previously to varying success, my key takeaway is this:
Should be... Easy to start, easy to understand a clear path, non-trivial to complete.

Deprecated features still need to be implemented and can't be removed because of backward compatibility. I don't think you're suggesting removing them entirely from the spec, just hiding them away somewhere so people don't stumble on them and want to use them. I think I'd prefer that they were collapsed by default rather than somewhere else entirely. We want to hide them away from schema authors, but implementers still need to be able to find them.

I stumbled onto the following FAQ for TC39...

Why don't we deprecate features?
Deprecation doesn't work on the web. Since we can't remove bad features, a developer has little incentive to stop using a feature just because somebody somewhere doesn't like it. Labeling features "deprecated" without ever removing them is pointless.

While JSON Schema isn't exclusivly in browser or web based, it does raise a good point. I think having keywords deprecated still has its uses, but we could do this as part of the official recommended linting rules (when such a thing exists) as opposed to just in the spec.

My thinking about including unstable features directly in the spec was in line with what @handrews said, that having them in the spec encourages people to implement them. If we aren't ready to encourage people to implement something, it's not ready for even STAGE-1. Therefore, I think a proposal doesn't belong in the spec, but STAGE-1 and STAGE-2 features do.

I have some more thoughts on your comments, but I have to restart my machine for some reason -_-

[jdesrosiers]

I hadn't considered that we might provide the spec as a dynamic artifact (as opposed to static content). I'm not super keen on this approach. Doing so would make moving to a standards org at a later date more difficult. (I guess if we decided to do such a thing, our process would change anyway.)

If we get to the point that we want to move to a standards org later, it would be because we don't need stability flags anymore and therefore we don't need filters anymore. I don't think it would be a problem.

I'd like to see a formally declared "Stage-0"

I can certainly see the benefit of an additional stage before STAGE-1. Given the criteria currently listed for STAGE-1, the only thing I can see that makes sense to move to a STAGE-0 would be consensus to add the feature. Maybe that's all it needs. Maybe STAGE-0 is just that and an indicator that people should be actively working the step for STAGE-1. I'm open to suggestions on other ways to break it up.

I wouldn't suggest STAGE-0 be added to the spec yet, so how do you think we should manage it? A label on a github issue is probably the easiest solution.

I'd also like to see a champion being a requirement

Agreed!

I think the process would benefit from being perscriptive about the order of things to be done in order to reach a Stage.

I'm not sure I agree. From looking at the list of criteria for each stage, there are certainly things that could be done in parallel and others that have a clear natural order. So, I'm not sure what the benefit of being prescriptive is. But, I'm open to suggestions about what it might look like.

Why don't we deprecate features?

I remember reading something where they said, they may deprecate features, but they very rarely remove them because it's practically impossible to be sure that a feature isn't relied on by someone somewhere. That was what I had in mind with this proposal. I don't think it will be a tool we use often (hopefully never), but I think it's a tool we want to have. For example, if a feature is discovered to be a security issue, we want to be able to at least strongly discourage it's use.

[Julian]

A few scattered comments in case any are helpful -- I'll admit I have not read the long thread, just the original post, and that broadly the below is (somewhat as usual) my thoughts which are welcome to be ignored if others disagree:

• I strongly support committing to more backwards compatibility, which seems to be an important piece of the puzzle. Some of the changes we made over the last few releases seem to me to create churn, and if we can agree to do less (or none) of those, strong +1.

• Same for disallowing unknown keywords by default, strong +1, the current model doesn't make a ton of sense to me, and consistently confuses users (of my implementation, and from what I see in general).

• I think it's harmful to use opaque names like stage-n rather than terms like "unstable" (for stage-1) or "beta" (for stage-2) or some even better terms than those

• As an implementer, I strongly suspect I wouldn't implement stage-1 or stage-2 features in my core implementation at all -- and I suspect others may do the same. Doing so would I think just be too confusing for end-users of my own library, who don't know much "on average" about the spec itself, they have some data validation problem and manage to find JSON Schema. Within the Python language itself say, there are modules in the stdlib which behave this way (where they're released but have lower standards for backwards compatibility) and in general they're really painful -- so I as a library author would, yeah, probably leave them for external add-on packages to "bolt on" on top of my library. This isn't a concern per se, I'm just opining since I think in the model being proposed here, others may behave similarly.

• I have a general sense that there's "too much change" in this one proposal if I'm honest, and that there are many independent decisions we could have decided on. E.g.:

  • are we comfortable with no more backwards incompatible changes
  • should we change the default for unknown keywords
  • do we like meta schema URIs without versions in them?
  • do we want to change the version schema
  • do we want to introduce the notion of unstable features
  • do we want to proscribe specific deprecation behavior
  • do we want to move to GitHub markdown
  • should we publish yearly review posts on the blog
  • ...

Each of these to me really could (have) been treated individually without changing all at once in my opinion -- but I see in skimming some other posts that others may have found the opposite helpful, that since this is comprehensive, it's easier to agree on. Different strokes for different folks I suppose. I guess this too isn't a concern -- what's done is done here and it's good work, but if it turns out any of those are more contentious than others perhaps a way to proceed would be to reduce scope.

The above is very much a brain dump as I scan the post, so I'll (well, apologize for not being more ordered or timely with the feedback) but also stress again that I like this train of thought (and general proposal) quite a lot personally!

[gregsdennis]

I like proposal -> beta -> release candidate. It does disconnect the alpha/beta theme, but I think that's okay.

[gregsdennis]

Or proposal -> experimental/research -> release candidate

[jdesrosiers]

I think expermental/research would work better for a STAGE-0 than a STAGE-1, but I'm still liking alpha/beta better for STAGE-0/STAGE-1.

[Relequestual]

I like the stages closer to how they are defined in the TC39 process.
Stage-0 for "here's an idea, anyone like it enough? should I write some more details?" - a sort of strawperson.
Stage-1 for "Here's some more details about the idea, what might be involved, the challenges and impact" - We could make a form / template for the info we would want to see.

Once the details are provided for stage-1, we would then review and approve the application to be in stage-1.
Stage-1 and above would be in a list somewhere prominant for people to see, read, review, comment on etc. (I imagine similar to TC39).

[jdesrosiers]

I don't feel too strongly about whether we use numbered stages like ECMAScript or semantic names. Whatever everyone wants will be fine with me. However, I definitely see @Julian's point about numbered stages being confusing to users.

The naming of the stages isn't a crucial piece of the plan. I'd rather focus on defining the stages at this point. In the spirit of this proposal being a baseline for us to iterate off of, if naming isn't something we can agree on quickly, I suggest we keep the numbered stages for now and create an issue to discuss changing the names once the proposal is approved.

[Relequestual]

@jdesrosiers Can you create an issue for capturing a list of the issues related to this discussion (the SpDLC) please? It would be worth also including a summary of what we have so far, but not much detail. As a locked issue, and asking for no comments, to be used as a tracking location for the sub-discussions.

I think we all feel this direction makes sense, but now we're moving, as @Julian pointed out, there are a number of discussions/topics which could do with their own space in the form of an Issue to discuss, and so we can easily see the summary of discussions so far (by editing the first post in the Issue as required).

[handrews]

I think we all feel this direction makes sense

While this is more-or-less true, I do still have substantial concerns (which I expect to be resolvable - this is not an effort to block progress) which I am working on articulating. We are not at the point of moving to a TODO list for this.

[jdesrosiers]

As a compromise, I'm going to add a section to the initial post that lists unresolved discussion points for the proposal. However, I'd like to be able to start moving on a TODO list for this by this time next week unless any blocker issues are identified.

[handrews]

Since STAGE-1 and STAGE-2 keywords are not available by default, is their inclusion handled through additional dialects or is there expected to be some other mechanism?

Specifically, I am asking from the perspective of a schema author using STAGE-1 and/or STAGE-2 keywords and wants to ensure they are available before the schema is processed. How do they do that? For STAGE-2 in particular, we expect these to be widely used and stable enough to be relied upon in an open system.

[Relequestual]

I'd imagine through additional dialects, where the vocabularies are the individual keywords / proposals.

[jdesrosiers]

At this point, the mechanism is undefined as you requested. Unstable keywords are part of the normal vocabularies. For example, $dynamicRef would be in the core vocabulary. Currently the vocabulary system has no way of declaring that an unstable keyword should be enabled, but I'm not sure it needs to. Because undefined keywords are no longer allowed, there is no ambiguity. The validator should reject the schema if it uses a keyword it doesn't support.

There are a couple reasons why enabling unstable keywords using the vocabulary system isn't great. First, each keyword needs to be able to be enabled individually, which would mean a bunch of single keyword vocabularies. Most importantly, if unstable keywords are in a special vocab, it needs to be moved when it becomes stable. Schema authors would have to change their schemas when a keyword becomes stable. Schema authors shouldn't have to update their schemas unless we make a backward incompatible change to an unstable keyword.

Another issue is that using an unstable keyword comes with risks, so if consumers of a schema don't want to accept that risk, they should be able to reject a schema that uses unstable keywords regardless of what the author of the schema intended or what the implementation supports. I image the implementation would have to provide some sort of configuration option.

[handrews]

Thanks, this answers my question, and then some - I think we may have slightly different notions of what I'm asking here and what I've asked to be left undefined. And ideally discussed in #243, where the concerns you raise here about wanting consumers to be able to decline to evaluate a schema with unstable keywords fit well with what I proposed. I also agree with your granularity concerns which we can and should discuss elsewhere as I don't think we need to solve this to approve the SDLC.

I was just trying to figure out if you saw managing unstable keywords as a subset of managing keywords in general, or if they were a totally orthogonal things. It sounds like they are a subset of managing keywords in general: There may be additional controls, but not completely separate controls. The concerns over granularity are things that we need to address more broadly as well, so I think that stays within "subset" as opposed to "orthogonal."

[jdesrosiers]

Yes, I want unstable keywords to be defined, implemented, and used used like normal keywords as much as possible. The only difference should be that they have a concept of being enabled/disabled by schema consumers.

[handrews]

@jdesrosiers sounds great to me!

[handrews]

[sorry for the delay- after the reactions in yesterday's call I scrapped everything I'd written and started over with a new approach]

Goal for this feedback

The goal of this feedback thread is to ensure that the SDLC's stability guarantees are applied to the correct things in order for us to have the healthy vocabulary ecosystem necessary to follow our plan of asking people to create extension keywords prior to attempting to get them into the JSON Schema specification.

It hinges on the idea that we have three audiences:

• Schema authors & consumers
• Keyword designers
• Implementers

As far as schema authors/consumers are concerned, only keywords and a few other things (e.g. output, turning format validation on and off, etc.) have observable behaviors. However, keyword designers and implementers "observe" the keyword support interface, whether it is a formal plugin API or not. This is independent of the vocabulary control mechanism, which is only about how to enable or disable keywords/vocabularies.

Assumptions

For several years now, we have told people that keywords should be developed first as extensions. The SDLC would formalize this approach by placing implementation requirements on keywords prior to accepting them to STAGE-1 status within the specification. This is a very good thing, and one reason I generally support the direction of the SDLC.

My assumption is that we will continue this, and also that many useful and widely supported keywords will never be added to the spec. We will expect them to be broadly and interoperably usable anyway.

This keyword design funnel will work well if most keyword proposals can be implemented in most implementations, which will only happen if:

• Most implementations support extension keywords
• Keyword designers know what sort of capabilities a keyword can definitely have
• Implementers have clear normative requirements to support those capabilities

Or in other words: The right thing must be easy, and the easy thing must be right.

Keyword support needs to be consistent, at least within well-understood areas at a conceptual level, across implementations so as to minimize friction when trying out reasonable new keywords. Of course, there will always be keywords stretching the boundaries — we are not trying to prevent that. We just want it to be clear to everyone when a boundary is being pushed so that we can consider the impact to the ecosystem properly.

Desired SDLC-level Outcome

As I understand it, the SDLC stability process is currently defined in terms of keywords, the standard dialect, and observable non-keyword features. This thread proposes adding capabilities, as demonstrated in the examples posted as replied to this comment, as an additional target for that process.

This thread is not about enumerating the capabilities or defining the requirements and stability levels currently involved. It is not even necessary to completely enumerate/define them for the next release. I assume that we can figure out the right granularity and level over time.

This feedback is about illustrating what capabilities might look like, and making the case that they are a valid interface for two of our three audiences and therefore the SDLC ought to apply to them. This would be a significant change to our thinking, and would need to be integrated into our process to ensure appropriate levels of attention.

Examples of benefits

In responses to this comment, I will go through several detailed examples to highlight benefits. As a TL;DR, here are the highlights:

• Anyone who can read the spec can determine if a proposed keyword will definitely be broadly implementable by correlating its observable behavior with mandatory capabilities (postfixItems)
• A keyword with observable behavior that requires breaking a STABLE capability is immediately obviously out-of-scope for JSON Schema (optional from early drafts)
• A keyword with implementation details that require breaking a STABLE capability clearly needs to be reconsidered in terms of the available (or at least not forbidden) capabilities (minContains)
• A keyword whose observable behavior cannot be expressed in terms of existing capabilities but also does not violate any existing capabilities will trigger a deeper discussion of impacts, resulting in new capabilities that must go through STAGE-1, STAGE-2, etc. (concat and other complex array validation keywords)

In conclusion

That last point from the examples about sending a new capability through the STAGE-1, STAGE-2, etc. process is extremely important.

I am convinced that one of the worst things about how we introduced unevaluated* and $recursive* is that we justified their observable behaviors without getting adequate feedback on and consideration of the necessary underlying capabilities and their impacts.

It's not that we needed to ratify implementation details, it's that we didn't think through the implications for things like parallel evaluation, runtime state management, etc. Some of you likely did, but there was no process in place to force my attention towards those concerns. I don't recall giving them much thought, TBH.

If we had had to put the child-to-parent and parent-to-child runtime dependency relationship capabilities through a STAGE-0, STAGE-1, STAGE-2 process, we would have gotten far more feedback on the impact of those changes (see examples in comment replies for definitions of these capabilities).

We would have empowred the community to give clear feedback on both the observable behavior of the keywords and the expected underlying changes prior to any of them becoming baked-in. This could even have included feedback no whether the underlying capabilites ought to be available to non-standard keywords. Instead, here we are three years after their publication and only now debating whether these should be limited to standard or builtin keywords or available more broadly.

To me, the fact that the SDLC, if extended to capabilites, would have flushed these concerns out up front, is alone enough for me to endorse the new process. But I only see this benefit happening reliably in a way that will be robust to any of us leaving the project over time if capabilities are part of the process alongside of keywords, the standard dialect, and end-user-observable non-keyword features.

[handrews]

Avoiding breakage: minContains

In draft-06, we added contains, and quite possibly would have promoted it to STAGE-2 by the draft-07 patch release (more than a year later).

In draft 2019-09, we added minContains and maxContains. From a schema author perspective, the observable behavior was fine. But it was specified in a way broke assertion requirement 3.

If we had been explicit about that normative requirement, then it would have been immediately apparent that the proposed language required breaking a STABLE guarantee, and therefore absolutely could not be added to the spec. Breaking that guarantee would break other guarantees around how keywords can be processed safely, which is why it would be a normative requirement.

Hopefully at that point we would have worked out whether it was possible to get the keyword's observable behavior with the then-current capabilities, which it definitely was.

Normative capability requirements would have helped with this as well. One reason I wrote all of the "use annotations like this to implement dependencies" language for each keyword is that there was nothing to point to regarding possible mechanisms. My sample capabilities define annotations, dependency sources, and dependency relationships separately.

The annotation capabilities guarantee that there is at least one way to implement dependencies that is covered by normative requirements. However, there is no requirement to use that mechanism.

[handrews]

Understanding novel demands: concat and friends

The proposal to essentially implement regular expressions for arrays using several applicator and flow control keywords cannot be described in terms of existing JSON Schema capabilities.

As noted, these keywords require backtracking or non-deterministic evaluation to figure out how to apply subschemas to which instance locations.

It might be possible to define this behavior in terms of the dependency source and dependency relationship capabilities already defined (at least if we can articulate the parent-to-child dependency relatoinship, which I punted on). If so, then work on such keywords will be do-able across enough implementations to get real feedback.

If not, new capabilities need to be defined as STAGE-1 capabilities and get implemented, possibly at the same time as the proposed keywords. But there is a huge benefit here which is that we have to reckon with the full implications of adding these behaviors.

It might be that not very many people get excited by complex array validation. But if we publish a new STAGE-1 capability, implementers will be more motivated to weigh in. Others would be likley to experiment with more keywords taking advantage of the capabiity, which would help us understand whether we'd be adding a useful, contained feature, or whether we'd be opening up a horrible can of worms.

[jdesrosiers]

As I understand it, the SDLC stability process is currently defined in terms of keywords, the standard dialect, and observable non-keyword features. This thread proposes adding capabilities [...] as an additional target for that process.

I have no problem with that. I defined the stability and compatibility models using the very generic term, "features". That term was intentionally kept vague so it could include concepts we didn't have a name or conceptual model for yet. That would include "capabilities". I agree that defining capabilities is worthwhile, that it should be subject to the stability process and compatibility rules, and that it doesn't need to be figured out before the right release.

I think it's great that you've identified this concept. It's not exactly new, but we never had a label for it and therefore it hasn't been in the forefront of our considerations. I've been concerned that something was going to accidentally sneak into "stable" when we do the initial release because it's implied by some stable feature. I think the concept of capabilities will be useful to identify those cases and help decouple stable behaviors from the unstable mechanisms.

[handrews]

Great! And thank you.

I agree that the capabilities concept isn't truly new, and I did not think that you were necessarily excluding it. But the focus on keywords end-user-observable definitions of "features", plus the lack of any consensus on how to talk about capabilities and make sure they'll be addressed is what had me concerned. Two weeks ago, I was still feeling like this process would make it more likely that we'd back ourselves into a corner somehow.

This capabilities approach, the earlier keyword/feature/architecture idea (which I developed a lot more before abandoning it), and the even earlier keyword behaviors slide decks are all attempts to articulate the same concerns. None of them prior to now were sufficiently well-received to reassure me that we could handle this through the proposed process. Which is fine! The feedback to me was valuable and obviously I agree that the earlier formulations were not quite right.

Thank you for your patience. I realize this was a frustratingly long road, and in the end might even seem a bit anti-climactic given the amount of time involved. And I know it looked like I was going down all sorts of irrelevant roads at times. But ultimately that process did what I hoped it would do, which was allow me to find a way to support this without having to argue it all out in detail. Even a week ago I was still skeptical of quite a few things. It wasn't until I wrote the "In Conclusion" section about 15 minutes before posting this last night that I got to the point of not just "it will probably be acceptable" but "we would have been better off if we had adopted this proposal years ago."

At this point I am comfortable endorsing the proposal and moving forward with whatever next steps are deemed appropriate. If I listed any other concerns elsewhere, I was able to resolve them one way or another along the way, or one of those side roads convinced me it could be handled separately.

[handrews]

I think the concept of capabilities will be useful to identify those cases and help decouple stable behaviors from the unstable mechanisms.

@jdesrosiers I just want to add that I agree that this is particularly important, and note that it was various comments from you over the past several months that got me thinking about this distinction. It's probably the biggest difference from the keyword behaviors idea, which was a lot more mechanism-oriented.