"unevaluatedItems" should consider "contains"

[handrews]

The interaction of these seems underspecified. @Relequestual what was the problem you ran into here exactly? I've already forgotten 😬

[Relequestual]

This was me being unable to read that unevaluatedItems only applies when items is defined as an array.

With that light, there’s no valid argument that I can think of that would warrant interaction between these keywords.

--

Edit: The intent of unevaluatedItems was to be effected by all array based applicators, which includes contains. The spec as written currenltly, doesn't allow this.
#810 (comment)

[handrews]

Actually, I'm pretty sure I left out something important. Consider the following schema:

{
  "type": "array",
  "oneOf": [
    {
      "contains": {"type": "string"},
    },
    {
      "items": {"type": "integer"}
    }
  ],
  "unevaluatedItems": {"type": "boolean"}
}

The following instances should all be valid:

[true, "hello", false]
[1, 3, 4]
[true, false]

I had intended for all array child applicators to affect unevaluatedItems, and all object child value applicators to affect unevaluatedProperties. (propertyNames is an object child name applicator).

I seem to have left out the necessary specification of a contains annotation and how unevaluatedItems could use it.

Not sure if there's a use case, but logical consistency was the goal.
@Relequestual does this make sense?

[Relequestual]

Logical.
Would contains collect all items that were valid for it? As such, would unevaluatedItems only apply to those which were not valid according to contains?

Sorry, on mobile.

[handrews]

@Relequestual

Would contains collect all items that were valid for it?

Yes, there is language in the existing description that requires contains to be checked against all array items when collecting annotations. If you're using unevaluatedItems you have to collect annotations. I'll make sure that's clear, though, good question.

would unevaluatedItems only apply to those which were not valid according to contains?

That is correct. contains gets applied to all but "evaluated" really means "successfully evaluated" because annotations are dropped from failed evaluations.

[handrews]

Hm... lots of test cases there.

[Relequestual]

Well, my understanding / expectation is correct then.
I'd have to re-read the spec to understand where you're saying there's a deficit though.

[handrews]

@Relequestual if you find anywhere that makes that behavior explicit, let me know.

I don't think there's anything there that contradicts it, and it obviously won't require a meta-schema update. I made that "draft-08-patch1" milestone to collect compatible fixes based on early feedback. Or if we end up thinking that this is a kind of glaring error we can put this and whatever else we've noticed in and re-publish the core spec sooner rather than later.

[Relequestual]

I think the phrase "annotations are dropped from failed evaluations" is key here.

Does contains return an array of annotations with nulls where the annotations have been dropped (becaused of failed validation), or an object where the keys are the indexes at which the array successfully validated? Latter sounds better. I don't think it's currently defined.

It only really matters what format the annotations take for the output format data.

I guess contains is an applicator keyword like anyOf. At least the intent is clearer and should help people avoid using oneOf here... =D

The prose / explanation about how annotation results are used will need to be updated here:

json-schema-spec/jsonschema-core.xml

Lines 2209 to 2217 in 38e1606

	The behavior of this keyword depends on the annotation results of
	adjacent keywords that apply to the instance location being validated.
	Specifically, the annotations from "items" and "additionalItems",
	which can come from those keywords when they are adjacent to the
	"unevaluatedItems" keyword. Those two annotations, as well as
	"unevaluatedItems", can also result from any and all adjacent
	<xref target="in-place">in-place applicator</xref> keywords.
	This includes but is not limited to the in-place applicators
	defined in this document.

[Relequestual]

Given the proceeding paragraphs, making this fix would legalistically be a non backwards compatible change, but I think it's clearly a mistake.

json-schema-spec/jsonschema-core.xml

Lines 2219 to 2232 in 38e1606

	<t>
	If an "items" annotation is present, and its annotation result
	is a number, and no "additionalItems" or "unevaluatedItems"
	annotation is present, then validation succeeds if every instance
	element at an index greater than the "items" annotation validates
	against "unevaluatedItems".
	</t>
	<t>
	Otherwise, if any "items", "additionalItems", or "unevaluatedItems"
	annotations are present with a value of boolean true, then
	"unevaluatedItems" MUST be ignored. However, if none of these
	annotations are present, "unevaluatedItems" MUST be applied to
	all locations in the array.
	</t>

[handrews]

@Relequestual yeah, this falls under "bugfix"

[gregsdennis]

@handrews I don't think that [true, false] should pass: it doesn't meet the oneOf requirement. For this case, unevaluatedItems is irrelevant.

[gregsdennis]

would unevaluatedItems only apply to those which were not valid according to contains?

That is correct. contains gets applied to all but "evaluated" really means "successfully evaluated" because annotations are dropped from failed evaluations.

Also, I don't think I'm doing this right.

(Edit: Fixed with v12.1.0)

[handrews]

@gregsdennis

@handrews I don't think that [true, false] should pass: it doesn't meet the oneOf requirement. For this case, unevaluatedItems is irrelevant.

Yes, that's correct, good catch.

[Relequestual]

I read that and made a wrong assumption. Yikes.
Let me fish out my example from my slides for this. It's a tricky one to reason about.

[Relequestual]

This issue has changed from it's original, but that's not a problem, as a bug is still highlighted.
I'll modify the title and first comment to reflect.

[Relequestual]

Wait, no. We are forgetting something here.

additionalItems is only applicable when items is an array, not when items is a single schema.
SO, the example @handrews used above (#810 (comment)) wouldn't be useful for additionalItems, and the logic behind unevaluatedItems relies on annotation collection from items as an array only.

It's here in draft 2019-09:

json-schema-spec/jsonschema-core.xml

Lines 2219 to 2225 in d561ae6

	<t>
	If an "items" annotation is present, and its annotation result
	is a number, and no "additionalItems" or "unevaluatedItems"
	annotation is present, then validation succeeds if every instance
	element at an index greater than the "items" annotation validates
	against "unevaluatedItems".
	</t>

From draft-7...

json-schema-spec/jsonschema-validation.xml

Lines 527 to 536 in 1afc34b

	<t>
	If "items" is an array of schemas, validation succeeds
	if every instance element at a position greater than the size
	of "items" validates against "additionalItems".
	</t>
	<t>
	Otherwise, "additionalItems" MUST be ignored, as the "items"
	schema (possibly the default value of an empty schema) is
	applied to all elements.
	</t>

If we want to change this, so you COULD do additionalItems or unevaluatedItems where items isn't an array, then the annotation collection needs to be changed to not just track the max index to which the subschema value of items was applied, but the result of applying it to every item in the array it's being applied to.

Thoughts?

---

Here's my example of unevaulatedItems being used, where items is an array. I remember this triping me up before now.
https://speakerdeck.com/relequestual/json-schema-draft-8-to-vocabularies-and-beyond?slide=25

[Relequestual]

I'm opening a channel for this on slack as I think we will have some discussion. We should make sure we are on the same page and understand correctly before determining what to do here.

[handrews]

contains should just return a list of indices that validated as its annotation. That's the minimum amount of information needed to use unevaluatedItems. And if array items and contains were both used, then an implementation can just look for indices greater than the highest index from the items array.

[Relequestual]

@handrews @gregsdennis I think I follow now.
Here are some examples of how the annotation collection for contains should work and how it relates to unevaluatedItems

{
  "type": "array",
  "oneOf": [
    {
      "contains": {"type": "string"}
    },
    {
      "items": {"type": "integer"}
    }
  ],
  "unevaluatedItems": {"type": "boolean"}
}

// Valid Instances

  [1, 3, 4 ],
  // each item is valid according to the `items` schema value which results in a `true` annotation, 
  // so `unevaluatedItems` is not applied

  ["a", "b", "c"],
  // each item is valid when the `contains` schema value is applied, resulting in an annotation of [1, 2, 3],
  // there are no additional items in the array, so `unevaluatedItems` is not applied

  [true, "hello", false],
  // `items` fails validation, so produces no annotation
  // `contains` is applied, and results in an annotation of [1] for the index of which item is valid
  // `unevaluatedItems` is applied to the other indexs of the array

// Invalid instances 

  ["a", false, 2, "b", 8 ]
  // `items` is not met, so produces no annotation
  // `contains` is applied and results in an annotation of [0, 3]
  // `unevaluatedItems` is applied to the other items in the array, and fails because numbers are not booleans


  // Another example
{
  "allOf": [
    {
      "items": [
        { "type": "integer" },
        { "type": "string" }
        true,
      ]
    },
    {
      "contains": { "type": "object", "required": ["error"] }
    }
  ],
  "unevaluatedItems": { "type": "object", "required": ["debug"] }

}

// Valid

  [1, "errorABC", { "error": 123 }]
  // Int, string, anything is fine
  // Must contain an object with an error
  // `items` results in annotation of [0,1,2]
  // `contains` results in annotation of [2]
  // `unevaluatedItems` is not applied to the array as there are not additional items beyond item at index 2.

  [2, "errorABC", "fail reason as string", { "error": 123 }]
  // `items` results in annotation of [0,1,2]
  // `contains` results in annotation of [3]
  // `unevaluatedItems` is not applied to the array as above

  [3, "errorABC", "fail reason as string", { "error": 123 }, { "error": 456 }]
  // `items` results in annotation of [0,1,2]
  // `contains` results in annotation of [3, 4]
  // `unevaluatedItems` is not applied to the array as above

  [4, "errorABC", "fail reason as string", { "error": 123 }, { "debug": "some debug info" }]
  // Same annotations as previous
  // item at index 4 has `unevaluatedItems` schema value applied

  [5, "errorABC", "fail reason as string", { "debug": "some debug info" }, { "error": 123 }]
  // `items` results in annotation of [0,1,2]
  // `contains` results in annotation of [4]
  // `unevaluatedItems` is applied to item at index 3

// Invalid

  [6, "errorABC", "fail reason as string", { "info": "some other info" }, { "error": 123 }]
  // `items` results in annotation of [0,1,2]
  // `contains` results in annotation of [4]
  // `unevaluatedItems` is applied to item at index 3
  // Validation fails for item at index 3 when `unevaluatedItems` schema value is applied.

  [7, "errorABC", "fail reason as string", false, { "error": 123 }]
  // Same as above

  [8, "errorABC", "fail reason as string", { "error": 123 }, { "info": "some other info" }, { "error": 456 }]
  // `items` results in annotation of [0,1,2]
  // `contains` results in annotation of [3, 5]
  // `unevaluatedItems` is applied to item at index 4
  // Validation fails for item at index 4 when `unevaluatedItems` schema value is applied.

[gregsdennis]

I think that lines up with what @handrews is saying, but I don't agree with unevaluatedItems not covering items that the contains subschema matches on.

In my interpretation, contains applies not to the item, but to the array. While it validates that one of the items matches its subschema, its annotations should apply to the array, not the item that passes.

That means that your third case changes:

  [true, "hello", false],
  // `items` fails validation, so produces no annotation
  // `contains` is applied, and returns a result of true on the array because the array contains a matching item (the index of the item is 1, and this can be an annotation)
  // `unevaluatedItems` is applied to all indexes of the array because the items themselves haven't been annotated.

---

Writing that out, it feels wrong. Maybe I'm getting hung up on the word "contains" meaning that its target is the array and is validating whether it contains something. Instead it's working as sort of a "some of the items should be this, and I want to know which items they are, then don't re-evaluate them, like ever." But I don't know of a single word that does that, so maybe "contains" works, but it still feels weird.

In short, @handrews wrote the unevaluatedItems stuff so he knows best how he intended it to work. Go with what he says.

[Relequestual]

In my interpretation, contains applies not to the item, but to the array. While it validates that one of the items matches its subschema, its annotations should apply to the array, not the item that passes. - @gregsdennis

@handrews For me, this is enough justification for additionalItems and unevaulatedItems to NOT consider the annotation result of contains.

That being said, it may still be useful to collect the annotations which indicate WHICH items in the array pass the contains schema.

Assuming we resolve #864 and end up with tupleItems etc, then unevaluatedITupleItems, hopefully it will be clearer that there's no interaction with `contains.

Thoughts?

[handrews]

@Relequestual I disagree with @gregsdennis's interpretation. contains applies the subschema to each item in turn, and the annotations from that subschema are most definitely attached to the item. This is exactly the same as what items does. The only difference between items and contains is that the former is an AND and the latter is an OR.

Let's pretend we had a keyword singleItem that takes a schema and an index of the item to which it applies. It puts that index in its annotations results. So if it successfully applies to index 2, then it has an annotation of 2, plus the subschema's annotations.

If we could write out an infinite sequence of these, so that they could cover an indefinitely long array, we could write items and contains in terms of this keyword.

items would be:

"allOf": [
  {
    "singleItem": [0, {...}]
  },
  {
    "singleItem": [1, {...}]
  },
  {
    "singleItem": [2, {...}]
  },
  ...
]

contains would be:

"anyOf": [
  {
    "singleItem": [0, {...}]
  },
  {
    "singleItem": [1, {...}]
  },
  {
    "singleItem": [2, {...}]
  },
  ...
]

So, if we said that unevaluatedItems does not apply to any index that singleItem returns as an annotation, that is the desired behavior.

Is this more clear?

[jdesrosiers]

Conceptually, unevaluatedItems should work exactly like additionalItems, it just has the extra super power of being able to consider subschemas. Right? If contains can influence unevaluatedItems, it no longer works the same way as additionalItems. Here's an example.

{
  "type": "array",
  "items": [{ "type": "string" }, { "type": "number" }],
  "contains": { "const": 42 },
  "additionalItems": { "type": "number", "maximum": 10 }
}

Example 1: ["foo", 42, 1] => Valid
Example 2: ["foo", 1, 42] => Invalid

If we replace additionalItems with unevaluatedItems, unevaluatedItems's super power is not relevant because there are no subschemas to consider. Therefore, we should expect unevaluatedItems to behave exactly like additionalItems did in the previous schema.

{
  "type": "array",
  "items": [{ "type": "string" }, { "type": "number" }],
  "contains": { "const": 42 },
  "unevaluatedItems": { "type": "number", "maximum": 10 }
}

Assuming contains can impact unevaluatedItems
Example 1: ["foo", 42, 1] => Valid
Example 2: ["foo", 1, 42] => Valid

Notice that the validation result for the second example has changed.

I strongly believe that nothing should change, but if there's consensus that contains should impact unevaluatedItems, then additionalItems should be changed to match that behavior.

[gregsdennis]

Core Specification Section 9.1 (keyword independence):

Schema keywords typically operate independently, without affecting each other's outcomes.

For schema author convenience, there are some exceptions among the keywords in this vocabulary:

• "additionalProperties", whose behavior is defined in terms of "properties" and "patternProperties"
• "unevaluatedProperties", whose behavior is defined in terms of annotations from "properties", "patternProperties", "additionalProperties" and itself
• "additionalItems", whose behavior is defined in terms of "items"
• "unevaluatedItems", whose behavior is defined in terms of annotations from "items", "additionalItems" and itself

This makes no mention of contains.

Section 9.3.1.4 (contains):

The value of this keyword MUST be a valid JSON Schema.

An array instance is valid against "contains" if at least one of its elements is valid against the given schema. Note that when collecting annotations, the subschema MUST be applied to every array element even after the first match has been found. This is to ensure that all possible annotations are collected.

The array is valid, not the element. The subschema must be applied to all elements when collecting annotations, but it does not say that the annotation applies to the element. These two combined implies that the annotations apply to the array.

[Relequestual]

@gregsdennis I'm not quite sure why you're quoting the spec here. We're trying to work out if the spec IS right or needs changes.

@jdesrosiers I can see your point, and conceptually I agree, but in reality I think that IS OK.

After all, the keyword unevaulatedItems is looking to catch items which have not been evaluated and produced annotation results (aka have a pass assertion).

@handrews I think I agree with you here.
For the sake of clarity, can you confirm if you agree with my examples in my previous comments and how they are reasoned?

The example @jdesrosiers provided showing how the *items keywords work differently than one might conceptualize could prove valuable for adding to tests.

[handrews]

I probably agree with @jdesrosiers on

I strongly believe that nothing should change, but if there's consensus that contains should impact unevaluatedItems, then additionalItems should be changed to match that behavior.

at least in theory. Whether that means unevaluatedItems should align with additionalItems after all, or if additionalItems should define behavior w.r.t. contains is less clear to me.

I don't think anyone has ever asked how or if additionalItems and contains interact. I don't recall giving it any thought at all. Mostly because we didn't have a way to talk about how these keywords worked, they just each did stuff.

So, on the plus side, defining consistent mechanisms is helping us figure out how keywords should behave. On the minus side, that's kind of irritating 😛

[gregsdennis]

@Relequestual I'm quoting the spec because my understanding is derived from reading it. (It provides evidence for my arguments.) Ergo, if my understanding is wrong, then the spec needs to change to be more explicit about the intended meaning.

It's always helpful to know what the current state is when deciding whether it should change.

[Relequestual]

you are right @gregsdennis in your reading and understading of the spec 2019-09, but it also says the subschema is applied to each item in the array... but I get your point.

[Relequestual]

I'm marking this as blocked till #864 is resolved.

[handrews]

Good use case from @rjmill

For example, it could be cool to ensure that the “contains” item is in the array, but that everything else validates against another schema.

[handrews]

We decided to go with the two-keyword prefixItems + items, which removes the concern over whether contains needed to impact additionalItems in order to parallel any impact on unevaluatedItems.

This further validates breaking the array/object symmetry by dumping additionalItems. items is defined purely in terms of prefixItems (the items before items) rather than a more general question of what else has been validated (additional beyond anything else in the same schema object). unevaluatedItems is a general "what has been (successfully) validated" keyword.

This means that contains + unevaluatedItems can be used for the "apply a subschema to every item that doesn't match contains" use case in the previous comment, while contains + items can be used for layering an additional contains check on top of a uniform schema for all (or all after prefixItems) positions.

I'm going to write a PR for both this and #864 as they need to rework basically the same areas of the spec.

[handrews]

@gregsdennis ok I lied I have one thing to say since I decided to reread and figure out why I thought this was done.

The array is valid, not the element. The subschema must be applied to all elements when collecting annotations, but it does not say that the annotation applies to the element. These two combined implies that the annotations apply to the array.

The annotations from the subschema apply to the element(s) because the subschema applies to the element(s). That's how annotations work.

The annotation from contains, if any, is associated with the array, just like the annotations from items/prefixItems. These are all the same: they apply one or more subschemas to one or more array elements, providing validation and annotation results for the subschema + instance element. They then combine the validation results (contains ORs them, the others AND them as it the usual behavior), and produce an annotation for the array as a whole indicating what parts of the array the application of the subschema covered.

This is why it never occurred to me that this was unresolved. There's nothing strange about contains except that it ORs instead of ANDs. But that's also true of anyOf vs allOf and we don't do anything strange with anyOf regarding annotations.

[handrews]

@Relequestual this needs a final call ASAP.

[gregsdennis]

I have discussed this thoroughly in Slack in a PM and have come to the conclusion that contains for arrays is analogous to required for objects.

(Validation) 6.5.3. required

The value of this keyword MUST be an array. Elements of this array, if any, MUST be strings, and MUST be unique.

An object instance is valid against this keyword if every item in the array is the name of a property in the instance.

Omitting this keyword has the same behavior as an empty array.

required considers the object as a whole. Consider the case that this keyword only has a single property. If the object does not contain that property, the entire object is marked as failed. If the object does contain this property, then the object acquires an annotation (as opposed to the property that was required).

(Core) 9.3.1.4. contains

The value of this keyword MUST be a valid JSON Schema.

An array instance is valid against "contains" if at least one of its elements is valid against the given schema. Note that when collecting annotations, the subschema MUST be applied to every array element even after the first match has been found. This is to ensure that all possible annotations are collected.

NOTE Italic text is unchanged since the keyword was introduced in Draft 6.

Similarly contains considers the array as a whole. If the array does not contain a matching item, the array is marked as failed. If the array does contain a matching item, then the array acquires an annotation, not the matching item.

The note that was added about annotations states that the contains subschema should be applied to all items to ensure all annotations are collected. However, it does not state that those annotations should apply to the items. The validation result applies to the array, not the items; therefore the annotations should also apply to the array, not the items.

The annotations from the subschema apply to the element(s) because the subschema applies to the element(s). That's how annotations work.

Annotations should be applied as specified by the keyword that produces them. contains is non-specific about annotation application, but it is specific about where the validation result is applied, so annotations should be applied to the same target.

unevaluatedProperties does not consider required; and similarly unevaluatedItems should not consider contains.

If unevaluatedItems should consider contains, then I assert that unevaluatedProperties should consider required, which it currently does not do.

Not only does this interpretation yield symmetry between objects and arrays, it matches more closely to the definition of the word "contains" and how the schema authors (not spec authors; we're a strange lot) would interpret that keyword and the definition that the spec currently gives for it.

---

As a consequence of the realized parallelism with required, this opens a door to contains possibly supporting an array form where all subschemas must have matches within the array. But this is best discussed in another issue.

[notEthan]

I don't think required is a good parallel to contains. it doesn't evaluate any subschema, which seems kind of central to unevaluatedItems. required is more parallel to minItems - they both simply assert the presence of child-instances with given property names / array indices.

I don't think there is a parallel keyword to contains which applies to objects.

[gregsdennis]

Had another discussion, and the point came up that single-form-items represents an AND operation (all items must match) whereas contains represents an OR operation (at least one must match). In that respect, these also have a symmetry.

Building on that symmetry, items attaches annotations to the items in the array, so why shouldn't contains?

I'd be okay with applying annotations at both the array and item level, given this.

[handrews]

@gregsdennis yes, contains is the OR version of items, otherwise they're identical (well, now except that prefixItems only prefixes items).

[notEthan]

I would like to clarify the instance location of the annotations of a child-schema applicator keyword - be it items or contains or properties. I thought this was clear from the spec, but it seems quite muddied from this discussion.

@handrews says (#810 (comment)):

contains applies the subschema to each item in turn, and the annotations from that subschema are most definitely attached to the item. This is exactly the same as what items does.

and @gregsdennis says (#810 (comment))

items attaches annotations to the items in the array

(bolding added)

but the spec seems to me to indicate that the annotation of items is attached to the array, not to its (relevant) items.

json-schema-core.html 9.3.1

This keyword produces an annotation value which is the largest index to which this keyword applied a subschema.

it doesn't quite explicitly say this is attached to the array, but the section on collecting annotations indicates the instance location of an annotation is the one to which the keyword applies. for items that's the array.

moreover, it doesn't make sense to apply the annotations from items or prefixItems or properties to individual items or object values. those annotations indicate where in the array/object a subschema has been evaluated. if a schema properties: {foo: {}, bar: {}} produces an annotation of [foo, bar] against some instance, that annotation doesn't make sense on foo or bar or both, it applies to the object.

same with array subschema keywords producing annotations indicating applied indices, whether it's using true or max index or an array of indices.

this is long-winded for a bit of a sidetrack, but I think it's worth getting everybody on the same page, given how much of the above discussion is hung up on where these annotations apply.