Bug: JSON Schema Not Respected ? #7703
Comments
ArEnSc
added
bug-unconfirmed
medium severity
|
Sampler settings here |
|
json schema: I made a minor addition to the enum and added "lay_down" Input Prompt: this one just kept going on forever, which seems odd, but its very likely has to do with just the schema bugs presented, even though we know that schemas may not terminate, I don't think that this is the case. |
|
More information |
|
@ochafik |
|
I'm still learning about schema-to-grammar conversion, but I figured I would have a go at confirming this report.
Yes, that's correct. There is a CPP program that does this (common/json-schema-to-grammar.cpp), as well as a Python program (examples/json_schema_to_grammar.py). The server uses the CPP route (so these reproduction steps won't be perfect), but we can easily call the Python program thusly: Which results in the output: I then tested this schema in the If I plug your schema and the example output from your report into an online validator, then it tells me: This confirms to me that the problem lies in the json-to-grammar conversion -- specifically in our handling of Note that I also tested all of the above against #6640 (given that it has so many pending updates to schema-to-grammar conversion), but the same problems exist in that branch (indeed, the generated grammar seems to be identical between the branches). Makes me wonder if |
|
I've spent a few hours digging into this problem this evening, and have confirmed your problem. I started trying to wrap your bug in unit tests and uncovered some other issues along the way (see #7789 ). I have not yet added your specific test-case to the integration tests in #7790 , but I still intend to do so. |
|
@ArEnSc combining llama.cpp/common/json-schema-to-grammar.cpp Line 629 in f5d7b26 (tbh it hadn't occurred to me one might want to combine the two, examples I saw weren't doing so, and I thought I'd seen the worst of it w/ tsconfig.json 😅) This limitation should come w/ a warning or error (it's probably a bit hairy to support, and a bit niche), but in the meantime you can adjust your grammar as follows (you already had duplicated {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"actions": {
"type": "array",
"items": {
"oneOf": [
{
"properties": {
"action": {
"const": "go_to"
},
"location": {
"type": "string",
"enum": [
"mall",
"backyard"
]
}
},
"required": [
"action",
"location"
]
},
{
"properties": {
"action": {
"const": "lay_down"
},
"data": {
"type": "string",
"enum": [
"None"
]
}
},
"required": [
"action",
"data"
]
},
{
"properties": {
"action": {
"const": "pick_up_item"
},
"item": {
"type": "string",
"enum": [
"vase",
"coke",
"cake"
]
}
},
"required": [
"action",
"item"
]
},
{
"properties": {
"action": {
"const": "drop_item"
},
"item": {
"type": "string",
"enum": [
"vase",
"coke",
"cake"
]
}
},
"required": [
"action",
"item"
]
}
]
}
}
},
"required": [
"actions"
]
} |
yes this is a good solution thank you! |
|
Closing as this is now covered by grammars/README.md (#7841) :-) |
What happened?
Given this JSON Schema
output:
{"actions": [ {"location":"mall"}, {"item":"cake", "action":"pick_up_item"}, {"item":"cake", "action":"drop_item"} , {"item":"coke", "action":"pick_up_item"}, {"item":"coke", "action":"drop_item"} ] }llama cpp server or llama cpp doesn't respect the JSON schema requiring an action parameter
more over there are a few more instances of this when you try for a longer schema that I can show that it seems to be not the model failing but the json schema
Name and Version
{"tid":"13408","timestamp":1717371771,"level":"INFO","function":"main","line":2918,"msg":"build info","build":2793,"commit":"8f8acc86"}
What operating system are you seeing the problem on?
Windows
Relevant log output
The text was updated successfully, but these errors were encountered: