json: $ref + object overhaul (https & recursive $refs, mix properties & allOf)

[ochafik]

Update $ref & object handling in JSON Schema -> GBNF converters:

• Support recursive $refs (fixes Bug: JSON Schema - enum behind a $ref generates an object with unrestricted properties #8073)

• Resolve https $refs in C++ converter when built w/ LLAMA_CURL=1 (caching result in RAM for now, may want to introduce HTTP caching semantics at some point)

  ./llama-cli \
    -j '{"$ref": "https://json.schemastore.org/tsconfig.json"}' \
    -mu https://huggingface.co/bartowski/Meta-Llama-3-8B-Instruct-GGUF/resolve/main/Meta-Llama-3-8B-Instruct-Q8_0.gguf 
    -p "Write a tsconfig.json: "

• Allow mixing properties & anyOf (simplified / unified object conversion code)

  Show example

  echo '{
    "properties": {
      "common": {"$ref": "#/$defs/SomeVal"}
    },
    "allOf": [
        {"$ref": "#/$defs/foo"},
        {"$ref": "#/$defs/bar"},
        {
          "anyOf": [
              {"$ref": "#/$defs/baz"},
              {"$ref": "#/$defs/bam"}
          ]
        }
    ],
    "required": ["common"],
    "$defs": {
        "SomeVal": {"type": "number"},
        "foo": {"properties": {"a": {"$ref": "#/$defs/SomeVal"}}},
        "bar": {"properties": {"b": {"$ref": "#/$defs/SomeVal"}}},
        "bam": {"properties": {"c": {"$ref": "#/$defs/SomeVal"}}},
        "baz": {"properties": {"d": {"$ref": "#/$defs/SomeVal"}}}
    }
  }' | node examples/json_schema_to_grammar.mjs -

• Remove allowFetch options from Python & JS converters (kinda breaking change)

• Align JavaScript conversion CLI (which was only used for tests) with the Python version (both in examples folder)

  # schema.json arg can be "-" for stdin, a URL or a file.
  node   examples/json_schema_to_grammar.mjs schema.json
  python examples/json_schema_to_grammar.py  schema.json

cc/ @HanClinto more JSON :-)