must be specified when not using one_of using zod.union() on google provider

[p3droml]

Description

Related issue: #4153

Error doesn't happen if structuredOutputs: false

Code example

const model = google('gemini-2.0-flash', { structuredOutputs: true })

const mainSchema = z.array(z.union([subSchema1, subSchema2]))

const response = await generateObject({
model: this.model,
messages,
schema: z.object({
chain_of_thought: z.string(),
locator: mainSchema,
}),
});

AI provider

@ai-sdk/google@1.1.10

Additional context

No response

[natac13]

I have seen similar with enums. I tried to use the new gemini-2.0-flash-001 model in place of where Im using open ai to generate and object that has an property that is an array of objects value; those objects have an enum. And I got an error about invalid JSON input. I tried the structureOutputs: false and it ran but still failed the zod schema validation.

For reference my schema looks like the following:

const MockSchema = z.object({
  fieldA: z.string(),
  fieldB: z.number(),
  fieldC: z.array(z.object({
    fieldD: z.enum(['a', 'b', 'c']),
  }))
})

And the error looks like:

responseBody: '{\n' +
    '  "error": {\n' +
    '    "code": 400,\n' +
    `    "message": "Invalid JSON payload received. Unknown name \\"type\\" at 'generation_config.response_schema.proper
ties[2].value.items.properties[2].value.any_of[0].properties[2].value.items.properties[2].value': Proto field is not rep
eating, cannot start list.\\nInvalid value at 'generation_config.response_schema.properties[2].value.items.properties[2]
.value.any_of[1].type' (type.googleapis.com/google.ai.generativelanguage.v1beta.Type), \\"null\\"",\n` +
    '    "status": "INVALID_ARGUMENT",\n' +
    '    "details": [\n' +
    '      {\n' +
    '        "@type": "type.googleapis.com/google.rpc.BadRequest",\n' +
    '        "fieldViolations": [\n' +
    '          {\n' +
    '            "field": "generation_config.response_schema.properties[2].value.items.properties[2].value.any_of[0].pro
perties[2].value.items.properties[2].value",\n' +
    `            "description": "Invalid JSON payload received. Unknown name \\"type\\" at 'generation_config.response_s
chema.properties[2].value.items.properties[2].value.any_of[0].properties[2].value.items.properties[2].value': Proto fiel
d is not repeating, cannot start list."\n` +
    '          },\n' +
    '          {\n' +
    '            "field": "generation_config.response_schema.properties[2].value.items.properties[2].value.any_of[1].typ
e",\n' +
    `            "description": "Invalid value at 'generation_config.response_schema.properties[2].value.items.propertie
s[2].value.any_of[1].type' (type.googleapis.com/google.ai.generativelanguage.v1beta.Type), \\"null\\""\n` +
    '          }\n' +
    '        ]\n' +
    '      }\n' +
    '    ]\n' +
    '  }\n' +
    '}\n',

Where the type the error mentions is the fieldD from the example

I know OpenAI struggles with some complex zod schema which I have to use Anthropic to handle; ie default(), discriminatedUnion(). So maybe Google has similar limitation?

[williamlmao]

I am also running into the same error: "Invalid value at 'generation_config.response_schema.properties[1].value.any_of[1].type' (type.googleapis.com/google.ai.generativelanguage.v1beta.Type)"

I think @natac13 is right that the enums are the issue - my type is an enum. type: z.enum(['melee', 'ranged', 'spell'])

[lgrammel]

Does this only happen on the 2.0 models or also on 1.5 models?

[williamlmao]

@lgrammel Getting it on all google models - just tested 1.5 pro and 1.5 flash, same thing.

I'm also getting a * GenerateContentRequest.contents: contents is not specified error on generateText requests.

[lgrammel]

I wonder if they changed something - was existing code suddenly broken or are these new schemas?

[williamlmao]

I'm not sure, this is actually my first time using the google models so I can't say if it was working before or not - sorry I can't be of more help here!

[p3droml]

I just found this thread: https://discuss.ai.google.dev/t/oneof-in-response-schema/55926

It seems like one_of (zod union) is currently not supported by Google, though it's supported by OpenAI.

[natac13]

Does this only happen on the 2.0 models or also on 1.5 models?

I was trying out gemini-2.0 only.

I wonder if they changed something - was existing code suddenly broken or are these new schemas?

Not new schema. Just a schema that was working with 4o and 4o-mini, o1 also fails on the comlexitity of the schema.

I have found that openai does the 'best' with a huge generation with a somewhat simple schema. It can contain enums though.

When I need .default() or discriminatedUnion from zod even openai 4o fails with a hard error on the schema. It does mention it in their docs though.
That is when I realized Anthropic 3.5 models can handle detailed schemas but have trouble generating a lot of info from those schemas.

[desiprisg]

Optimal DX in this case would be for generateObject to always accept a valid zod schema and transform based on the provider and their specs. If oneOf is not supported, it should not be included in the request as such, but as instructions instead (possibly as description, not sure how it's currently implemented for models that don't support structured outputs), and the error should be thrown only if the model generated a response that could not be validated against the schema (https://sdk.vercel.ai/docs/ai-sdk-core/generating-structured-data#error-handling).

This approach is consistent with the ai sdks mission (https://sdk.vercel.ai/docs/introduction#why-use-the-ai-sdk).