Function calling only considers functions in the first tool?

[MrCsabaToth]

Description of the bug:

Using gemini-1.5-flash.
Prompt: "What is the weather today?"

Tool configuration 1:

[
  Tool(
    functionDeclarations: [
      FunctionDeclaration(
        'fetchWeatherForecast',
        'Returns the weather in a given location.',
        Schema(
          SchemaType.object,
          properties: {
            'latitude': Schema.number(
              description: 'Latitude of the weather observation and forecast',
            ),
            'longitude': Schema.number(
              description:
              'Longitude of the weather observation and forecast',
            ),
          },
          requiredProperties: ['latitude', 'longitude'],
        ),
      ),
    ],
  ),
  Tool(
    functionDeclarations: [
      FunctionDeclaration(
        'fetchGpsLocation',
        'Returns the latitude and longitude of the current GPS location.',
        Schema(SchemaType.string),
      ),
      FunctionDeclaration(
        'fetchHeartRate',
        'Returns the current heart rate measurement.',
        Schema(SchemaType.integer),
      ),
    ],
  ),
]

Response 1: I need to know your location to get the weather forecast. Can you tell me your latitude and longitude?

Tool configuration 2 (switching the order of the two tools):

[
  Tool(
    functionDeclarations: [
      FunctionDeclaration(
        'fetchGpsLocation',
        'Returns the latitude and longitude of the current GPS location.',
        Schema(SchemaType.string),
      ),
      FunctionDeclaration(
        'fetchHeartRate',
        'Returns the current heart rate measurement.',
        Schema(SchemaType.integer),
      ),
    ],
  ),
  Tool(
    functionDeclarations: [
      FunctionDeclaration(
        'fetchWeatherForecast',
        'Returns the weather in a given location.',
        Schema(
          SchemaType.object,
          properties: {
            'latitude': Schema.number(
              description: 'Latitude of the weather observation and forecast',
            ),
            'longitude': Schema.number(
              description:
              'Longitude of the weather observation and forecast',
            ),
          },
          requiredProperties: ['latitude', 'longitude'],
        ),
      ),
    ],
  ),
]

Response 2:
I am sorry, I cannot fulfill this request. I do not have access to weather information.

Artificially unifying the two tools into a virtual one, merging all functions under that:

[
  Tool(
    functionDeclarations: [
      FunctionDeclaration(
        'fetchWeatherForecast',
        'Returns the weather in a given location.',
        Schema(
          SchemaType.object,
          properties: {
            'latitude': Schema.number(
              description: 'Latitude of the weather observation and forecast',
            ),
            'longitude': Schema.number(
              description:
              'Longitude of the weather observation and forecast',
            ),
          },
          requiredProperties: ['latitude', 'longitude'],
        ),
      ),
      FunctionDeclaration(
        'fetchGpsLocation',
        'Returns the latitude and longitude of the current GPS location.',
        Schema(SchemaType.string),
      ),
      FunctionDeclaration(
        'fetchHeartRate',
        'Returns the current heart rate measurement.',
        Schema(SchemaType.integer),
      ),
    ],
  ),
]

The model properly asks first for the location tool for the lat / lon coordinates, and then with a second round of call (where I pass the location function call result down) it properly asks to call the weather function with the proper GPS coordinates.

Respnose: The weather today is clear and the temperature is 25 degrees Celsius. The wind is blowing from the southwest at 2 meters per second.

Actual vs expected behavior:

I'd expect to be able to keep the two tools apart for extensible software architecture. So far I'm aiming for 8 tools, many of those have multiple functions.

Any other information you'd like to share?

I peeked at the code and it looks to me that ultimately a REST API call is made. The _generateContentRequest seems to serialize the tools OK (by looking at it) if (tools != null) 'tools': tools.map((t) => t.toJson()).toList(),

generative-ai-dart/pkgs/google_generative_ai/lib/src/model.dart

Line 333 in 8a58d77

if (tools != null) 'tools': tools.map((t) => t.toJson()).toList(),

so I don't know yet where's the problem. Maybe in my code?

[MrCsabaToth]

Correction: this is with Gemini 1.5 Flash, I'll experiment with Pro again, possibly file another issue (cannot deal with more than 8 functions)

[MrCsabaToth]

Something which could affect this bug: while I was converting over the code to use https://pub.dev/packages/firebase_vertexai I got the error Unable to submit request because function parameters schema should be of type OBJECT. Learn more: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling. The two affected functions indeed have non object SchemaType, because the heart rate is an integer (SchemaType.integer) and I declare the location as a SchemaType.string which would just contain the two coordinates. And neither of the functions have any input parameters.

[Arkangel12]

I am looking for the right way to load a function calling with Flutter/Dart. I have done it as follows:

In a nutshell, am I doing the configurations wrong, or is something missing in this API for function calling? 🤔

I am using this package google_generative_ai.

Option One

GenerativeModel(
      model: 'gemini-2.0-flash-exp',
      apiKey: 'apiKey',
      systemInstruction: Content(
        'model',
        [TextPart('prompt')],
      ),
      tools: [
        Tool(
          functionDeclarations: [
            FunctionDeclaration(
              'returnEngagementRate',
              'Returns the engagement percentage.',
              Schema.object(
                properties: {
                  'engagementRate': Schema.integer(
                    description: 'The engagement percentage of the brand between 0 and 100.',
                  ),
                },
              ),
            ),
          ],
          codeExecution: CodeExecution(),
        ),
      ],
      generationConfig: GenerationConfig(maxOutputTokens: 200),
    ); 

Response:

Option Two

GenerativeModel(
      model: 'gemini-2.0-flash-exp',
      apiKey: 'apiKey',
      systemInstruction: Content(
        'model',
        [TextPart('prompt')],
      ),
      tools: [
        Tool(
          functionDeclarations: [
            FunctionDeclaration(
              'returnEngagementRate',
              'Returns the engagement percentage.',
              Schema(
                 SchemaType.object,
                 description: 'The engagement percentage of the brand between 0 and 100.',
                 properties: {
                   'Low': Schema(
                     SchemaType.integer,
                     description: 'Indicates if the engagement is bellow the average',
                   ),
                   'Average': Schema(
                     SchemaType.integer,
                     description: 'Engagement is average for the brands standards',
                   ),
                   'Excellent': Schema(
                     SchemaType.integer,
                     description: 'Engagement is over the brands average',
                   ),
                 },
               ),
            ),
          ],
          codeExecution: CodeExecution(),
        ),
      ],
      generationConfig: GenerationConfig(maxOutputTokens: 200),
    ); 

Response: