Infinite Loop in Weather Agent with Missing Forecast Data - due to Field missing in the structured response

[ishswar]

Description:
When querying the weather agent with a city name (e.g., "Buffalo"), the agent repeatedly calls the OpenAI model to generate the weather forecast without stopping. This issue arises because the field summery_of_next_3_days in the WeatherData model relies on the forecasts field, which is not populated, causing the agent to loop continuously.

Even though retries=2 and result_retries=2 are set, these parameters do not prevent the agent from entering this loop, as they seem to be ineffective in controlling the repetitive weather requests.

Steps to Reproduce:

1. Run the provided code with the city "Buffalo".
2. Observe that the system repeatedly makes requests to OpenAI and never completes, continuing to ask for the weather data.

Expected Behavior:
The agent should stop requesting the weather data after the maximum retry attempts have been reached. It should not enter a loop trying to fetch data indefinitely. If the necessary data, like summery_of_next_3_days, is not available due to missing fields, the agent should stop after the retry limit is hit.

Actual Behavior:
The agent continuously sends requests and attempts to populate the summery_of_next_3_days field without having the forecasts data, resulting in multiple requests that do not stop.

Error Logs:

2024-12-15 22:26:23,501 - PydanticAI version: 0.0.12
2024-12-15 22:26:23,501 - OpenAI version: 1.57.2
/bug.py:179: LogfireNotConfiguredWarning:

No logs or spans will be created until `logfire.configure()` has been called. Set the environment variable LOGFIRE_IGNORE_NO_CONFIG=1 or add ignore_no_config=true in pyproject.toml to suppress this warning.

2024-12-15 22:26:24,543 - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-12-15 22:26:24,555 - Getting lat and lng for Buffalo and Country US
2024-12-15 22:26:26,223 - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-12-15 22:26:26,226 - Getting weather for Buffalo in US with temperature unit C
2024-12-15 22:26:26,226 - Thoughts: Weather forecast for the next three days.
2024-12-15 22:26:27,195 - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-12-15 22:26:27,199 - Getting lat and lng for Buffalo and Country US
2024-12-15 22:26:28,693 - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-12-15 22:26:28,696 - Getting weather for Buffalo in United States with temperature unit C
2024-12-15 22:26:28,696 - Thoughts: Weather forecast for the next three days.
2024-12-15 22:26:33,854 - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-12-15 22:26:33,857 - Getting weather for Buffalo in US with temperature unit C
2024-12-15 22:26:33,858 - Thoughts: Weather forecast for the next three days.
2024-12-15 22:26:35,513 - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-12-15 22:26:35,520 - Getting weather for Buffalo in United States with temperature unit C
2024-12-15 22:26:35,520 - Thoughts: Weather forecast for the next three days.
2024-12-15 22:26:37,127 - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-12-15 22:26:37,131 - Getting weather for Buffalo in US with temperature unit C
2024-12-15 22:26:37,131 - Thoughts: Weather forecast for Buffalo, NY.
2024-12-15 22:26:38,678 - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-12-15 22:26:38,682 - Getting weather for Buffalo in United States with temperature unit C
2024-12-15 22:26:38,682 - Thoughts: Requesting latest weather data.
2024-12-15 22:26:40,038 - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-12-15 22:26:40,042 - Getting weather for Buffalo in United States with temperature unit C
2024-12-15 22:26:40,042 - Thoughts: Weather forecast for Buffalo, NY.
2024-12-15 22:26:41,542 - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-12-15 22:26:41,546 - Getting weather for Buffalo in US with temperature unit C
2024-12-15 22:26:41,546 - Thoughts: Requesting weather information.
2024-12-15 22:26:43,072 - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-12-15 22:26:43,076 - Getting weather for Buffalo in United States with temperature unit C
2024-12-15 22:26:43,076 - Thoughts: Weather forecast inquiry.
2024-12-15 22:26:44,657 - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-12-15 22:26:44,661 - Getting weather for Buffalo in US with temperature unit C
2024-12-15 22:26:44,661 - Thoughts: Get the weather data.
2024-12-15 22:26:46,249 - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-12-15 22:26:46,262 - Getting weather for Buffalo in US with temperature unit C
2024-12-15 22:26:46,262 - Thoughts: Requesting weather information for Buffalo.
2024-12-15 22:26:47,989 - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-12-15 22:26:47,994 - Getting weather for Buffalo in US with temperature unit C
2024-12-15 22:26:47,994 - Thoughts: Final weather request for Buffalo.
2024-12-15 22:26:49,597 - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-12-15 22:26:49,602 - Getting weather for Buffalo in US with temperature unit C
2024-12-15 22:26:49,602 - Thoughts: Weather forecast for Buffalo, NY.
2024-12-15 22:26:51,063 - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-12-15 22:26:51,067 - Getting weather for Buffalo in US with temperature unit C
2024-12-15 22:26:51,067 - Thoughts: Latest weather for Buffalo.
2024-12-15 22:26:52,553 - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-12-15 22:26:52,556 - Getting weather for Buffalo in US with temperature unit C
2024-12-15 22:26:52,556 - Thoughts: Requesting weather overview for the next three days.
2024-12-15 22:26:54,352 - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-12-15 22:26:54,356 - Getting weather for Buffalo in United States with temperature unit C
2024-12-15 22:26:54,356 - Thoughts: Current weather data.
2024-12-15 22:26:55,946 - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-12-15 22:26:55,950 - Getting weather for Buffalo in United States with temperature unit C
2024-12-15 22:26:55,950 - Thoughts: Weather forecast for Buffalo, NY.
2024-12-15 22:26:57,499 - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-12-15 22:26:57,504 - Getting weather for Buffalo in US with temperature unit C
2024-12-15 22:26:57,504 - Thoughts: Requesting weather forecasts.
2024-12-15 22:26:58,837 - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-12-15 22:26:58,839 - Getting weather for Buffalo in US with temperature unit C
2024-12-15 22:26:58,839 - Thoughts: Current weather.
2024-12-15 22:27:00,313 - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-12-15 22:27:00,317 - Getting weather for Buffalo in US with temperature unit C
2024-12-15 22:27:00,317 - Thoughts: Weather forecast inquiry.
Traceback (most recent call last):
  File "/bug.py", line 190, in <module>
    asyncio.run(main())
  File "miniconda3/envs/pydantic_ai/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "miniconda3/envs/pydantic_ai/lib/python3.10/asyncio/base_events.py", line 636, in run_until_complete
    self.run_forever()
  File "/miniconda3/envs/pydantic_ai/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
    self._run_once()
  File "miniconda3/envs/pydantic_ai/lib/python3.10/asyncio/base_events.py", line 1871, in _run_once
    event_list = self._selector.select(timeout)
  File "miniconda3/envs/pydantic_ai/lib/python3.10/selectors.py", line 562, in select
    kev_list = self._selector.control(None, max_ev, timeout)
KeyboardInterrupt

Environment:

• Python version: 3.11
• Pydantic AI version: 0.0.12
• OpenAI API version: 1.57.2
• Dependencies: openai, pydantic, pydantic_ai, httpx, etc.

Code

from __future__ import annotations as _annotations

import asyncio
import logging
import os
import random
import time
import uuid
from dataclasses import dataclass
from datetime import datetime, timedelta
from typing import Any, List, Optional, Literal

import dotenv
import openai
import pydantic_ai
from httpx import AsyncClient
from pydantic import BaseModel
from pydantic import Field
from pydantic_ai import Agent, ModelRetry

# Set up logging configuration
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(message)s',  # This adds the timestamp to each log message
    handlers=[logging.StreamHandler()]  # This ensures logs are shown in the console
)

# Example usage
logger = logging.getLogger(__name__)

# load env
dotenv.load_dotenv()

# 'if-token-present' means nothing will be sent (and the example will work) if you don't have logfire configured
# logfire.configure()

class HourlyForecast(BaseModel):
    time: str  # Formatted time (e.g., "12:00 PM")
    temperature: float  # Temperature in the chosen unit
    description: str  # Weather description
    emoji: str  # Emoji representing the weather kind

class DailyForecast(BaseModel):
    date: str  # Formatted date (e.g., "Monday, December 11, 2024")
    average_temperature: float  # Average temperature for the day
    hourly_forecasts: List[HourlyForecast]  # List of hourly forecasts



class WeatherData(BaseModel):
    city: str = Field(..., description="City for which the weather data is retrieved")
    current_temperature: float = Field(..., description="Current temperature in the chosen unit")
    temp_unit: str = Field(..., description="Temperature unit (C or F)")
    latitude: Optional[float] = Field(None, description="Latitude of the city")
    longitude: Optional[float] = Field(None, description="Longitude of the city")
    # forecasts: List[DailyForecast] = Field(..., description="List of daily forecasts")
    country: Optional[str] = Field(None, description="Country of the city")
    region: Optional[str] = Field(None, description="Region of the city")
    unknown_city: Optional[bool] = Field(None, description="True if the city is unknown")
    reason: Optional[str] = Field(None, description="Reason for the unknown city")
    country_emoji: Optional[str] = Field(None, description="Emoji representing the country's flag")
    summery_of_next_3_days: Optional[str] = Field(None, description="Summery of next 3 days weather")

@dataclass
class Deps:
    client: AsyncClient
    conversation_id: uuid.UUID = None

weather_agent = Agent(
    'openai:gpt-4o-mini',
system_prompt=(
    "Be concise and respond in one sentence for 'summary_of_next_3_days'. Ensure that you directly address the user's question in the summary. "
    "For example: if the user asks 'Is it cloudy in Sydney?', and it is or will be cloudy, respond with 'Yes, it's cloudy'; otherwise, say 'No'. "
    "IMPORTANT: If the user is asking for the weather in a specific city or country, summarize the weather for the next three days in the 'summary_of_next_3_days' field and generate the forecast for those days. "
    "If the user provides just a city name without a specific question, summarize the weather for the next three days in 'summary_of_next_3_days' as a short 2-3 line description of the weather for that location. "
    "NEVER use your own knowledge of the world weather data as it might be outdated, and it was collected during your training. "
    "Always use the provided tools to fetch the latest weather data. DO NOT rely on your internal knowledge for weather-related responses; always use the tools provided to fetch the most up-to-date information."
    )
    ,
    deps_type=Deps,
    retries=2,
    result_type=WeatherData,
    result_retries=2,
)


async def get_weather_call(city_name: str, country_name: str | None, temp_unit_c_or_f: str) -> WeatherData:
    """Get the weather information for a given city."""
    forecasts = []
    current_date = datetime.now()
    for _ in range(3):
        daily_forecast = DailyForecast(
            date=current_date.strftime("%A, %B %d, %Y"),
            average_temperature=round(random.uniform(-10, 35), 1),
            hourly_forecasts=[
                HourlyForecast(
                    time=(current_date + timedelta(hours=i)).strftime("%I:%M %p"),
                    temperature=round(random.uniform(-10, 35), 1),
                    description=random.choice(["Sunny", "Cloudy", "Rainy", "Snowy"]),
                    emoji=random.choice(["☀️", "☁️", "🌧️", "❄️"])
                )
                for i in range(8)
            ]
        )
        forecasts.append(daily_forecast)
        current_date += timedelta(days=1)

    weather_ret_data = WeatherData(
        city=city_name,
        current_temperature=round(random.uniform(-10, 35), 1),
        temp_unit=temp_unit_c_or_f,
        # forecasts=forecasts,
        country=country_name,
        region="New York",
        country_emoji="🇺🇸"
    )
    return weather_ret_data

@weather_agent.tool_plain
async def get_lat_lng(city_name: str , iso_3166_alpha_2_country_name: str | None) -> dict[str, Any]:
    """Get the latitude and longitude of a given city."""
    logger.info(f"Getting lat and lng for {city_name} and Country {iso_3166_alpha_2_country_name}")
    try:
        # fake lat and lang
        lat = random.uniform(-90, 90)
        lng = random.uniform(-180, 180)
        return {'lat' : lat, 'lang': lng}
    except Exception as e:
        logger.error(f'Error getting lat and lng: {e}', exc_info=True)
        raise ModelRetry('Could not find the location')

# temperature_unit_celsius_or_fahrenheit = 'C' or 'F'

@weather_agent.tool_plain
async def getweather(
    city_name: str,
    country_name: str | None,
    temperature_unit_celsius_or_fahrenheit: Literal['C', 'F'],
    latitude: str,
    longitude: str,
    thoughts: str
) -> str:
    """Get the weather data for a given city."""

    logger.info(f"Getting weather for {city_name} in {country_name} with temperature unit {temperature_unit_celsius_or_fahrenheit}")
    # print thoughts
    logger.info(f"Thoughts: {thoughts}")
    # Await the cached result of get_weather_call
    weather = await get_weather_call(city_name, country_name, temperature_unit_celsius_or_fahrenheit)


    ret_data = WeatherData(
        city=city_name,
        # forecasts=weather.forecasts,
        current_temperature=weather.current_temperature,
        temp_unit=temperature_unit_celsius_or_fahrenheit,
        country=weather.country,
        region=weather.region,
        latitude=float(latitude),
        longitude=float(longitude),
    )
    #    debug(ret_data)
    return ret_data


async def main():
    async with AsyncClient() as client:
        # Print pydanticai and OpneAI version
        logger.info(f"PydanticAI version: {pydantic_ai.__version__}")
        logger.info(f"OpenAI version: {openai.__version__}")
        dotenv.load_dotenv()
        conversation_id = uuid.uuid4()
        deps = Deps(
            client=client,
            conversation_id=conversation_id
        )

        start_time = time.time()
        result = await weather_agent.run(
            'What is the weather like in Buffalo?', deps=deps
        )
        end_time = time.time()
        print(f"Time taken: {end_time - start_time:.2f} seconds")
        # debug(result)
        print('Response:', result.data.model_dump_json(indent=2))



if __name__ == '__main__':
    asyncio.run(main())

[samuelcolvin]

Thanks for reporting should fixed (or at least mitigated) by #70 which I know is urgent.