[Enhancement] Add LLM API key checks to LLM-based evaluators

[aybruhm]

Description

This PR enhances the backend by adding checks to ensure that an OpenAI API key is present for LLM-based evaluators, with clear exception messages if the key is missing. Additionally, test coverage has been expanded to include scenarios where the OpenAI API key is required, including a new test case for auto_ai_critique.

Related Issue

Closes AGE-532 & AGE-569

Acceptance Tests

Test 1: OpenAI API Key is Required for LLM-Based Evaluators

• Precondition: Ensure no OpenAI API key is set in the LLM Keys view on the frontend.
• Action:
  1. Run an LLM-based evaluator (e.g., auto_ai_critique) without an OpenAI API key.
• Expected Outcome:
  • The evaluator should raise a clear and informative exception message indicating that an OpenAI API key is missing.

Test 2: LLM-based Evaluator Runs Successfully with a Valid API Key

• Precondition: Ensure a valid OpenAI API key is set in the LLM Keys view on the frontend.
• Action:
  1. Run an LLM-based evaluator (e.g., auto_ai_critique) with the valid OpenAI API key.
• Expected Outcome:
  • The evaluator should run without errors.
  • The output should be consistent with expected results based on the inputs provided.

Test 3 (automated): Test for AI Critique (LLM as a Judge) API Key Checks

• Precondition: Ensure the test suite includes scenarios where the OpenAI API key is required.
• Action:
  1. This will be run automatically by GitHub.
• Expected Outcome:
  • The test suite should pass, covering scenarios with and without an OpenAI API key.
  • The new test case for auto_ai_critique should be included and verified as part of the coverage.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
agenta	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Aug 26, 2024 1:16pm
agenta-documentation	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Aug 26, 2024 1:16pm

[jp-agenta]

Thanks @aybruhm !
There is one thing missing though.
See 👇

agenta/agenta-backend/agenta_backend/routers/evaluation_router.py

Line 115 in d8a1bbd

success, response = await check_ai_critique_inputs(

We'd need to update that so that we immediately check for API keys for any LLM-based evaluator, not just AI critique. Does it make sense ?

[jp-agenta]

Left one suggestion to centralise evaluator info.

[jp-agenta]

This can't go to QA until we fix the issue whereby the LLM doesn't respond in numeric format anymore.
-- @mmabrouk @aybruhm

[aybruhm]

This can't go to QA until we fix the issue whereby the LLM doesn't respond in numeric format anymore. -- @mmabrouk @aybruhm

It works now, and I didn’t change anything. Safe to say the language model was probably hallucinating at the time, right?

[aybruhm]

QA Report:

https://agenta-hq.slack.com/archives/C05JT3NQB1R/p1724832222331489?thread_ts=1724757298.386189&cid=C05JT3NQB1R

[zenUnicorn]

I have QA the PR and I got the expected result ✅
Works fine!