Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(backend) json-diff evaluator #2001

Merged
merged 7 commits into from
Aug 21, 2024
+20 −19
Merged

fix(backend) json-diff evaluator #2001

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
bc3de96
Merge branch 'main' of github.com:Agenta-AI/agenta
jp-agenta Aug 12, 2024
fe75829
Merge branch 'main' of github.com:Agenta-AI/agenta
jp-agenta Aug 19, 2024
110d2e4
Fix auto_json_diff
jp-agenta Aug 19, 2024
05037a9
refactor (tests): resolve failing unit test for auto_json_diff
aybruhm Aug 20, 2024
916d243
clean up JSON checks
jp-agenta Aug 21, 2024
06012f2
Merge branch 'fix/json-diff-evaluator-issue-with-json-loads' of githu…
jp-agenta Aug 21, 2024
f1f0c42
remove debug print
jp-agenta Aug 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion agenta-backend/agenta_backend/services/evaluators_service.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -621,9 +621,25 @@ def auto_json_diff(
lm_providers_keys: Dict[str, Any], # pylint: disable=unused-argument
) -> Result:
try:
output = output.get("data", "") if isinstance(output, dict) else output

if isinstance(output, dict):
output = json.dumps(output)
elif isinstance(output, str):
try:
json.loads(output)
except:
raise Exception(
f"Evaluator 'auto_json_diff' requires string outputs to be JSON strings."
)
else:
raise Exception(
f"Evaluator 'auto_json_diff' requires the output to be either a JSON string or a JSON object, but received {type(output).__name__} instead."
)

correct_answer = get_correct_answer(data_point, settings_values)
average_score = compare_jsons(
ground_truth=correct_answer,
ground_truth=json.loads(correct_answer),
app_output=json.loads(output),
settings_values=settings_values,
)
Expand Down
21 changes: 3 additions & 18 deletions agenta-backend/agenta_backend/tests/unit/test_evaluators.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,12 +186,7 @@ def test_auto_contains_json(output, expected):
[
(
{
"correct_answer": {
"user": {
"name": "John",
"details": {"age": 30, "location": "New York"},
}
}
"correct_answer": '{"user": {"name": "John", "details": {"age": 30, "location": "New York"}}}'
},
'{"user": {"name": "John", "details": {"age": 30, "location": "New York"}}}',
{
Expand All @@ -205,12 +200,7 @@ def test_auto_contains_json(output, expected):
),
(
{
"correct_answer": {
"user": {
"name": "John",
"details": {"age": 30, "location": "New York"},
}
}
"correct_answer": '{"user": {"name": "John", "details": {"age": "30", "location": "New York"}}}'
},
'{"user": {"name": "John", "details": {"age": "30", "location": "New York"}}}',
{
Expand All @@ -224,12 +214,7 @@ def test_auto_contains_json(output, expected):
),
(
{
"correct_answer": {
"user": {
"name": "John",
"details": {"age": 30, "location": "New York"},
}
}
"correct_answer": '{"user": {"name": "John", "details": {"age": 30, "location": "New York"}}}'
},
'{"USER": {"NAME": "John", "DETAILS": {"AGE": 30, "LOCATION": "New York"}}}',
{
Expand Down
Loading