Code

[feature] Evaluators - Debugging#2071

Code

Merged

mmabrouk merged 193 commits intomainAgenta-AI/agenta:mainfrom AGE-587/-implement-evaluation-main-pageAgenta-AI/agenta:AGE-587/-implement-evaluation-main-pageCopy head branch name to clipboardSep 23, 2024

+4,685-2,842

Commits

Commits on Aug 1, 2024

• feat (backend): add utility function to ensure event loop is retrieved or created

  aybruhm

  committed
  5c5ad15

• feat (backend): add endpoint to run evaluation on a specific evaluator

  aybruhm

  committed
  60064ef

• refactor (backend): make use of ensure_event_loop utility function

  aybruhm

  committed
  0a66e26

Commits on Aug 2, 2024

• Merge branch 'rag' into feature/age-491-poc-1e-expose-running-evaluators-via-api-to-playground

  aybruhm

  committed
  7d534fc

• docs (backend): improve docstring in ensure_event_loop function

  aybruhm

  committed
  9ba496f

Commits on Aug 4, 2024

• minor refactor (build): replace use of 'docker-compose' to 'docker compose'

  aybruhm

  committed
  bd9b2d3

Commits on Aug 8, 2024

• feat (backend): created evaluator mapping and input interfaces

  aybruhm

  committed
  3f6b507

• feat (backend): implemented endpoints to map experiment data tree to evaluator interface and modified endpoint to evaluate llm app run

  aybruhm

  committed
  4a1604d

• refactor (backend): update evaluator handlers to make use of new handlers

  aybruhm

  committed
  55c727e

Commits on Aug 9, 2024

• Merge branch 'rag' into feature/age-491-poc-1e-expose-running-evaluators-via-api-to-playground

  aybruhm

  committed
  165f7a7

• chore (backend): remove interfaces that are redundant

  aybruhm

  committed
  6fb0b02

Commits on Aug 12, 2024

• refactor (backend): convert evaluator functions to asynchronous

  aybruhm

  committed
  0facf64

• refactor (backend): remove event_loop_utils module

  aybruhm

  committed
  b81d01b

• refactor (backend): improve run evaluator endpoint to be asynchronous

  aybruhm

  committed
  53bb2ed

• refactor (backend): update rag faithfulness and context relevancy evaluators

  aybruhm

  committed
  6f876f4

• refactor (backend): ensure rag evaluators is compatible with evaluate and run interface handlers

  aybruhm

  committed
  5214108

• refactor (tests): convert unit tests to async using pytest.mark.asyncio

  aybruhm

  committed
  133fc73

• refactor (backend): resolve pr#1956 comments (r1714066376, r1714068517, r1714064544, r1714061425)

  aybruhm

  committed
  7e229b3

• minor refactor (backend): remove redundant 'get_nested_value' function

  aybruhm

  committed
  bbbb925

Commits on Aug 13, 2024

• Merge branch 'main' into feature/age-491-poc-1e-expose-running-evaluators-via-api-to-playground

  aybruhm

  committed
  58760ee

• refactor (backend): run evaluator_service 'evaluate' asynchronous from celery task

  aybruhm

  committed
  dd805a2

• refactor (backend): improve error handling for auto_contains_json evaluator

  aybruhm

  committed
  d3f7315

• feat (tests): add tests for dictionary-based output handling in contains_json evaluator

  aybruhm

  committed
  08b9e87

Commits on Aug 14, 2024

• refactor (backend): add check for OpenAI API key with clear exception message

  aybruhm

  committed
  28320b8

• feat (tests): add test case for auto_ai_critique and evaluators requiring OpenAI API key

  aybruhm

  committed
  d8a1bbd

Commits on Aug 18, 2024

• feat (tests): added mock trace data for a simple finance assisstant

  aybruhm

  committed
  9c10025

• feat (tests): created fixtures for evaluator experiment tree map and run endpoints

  aybruhm

  committed
  03e28ee

• feat (tests): created tests for evaluator experiment tree map and run endpoint

  aybruhm

  committed
  6b6b6d6

• Merge branch 'main' into feature/age-491-poc-1e-expose-running-evaluators-via-api-to-playground

  aybruhm

  authored
  b808342

Commits on Aug 19, 2024

• refactor (backend): rewrite db function to check if evaluators exist in evaluators

  aybruhm

  committed
  e02fefa

• chore (backend): remove deprecated function 'check_ai_critique_inputs'

  aybruhm

  committed
  4cee49f

• feat (backend): implemented helper functions to:

  Show description for c6ee3c8
  aybruhm

  committed
  c6ee3c8

• refactor (backend): update evaluator_router to:

  Show description for a8c1273
  aybruhm

  committed
  a8c1273

Commits on Aug 20, 2024

• feat (tests): added test to create evaluation with no llm keys

  aybruhm

  committed
  f3367ef

• refactor (backend): added

  Show description for c499a19
  aybruhm

  committed
  c499a19

• Merge branch 'main' into feature/age-532-poc-1e-add-llm-api-key-checks-in-llm-based-evaluators

  aybruhm

  committed
  cc90567

• Merge branch 'main' into feature/age-491-poc-1e-expose-running-evaluators-via-api-to-playground

  aybruhm

  committed
  f26dd78

• Merge branch 'feature/age-491-poc-1e-expose-running-evaluators-via-api-to-playground' into feature/age-532-poc-1e-add-llm-api-key-checks-in-llm-based-evaluators

  aybruhm

  committed
  7197942

• Merge branch 'feature/age-491-poc-1e-expose-running-evaluators-via-api-to-playground' into feature/age-573-evaluators-fail-gracefully-when-we-send-a-dict-to-a-str-only

  aybruhm

  committed
  f0cc8c6

• chore (backend): remove redundant error message

  aybruhm

  committed
  d1fe5aa

• chore (backend): cleanup in levenshtein distance evaluato

  aybruhm

  committed
  169a994

• Merge branch 'feature/age-491-poc-1e-expose-running-evaluators-via-api-to-playground' into feature/age-573-evaluators-fail-gracefully-when-we-send-a-dict-to-a-str-only

  aybruhm

  committed
  05ae4b5

Commits on Aug 21, 2024

• Merge branch 'main' into feature/age-491-poc-1e-expose-running-evaluators-via-api-to-playground

  aybruhm

  committed
  3fe306f

• Merge branch 'feature/age-491-poc-1e-expose-running-evaluators-via-api-to-playground' into feature/age-573-evaluators-fail-gracefully-when-we-send-a-dict-to-a-str-only

  aybruhm

  committed
  ac1ac7e

• style (website): format cookbooks with black@23.12.0

  aybruhm

  committed
  c663fb4

• Merge branch 'feature/age-491-poc-1e-expose-running-evaluators-via-api-to-playground' into feature/age-573-evaluators-fail-gracefully-when-we-send-a-dict-to-a-str-only

  aybruhm

  committed
  9309f43

• refactor (backend): centralize validation of string and json output and use functions in evaluators

  aybruhm

  committed
  23be8b6

• feat (tests): update parameters for BaseResponse compatibility and reflect changes in test cases

  Show description for b6db4f1
  aybruhm

  committed
  b6db4f1

• minor refactor (backend): update 'validate_json_output' function return-type and docstring

  aybruhm

  committed
  80f3eff

• chore (style): format evaluators_service with black@23.12.0

  aybruhm

  committed
  892a351

• Merge branch 'feature/age-491-poc-1e-expose-running-evaluators-via-api-to-playground' into feature/age-532-poc-1e-add-llm-api-key-checks-in-llm-based-evaluators

  aybruhm

  committed
  a4ecc3c

Commits on Aug 22, 2024

• refactor (backend): clean up LLM key checks in evaluators

  Show description for 33e6e17
  aybruhm

  committed
  33e6e17

• chore (tests): add '@pytest.mark.asyncio' to test cases in test_user_profile

  aybruhm

  committed
  7c28f6d

Commits on Aug 23, 2024

• Enforce in Union[str, Dict[str, Any]] in BaseResponse in SDK

  jp-agenta

  committed
  3cad5db

• fix ai critique

  jp-agenta

  committed
  91d23d8

• initial commit: setup configure evaluator modal

  bekossy

  committed
  cd2546a

• minor refactor (backend): include ai_critique evaluator settings_values to EvaluatorInputInterface

  aybruhm

  committed
  31d10e1

• chore (style): format evaluators_service with black@23.12.0

  aybruhm

  committed
  b224f10

• minor refactor (backend): resolve ValueError when casting string to float for ai critique evaluator

  aybruhm

  committed
  ca81cea

• Merge branch 'main' of github.com:Agenta-AI/agenta

  jp-agenta

  committed
  6238cd8

• Merge branch 'main' into feature/age-573-evaluators-fail-gracefully-when-we-send-a-dict-to-a-str-only

  jp-agenta

  committed
  f3546ef

• fix exception message and bump SDK out of pre-release

  jp-agenta

  committed
  2402f94

• Merge pull request #1987 from Agenta-AI/feature/age-573-evaluators-fail-gracefully-when-we-send-a-dict-to-a-str-only

  Show description for 532a4bb
  jp-agenta

  authored
  532a4bb

• Merge branch 'main' into AGE-587/-implement-evaluation-main-page

  bekossy

  committed
  824c96e

Commits on Aug 24, 2024

• improved file structure(frontend)

  bekossy

  committed
  c160b72

• design(frontend): added evaluator modal component steps

  bekossy

  committed
  d5eb285

• fix(frontend): passed prop

  bekossy

  committed
  e8ee411

Commits on Aug 25, 2024

• fix(frontend): added create new evaluator section

  bekossy

  committed
  4a32a95

• Merge branch 'main' into AGE-587/-implement-evaluation-main-page

  bekossy

  committed
  c2813fb

Commits on Aug 26, 2024

• Merge branch 'feature/age-491-poc-1e-expose-running-evaluators-via-api-to-playground' into feature/age-532-poc-1e-add-llm-api-key-checks-in-llm-based-evaluators

  aybruhm

  authored
  0ce0022

• Update evaluators_service.py

  jp-agenta

  authored
  cc33a66

• fix(frontend): modified configure evaluator state to use query param, enable search feature and improved title style

  bekossy

  committed
  bfe4cdb

Commits on Aug 27, 2024

• fix(frontend): improved config evaluator modal

  bekossy

  committed
  8ba96f2

• fix(frontend): added 600px height to fixed overflow in config evaluator card views

  bekossy

  committed
  b4ca6fc

• fix(frontend): added fetch variants state

  bekossy

  committed
  ce455b1

• design(frontend): added select variant modal in evaluator config modal

  bekossy

  committed
  59d64f6

• fix(frontend): displayed evaluator table content

  bekossy

  committed
  a75d801

• design(frontend): ui improvements

  bekossy

  committed
  ebb5301

Commits on Aug 28, 2024

• fix(frontend): added helper func to get single model and ab testing evaluation result

  bekossy

  committed
  bf51d9b

• design(frontend): added human evaluation tabs

  bekossy

  committed
  c2580a0

• fix(frontend): fetched evaluation result list and passed result to each component

  bekossy

  committed
  8dd6ab4

Commits on Aug 29, 2024

• Merge pull request #1989 from Agenta-AI/feature/age-532-poc-1e-add-llm-api-key-checks-in-llm-based-evaluators

  Show description for 9212c7b
  aybruhm

  authored
  9212c7b

• fix(frontend): fixed config evaluator modal alignment and updated human eval table

  bekossy

  committed
  a338053

Commits on Aug 30, 2024

• Merge pull request #1956 from Agenta-AI/feature/age-491-poc-1e-expose-running-evaluators-via-api-to-playground

  Show description for f563103
  bekossy

  authored
  f563103

• Merge branch 'AGE-587/-implement-evaluation-main-page' of https://github.com/Agenta-AI/agenta into AGE-587/-implement-evaluation-main-page

  bekossy

  committed
  c55d1e4

• ui(frontend): implemented human evaluations

  ashrafchowdury

  committed
  35d5eb3

• fix(frontend): added evaluator mapping endpoints

  bekossy

  committed
  d6aadf5

Commits on Aug 31, 2024

• design(frontend): improved config evaluator ui

  bekossy

  committed
  41745ff

Commits on Sep 1, 2024

• fix(frontend): implemented select test case functionality

  bekossy

  committed
  244e8bd

• fix(frontend): added conditional to select test case

  bekossy

  committed
  54a0d49

Commits on Sep 2, 2024

• fix: resolved merge conflict

  ashrafchowdury

  committed
  704ef9b

• refactor: improved structure

  Show description for c8bc24e
  ashrafchowdury

  committed
  c8bc24e

• fix(frontend): implemented run variant functionality

  bekossy

  committed
  7d0acf4

• refactor(frontend): rename directory to improve clarity and context

  bekossy

  committed
  528d223

Commits on Sep 3, 2024

• refactor(frontend): removed unused codes

  ashrafchowdury

  committed
  fe0bc46

• fix(frontend): failing cypress tests

  ashrafchowdury

  committed
  0a02a48

• Merge branch 'main' into AGE-587/-implement-evaluation-main-page

  bekossy

  committed
  90afba5

• fix: invalid import in evaluation router and improved editor

  bekossy

  committed
  e91e37b

• Merge branch 'AGE-587/-implement-evaluation-main-page' into feat/implement-human-evaluations

  bekossy

  committed
  4732b7b

• test(frontend): tests for eval tabs

  ashrafchowdury

  committed
  959e773

• fix(frontend): prettier error

  ashrafchowdury

  committed
  538b852

• Merge pull request #2047 from Agenta-AI/feat/implement-human-evaluations

  Show description for 60d237e
  bekossy

  authored
  60d237e

• refactor(frontend): moved annotations to evaluations dir and cleanup

  bekossy

  committed
  f9c2c39

• Merge branch 'main' of https://github.com/Agenta-AI/agenta into enhance/automatic-tab-functionalitis

  ashrafchowdury

  committed
  e4c09b9

• ui(frontend): automatic eval funcational table

  ashrafchowdury

  committed
  0a7e551

• Merge branch 'AGE-587/-implement-evaluation-main-page' of https://github.com/Agenta-AI/agenta into enhance/automatic-tab-functionalitis

  ashrafchowdury

  committed
  71c36e2

• fix(frontend): search issue with numbers

  ashrafchowdury

  committed
  1fe2bab

• feat(frontend): added helper function to transfor trace tree to json

  bekossy

  committed
  4fc0fbf

• minor naming improvement

  bekossy

  committed
  0ca0bc1

Commits on Sep 4, 2024

• Merge branch 'AGE-587/-implement-evaluation-main-page' of https://github.com/Agenta-AI/agenta into enhance/automatic-tab-functionalitis

  ashrafchowdury

  committed
  32ea255

• ui(frontend): added table results column

  ashrafchowdury

  committed
  cacad57

• refactor(frontend): removed unsed code

  ashrafchowdury

  committed
  a1cbbd6

• refactor(frontend): improved component name for clarify

  bekossy

  committed
  f3c541b

• fix(frontend): transform trace tree and setup mapping

  bekossy

  committed
  dc2067e

Commits on Sep 5, 2024

• fix(frontend): failing cypress test due to layout change

  ashrafchowdury

  committed
  c7f1afd

• enhance(frontend): improved structure

  ashrafchowdury

  committed
  827680e

• fix(frontend): prettier format

  ashrafchowdury

  committed
  d8bca04

• fix(frontend): lint error

  ashrafchowdury

  committed
  baf3e68

• feat(frontend): implemented run evaluator logic and updated EvaluatorRunExecution Types

  bekossy

  committed
  2410b3f

• Merge branch 'main' into AGE-587/-implement-evaluation-main-page

  bekossy

  committed
  c620759

• Merge branch 'AGE-587/-implement-evaluation-main-page' into enhance/automatic-tab-functionalitis

  bekossy

  committed
  af8728e

• Merge branch 'AGE-587/-implement-evaluation-main-page' of https://github.com/Agenta-AI/agenta into enhance/automatic-tab-functionalitis

  ashrafchowdury

  committed
  cc8cf94

• fix(backend): Updated Evaluator model to requires_llm_api_keys field

  bekossy

  committed
  a1b048d

• refactor(frontend): improve handling of testcase mapping and evaluator values

  bekossy

  committed
  910999d

• Merge branch 'enhance/automatic-tab-functionalitis' of https://github.com/Agenta-AI/agenta into enhance/automatic-tab-functionalitis

  ashrafchowdury

  committed
  d04631c

• fix(backend): fixed rag evaluator inputs and bug in exact match evaluator

  bekossy

  committed
  aecbc5b

• refactor(frontend): clean up and optimize fetchEvalMapper logic

  bekossy

  committed
  7e0c469

• fix(frontend): bug fixes

  bekossy

  committed
  751245d

• fix(frontend): added EvaluationErrorPopover component and improve evaluation result output display

  bekossy

  committed
  c43e6c1

Commits on Sep 6, 2024

• Merge branch 'enhance/automatic-tab-functionalitis' of https://github.com/Agenta-AI/agenta into enhance/automatic-tab-functionalitis

  ashrafchowdury

  committed
  11bcc90

• fix(frontend): run variant with chat template

  bekossy

  committed
  21f193e

• enhance(frontend): edit columns and short columns

  ashrafchowdury

  committed
  9cf40f7

• Merge branch 'AGE-587/-implement-evaluation-main-page' of https://github.com/Agenta-AI/agenta into enhance/automatic-tab-functionalitis

  ashrafchowdury

  committed
  bbaf1e9

• fix(frontend): fixed status update issue

  ashrafchowdury

  committed
  2ab3da3

• fix(frontend): update json variant result, json field title and updated trace tree

  bekossy

  committed
  8c3cadf

Commits on Sep 7, 2024

• refactor(frontend): added hepler to transform trace settings to remove "trace." from payload and added helper text for json field title

  bekossy

  committed
  2985804

• fix(frontend): filter empty/falsy values from json data output

  bekossy

  committed
  10db7b7

• feat(frontend): setup view configuration and deletion

  bekossy

  committed
  72e8be4

Commits on Sep 8, 2024

• feat(frontend): setup open config and clone features

  bekossy

  committed
  84f7cba

• fix(frontend): removed old evaluators code

  bekossy

  committed
  55828ae

• minor fix

  bekossy

  committed
  51ebdfe

• Merge branch 'main' into AGE-587/-implement-evaluation-main-page

  bekossy

  committed
  fea95b7

• feat(frontend): generated color for evaluator configs and updated types

  bekossy

  committed
  0e88e11

• refactor(frontend): set evaluator display default to list and ui improvements

  bekossy

  committed
  db8793e

• Merge branch 'AGE-587/-implement-evaluation-main-page' of https://github.com/Agenta-AI/agenta into enhance/automatic-tab-functionalitis

  ashrafchowdury

  committed
  2d13427

• Merge branch 'AGE-587/-implement-evaluation-main-page' into enhance/automatic-tab-functionalitis

  bekossy

  committed
  0b8dbe7

• fix(frontend): removed bad code

  bekossy

  committed
  10098ab

Commits on Sep 9, 2024

• Merge branch 'enhance/automatic-tab-functionalitis' of https://github.com/Agenta-AI/agenta into enhance/automatic-tab-functionalitis

  ashrafchowdury

  committed
  74c32ea

• test(frontend): fixed evaluator tests

  ashrafchowdury

  committed
  b8480d3

• fix(backend): updated auto_custom_code_run default code

  bekossy

  committed
  27e3f84

• fix(frontend): added maxWidth to configuration form, disabled editor minimap, and removed custom header for table view

  bekossy

  committed
  4d1ae19

• Merge branch 'AGE-587/-implement-evaluation-main-page' into enhance/automatic-tab-functionalitis

  bekossy

  committed
  3d856ee

• fix(frontend): icons placement issue

  ashrafchowdury

  committed
  44a66ba

• fix(frontend): improved StatusRenderer logic to update status count

  bekossy

  committed
  5ebeb6c

• Merge pull request #2058 from Agenta-AI/enhance/automatic-tab-functionalitis

  Show description for 62b5745
  bekossy

  authored
  62b5745

• fix(frontend): evaluator category button border left style

  bekossy

  committed
  699f6ce

• fix(frontend): cleanup

  bekossy

  committed
  38c0cb7

• feat(frontend/backend): enabled evaluator filtering by category

  bekossy

  committed
  4fe5e1f

Commits on Sep 10, 2024

• Merge branch 'main' into AGE-587/-implement-evaluation-main-page

  bekossy

  committed
  275df9d

• fix(frontend): new cypress tests failures

  ashrafchowdury

  committed
  2517a07

• fix(frontend): select new evaluator table column issue

  ashrafchowdury

  committed
  cf37bb5

• fix(frontend): added tags and improved styles

  bekossy

  committed
  0b832ee

• fix(frontend): replaced version static in evaluator card with created at date

  bekossy

  committed
  5fe9aa6

• design(frontend): enabled hover effect and displayed arrow-right on hover

  bekossy

  committed
  68414aa

• fix(frontend): displayed updated_at in evaluator card

  bekossy

  committed
  05c18fb

Commits on Sep 11, 2024

• fix(frontend): improved naming, removed evaluator card view and improved style

  bekossy

  committed
  65f0a90

• fix(frontend): set font family

  bekossy

  committed
  3f6f0ae

• fix(frontend): fixed cypress test

  bekossy

  committed
  d804edc

• fix(frontend): modified font family

  bekossy

  committed
  c5efd0e

Commits on Sep 12, 2024

• fix(frontend): moved debug evaluator feature code to cloud

  bekossy

  committed
  65ac8a9

• fix(frontend): removed testcase tab

  bekossy

  committed
  7f737c9

Commits on Sep 13, 2024

• minor refactor (backend): explictly get testcase correct_answer key

  aybruhm

  committed
  c4a3e53

• minor refactor (tests): update the run inputs of rag_faithfulness evaluator fixture

  aybruhm

  committed
  381c628

• Merge branch 'AGE-587/-implement-evaluation-main-page' of https://github.com/Agenta-AI/agenta into AGE-587/-implement-evaluation-main-page

  bekossy

  committed
  f785729

Commits on Sep 14, 2024

• fix(frontend): removed checkbox and tags columns from evaluator table view

  bekossy

  committed
  973c256

• refactor(frontend): modified test toggle button icon

  bekossy

  committed
  6ebab98

• design(frontend): added transition and conditional to evaluator modal

  bekossy

  committed
  6c15e42

• fix(frontend): moved selected testset state to parent level

  bekossy

  committed
  c1f14df

• minor fix

  bekossy

  committed
  cea7ab9

• Merge branch 'main' into AGE-587/-implement-evaluation-main-page

  bekossy

  committed
  d1dc894

• bug fix(frontend)

  bekossy

  committed
  f8ebba1

Commits on Sep 15, 2024

• code cleanup(frontend)

  bekossy

  committed
  75ab1ce

Commits on Sep 16, 2024

• test(frontend): fixed failed evaluator test

  ashrafchowdury

  committed
  b186838

• fix(frontend): fixed edit column functionality to have all cols selected by default

  bekossy

  committed
  246e467

Commits on Sep 17, 2024

• fix(frontend): changed testcase state to an object

  bekossy

  committed
  a425cd5

• fix(frontend): edit columns and short dates

  ashrafchowdury

  committed
  2f3a5de

• test(frontend): fixes failed evaluation test

  ashrafchowdury

  committed
  68e8e9e

Commits on Sep 18, 2024

• fix(frontend): edit column checked by default

  bekossy

  committed
  9b466a0

Commits on Sep 19, 2024

• design(frontend): change evaluators view title to 16px and added divider in new evaluators view

  bekossy

  committed
  dc4c978

• fix(backend): removed correct_answer_key in settings_template for auto_exact_match in evaluators resources

  bekossy

  committed
  bac7a51

Commits on Sep 20, 2024

• cleanup(frontend)

  bekossy

  committed
  675169e

Commits on Sep 22, 2024

• fix(frontend): chat inputs not shown in ab testing and single model evaluation view

  bekossy

  committed
  e16c31e

Commits on Sep 23, 2024

• test(frontend): fixed failed evaluator test

  ashrafchowdury

  committed
  10797f4