Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature] Projects - Checkpoint 2 #2065

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
f7e8225
refactor (backend): update scope of testsets, applications, evaluator…
aybruhm Sep 8, 2024
8cfc131
chore (backend): resolve sqlalchemy.exc.ArgumentError after registeri…
aybruhm Sep 8, 2024
06406f1
refactor (backend): scope project_id to entities aside from testsets,…
aybruhm Sep 12, 2024
3ed63a3
Merge branch 'main' into oss-feature/age-667-project-2-update-scope-i…
aybruhm Sep 12, 2024
af0497b
minor refactor (migration): ensure migration points to the appropriat…
aybruhm Sep 12, 2024
96aea7a
refactor (backend): remove project_id from db entities
aybruhm Sep 12, 2024
31cdcdd
refactor (backend): make user_id column optional in DB entities and c…
aybruhm Sep 12, 2024
e5b8e1f
chore (backend): added helper function to check if unique constraint …
aybruhm Sep 12, 2024
8d41eee
refactor (backend): scope project_id to db models entities and create…
aybruhm Sep 12, 2024
1bae734
refactor (backend): include default project_id in authentication midd…
aybruhm Sep 12, 2024
43fc826
feat (migrations): add custom data migration to manage project_id in …
aybruhm Sep 13, 2024
0d5f922
minor refactor (backend): fix condition when checking for multiple de…
aybruhm Sep 13, 2024
7e5993f
chore (style): format projects with black@23.12.0
aybruhm Sep 13, 2024
e6eecfc
refactor (backend): scope 'project_id' in API router endpoints
aybruhm Sep 13, 2024
0beaf89
minor refactor (backend): revert app_id column removal from bases table
aybruhm Sep 13, 2024
667c40d
feat (backend): implement utility function get project_id from reques…
aybruhm Sep 13, 2024
9068ae6
refactor (backend): make use of utility function to get default proje…
aybruhm Sep 13, 2024
b275e15
refactor (backend):
aybruhm Sep 16, 2024
7eaa879
Merge branch 'feature/age-662-project-1-add-update-tables-columns' in…
aybruhm Sep 16, 2024
0294d35
Merge branch 'feature/age-662-project-1-add-update-tables-columns' in…
aybruhm Sep 16, 2024
f06e2a8
feat (migrations): created custom migration logic to update evaluator…
aybruhm Sep 16, 2024
37b9729
refactor (migration): reorder migration script application for consis…
aybruhm Sep 16, 2024
94e356d
feat (backend): added AppDB and EvaluatorConfigDB to deprecated model…
aybruhm Sep 16, 2024
ec6041e
refactor (migrations): use deprecated models to ensure correct table …
aybruhm Sep 16, 2024
f6d998b
refactor (backend): replace get_project_id functionality with retriev…
aybruhm Sep 16, 2024
97b0b91
refactor (backend): include application name when creating evaluators
aybruhm Sep 16, 2024
d2fb567
refactor (backend): improve authentication middleware to:
aybruhm Sep 16, 2024
4c3ba31
refactor (backend): clean up route endpoints to make use of project_i…
aybruhm Sep 16, 2024
17b4888
refactor (backend): remove body check for project_id in route request
aybruhm Sep 17, 2024
7e2c542
minor refactor (migrations): query models to fetch records that have …
aybruhm Sep 19, 2024
d28a70b
refactor (backend): remove object and object_type when checking for a…
aybruhm Sep 19, 2024
5e93801
refactor (backend): cleanup use of organization and workspace
aybruhm Sep 19, 2024
0076528
chore (style): format code
aybruhm Sep 19, 2024
62f10cc
Merge branch 'feature/age-662-project-1-add-update-tables-columns' in…
aybruhm Sep 24, 2024
8c60809
refactor (backend): remove deprecated organization and workspace IDs …
aybruhm Sep 24, 2024
a0540eb
fix (bug): resolve SyntaxError: keyword argument repeated: project_id
aybruhm Sep 24, 2024
a1e94bc
refactor (backend): remove organization and workspace attributes to r…
aybruhm Sep 24, 2024
b55ac52
refactor (backend): explictly fetch user when creating api_key for va…
aybruhm Sep 24, 2024
63c9856
refactor (backend): fetch apps by their workspace/organization projec…
aybruhm Sep 24, 2024
2704959
refactor (backend): only make use of workspace_id from request query …
aybruhm Sep 24, 2024
920ec3c
refactor (backend): make use of project_id from app_variant when sett…
aybruhm Sep 24, 2024
1027d18
refactor (backend): update use of project_id for rbac
aybruhm Sep 24, 2024
d855d4c
refactor (backend): resolve failing evaluation run
aybruhm Sep 25, 2024
21435d7
fix (bug): resolve TypeError: list_variants_for_base() takes 1 positi…
aybruhm Sep 25, 2024
6a3bdae
fix (bug): resolve TypeError: get_deployment_by_id() takes 1 position…
aybruhm Sep 25, 2024
ad738db
refactor (backend): apply app name uniqueness only within project sco…
aybruhm Sep 25, 2024
83ae0cc
refactor (tests0: resolve failing backend tests due to project struct…
aybruhm Sep 25, 2024
f1856b4
fix (bug): resolve TypeError when creating variant from base
aybruhm Sep 25, 2024
04595f6
refactor (web): remove User column from AbTestingEvalOverview
aybruhm Sep 26, 2024
f70f385
refactor (backend): added assertion to ensure that app_variant is not…
aybruhm Sep 26, 2024
b786335
chore (backend): resolve the warning:
aybruhm Sep 26, 2024
a253eb5
chore (web): remove redundant User column from HumanEvaluationResult …
aybruhm Sep 26, 2024
4a77d08
chore (backend): make use of app project_id to create app testsets an…
aybruhm Sep 26, 2024
3266bb7
refactor (backend): resolve 'NoneType' object has no attribute 'proje…
aybruhm Sep 26, 2024
2f0a8c3
chore (backend): added log when deleting evaluations
aybruhm Sep 26, 2024
648fe6d
refactor (backend): added evaluation object_type when updating app la…
aybruhm Sep 26, 2024
62dc92b
chore (backend): added assert to ensure app_id in db object cannot be…
aybruhm Sep 26, 2024
31f66fc
chore (backend): improve error message in update_last_modified_by fun…
aybruhm Sep 26, 2024
5e21615
Merge branch 'refactor/project-structure' into oss-feature/age-667-pr…
aybruhm Sep 27, 2024
9270508
fix (tests): resolve sqlalchemy.exc.InvalidRequestError: Entity names…
aybruhm Sep 27, 2024
e4efa52
refactor (migrations): modify update_evaluators_with_app_name logic to:
aybruhm Sep 27, 2024
805d430
test(frontend)
bekossy Sep 28, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
import os
import uuid
import traceback
from typing import Optional


import click
from sqlalchemy.future import select
from sqlalchemy import create_engine, delete
from sqlalchemy.orm import sessionmaker, Session

from agenta_backend.models.deprecated_models import (
DeprecatedEvaluatorConfigDB,
DeprecatedAppDB,
)


BATCH_SIZE = 1000


def get_app_db(session: Session, app_id: str) -> Optional[DeprecatedAppDB]:
query = session.execute(select(DeprecatedAppDB).filter_by(id=uuid.UUID(app_id)))
return query.scalars().first()


def update_evaluators_with_app_name():
engine = create_engine(os.getenv("POSTGRES_URI"))
sync_session = sessionmaker(engine, expire_on_commit=False)

with sync_session() as session:
try:
offset = 0
while True:
records = (
session.execute(
select(DeprecatedEvaluatorConfigDB)
.filter(DeprecatedEvaluatorConfigDB.app_id.isnot(None))
.offset(offset)
.limit(BATCH_SIZE)
)
.scalars()
.all()
)
if not records:
break

# Update records with app_name as prefix
for record in records:
evaluator_config_app = get_app_db(
session=session, app_id=str(record.app_id)
)
if record.app_id is not None and evaluator_config_app is not None:
record.name = f"{record.name} ({evaluator_config_app.app_name})"

session.commit()
offset += BATCH_SIZE

# Delete deprecated evaluator configs with app_id as None
session.execute(
delete(DeprecatedEvaluatorConfigDB).where(
DeprecatedEvaluatorConfigDB.app_id.is_(None)
)
)
session.commit()
except Exception as e:
session.rollback()
click.echo(
click.style(
f"ERROR updating evaluator config names: {traceback.format_exc()}",
fg="red",
)
)
raise e
Original file line number Diff line number Diff line change
@@ -1,41 +1,86 @@
import os
import traceback
from typing import Sequence


import click
from sqlalchemy.future import select
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker

from agenta_backend.models.db_models import ProjectDB
from sqlalchemy.orm import sessionmaker, Session

from agenta_backend.models.db_models import (
ProjectDB,
AppDB,
AppVariantDB,
AppVariantRevisionsDB,
VariantBaseDB,
DeploymentDB,
ImageDB,
AppEnvironmentDB,
AppEnvironmentRevisionDB,
EvaluationScenarioDB,
EvaluationDB,
EvaluatorConfigDB,
HumanEvaluationDB,
HumanEvaluationScenarioDB,
TestSetDB,
)


BATCH_SIZE = 1000
MODELS = [
AppDB,
AppVariantDB,
AppVariantRevisionsDB,
VariantBaseDB,
DeploymentDB,
ImageDB,
AppEnvironmentDB,
AppEnvironmentRevisionDB,
EvaluationScenarioDB,
EvaluationDB,
EvaluatorConfigDB,
HumanEvaluationDB,
HumanEvaluationScenarioDB,
TestSetDB,
]


def get_default_projects(session):
query = session.execute(select(ProjectDB).filter_by(is_default=True))
return query.scalars().all()


def check_for_multiple_default_projects(session: Session) -> Sequence[ProjectDB]:
default_projects = get_default_projects(session)
if len(default_projects) > 1:
raise ValueError(
"Multiple default projects found. Please ensure only one exists."
)
return default_projects


def create_default_project():
PROJECT_NAME = "Default Project"
engine = create_engine(os.getenv("POSTGRES_URI"))
sync_session = sessionmaker(engine, expire_on_commit=False)

with sync_session() as session:
try:
default_projects = get_default_projects(session)
if len(default_projects) > 1:
raise ValueError(
"Multiple default projects found. Please ensure only one exists."
)

default_projects = check_for_multiple_default_projects(session)
if len(default_projects) == 0:
new_project = ProjectDB(project_name=PROJECT_NAME, is_default=True)
session.add(new_project)
session.commit()

except Exception as e:
session.rollback()
click.echo(click.style(f"ERROR: {traceback.format_exc()}", fg="red"))
click.echo(
click.style(
f"ERROR creating default project: {traceback.format_exc()}",
fg="red",
)
)
raise e


Expand All @@ -45,18 +90,13 @@ def remove_default_project():

with sync_session() as session:
try:
default_projects = get_default_projects(session)
default_projects = check_for_multiple_default_projects(session)
if len(default_projects) == 0:
click.echo(
click.style("No default project found to remove.", fg="yellow")
)
return

if len(default_projects) > 1:
raise ValueError(
"Multiple default projects found. Please ensure only one exists."
)

session.delete(default_projects[0])
session.commit()
click.echo(click.style("Default project removed successfully.", fg="green"))
Expand All @@ -65,3 +105,84 @@ def remove_default_project():
session.rollback()
click.echo(click.style(f"ERROR: {traceback.format_exc()}", fg="red"))
raise e


def add_project_id_to_db_entities():
engine = create_engine(os.getenv("POSTGRES_URI"))
sync_session = sessionmaker(engine, expire_on_commit=False)

with sync_session() as session:
try:
default_project = check_for_multiple_default_projects(session)[0]
for model in MODELS:
offset = 0
while True:
records = (
session.execute(
select(model)
.where(model.project_id == None)
.offset(offset)
.limit(BATCH_SIZE)
)
.scalars()
.all()
)
if not records:
break

# Update records with default project_id
for record in records:
record.project_id = default_project.id

session.commit()
offset += BATCH_SIZE

except Exception as e:
session.rollback()
click.echo(
click.style(
f"ERROR adding project_id to db entities: {traceback.format_exc()}",
fg="red",
)
)
raise e


def remove_project_id_from_db_entities():
engine = create_engine(os.getenv("POSTGRES_URI"))
sync_session = sessionmaker(engine, expire_on_commit=False)

with sync_session() as session:
try:
for model in MODELS:
offset = 0
while True:
records = (
session.execute(
select(model)
.where(model.project_id != None)
.offset(offset)
.limit(BATCH_SIZE)
)
.scalars()
.all()
)
if not records:
break

# Update records project_id column with None
for record in records:
record.project_id = None

session.commit()
offset += BATCH_SIZE

except Exception as e:
session.rollback()
click.echo(
click.style(
f"ERROR removing project_id to db entities: {traceback.format_exc()}",
fg="red",
)
)
raise e
33 changes: 31 additions & 2 deletions agenta-backend/agenta_backend/migrations/postgres/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,14 @@

import click
import asyncpg

from sqlalchemy import inspect, text, Engine
from sqlalchemy.exc import ProgrammingError
from sqlalchemy.ext.asyncio import create_async_engine, AsyncEngine

from alembic import command
from alembic.config import Config
from sqlalchemy import inspect, text
from alembic.script import ScriptDirectory
from sqlalchemy.ext.asyncio import create_async_engine, AsyncEngine

from agenta_backend.utils.common import isCloudEE, isCloudDev

Expand Down Expand Up @@ -173,3 +174,31 @@ async def check_if_templates_table_exist():
await engine.dispose()

return True


def unique_constraint_exists(
engine: Engine, table_name: str, constraint_name: str
) -> bool:
"""
The function checks if a unique constraint with a specific name exists on a table in a PostgreSQL
database.

Args:
- engine (Engine): instance of a database engine that represents a connection to a database.
- table_name (str): name of the table to check the existence of the unique constraint.
- constraint_name (str): name of the unique constraint to check for existence.

Returns:
- returns a boolean value indicating whether a unique constraint with the specified `constraint_name` exists in the table.
"""

with engine.connect() as conn:
result = conn.execute(
text(
f"""
SELECT conname FROM pg_constraint
WHERE conname = '{constraint_name}' AND conrelid = '{table_name}'::regclass;
"""
)
)
return result.fetchone() is not None
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
"""Update evaluators names with app name as prefix

Revision ID: 22d29365f5fc
Revises: 6cfe239894fb
Create Date: 2024-09-16 11:38:33.886908

"""

from typing import Sequence, Union

from agenta_backend.migrations.postgres.data_migrations.applications import (
update_evaluators_with_app_name,
)


# revision identifiers, used by Alembic.
revision: str = "22d29365f5fc"
down_revision: Union[str, None] = "6cfe239894fb"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None


def upgrade() -> None:
# ### custom command ###
update_evaluators_with_app_name()
# ### end custom command ###


def downgrade() -> None:
# ### custom command ###
pass
# ### end custom command ###
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
"""add default project to scoped model entities

Revision ID: 55bdd2e9a465
Revises: c00a326c625a
Create Date: 2024-09-12 21:56:38.701088

"""

from typing import Sequence, Union

from alembic import op
import sqlalchemy as sa

from agenta_backend.migrations.postgres.data_migrations.projects import (
add_project_id_to_db_entities,
remove_project_id_from_db_entities,
)


# revision identifiers, used by Alembic.
revision: str = "55bdd2e9a465"
down_revision: Union[str, None] = "c00a326c625a"
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None


def upgrade() -> None:
# ### custom command ###
add_project_id_to_db_entities()
# ### end custom command ###


def downgrade() -> None:
# ### custom command ###
remove_project_id_from_db_entities()
# ### end custom command ###
Loading
Loading