Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite AutoGen Studio Database Layer to Use SQLModel ORM #2425

Merged
merged 23 commits into from
May 11, 2024

Conversation

victordibia
Copy link
Collaborator

@victordibia victordibia commented Apr 17, 2024

Why are these changes needed?

This PR rewrites the database module for autogenstudio using use SQLModel (Pydantic + SQLAlchemy). This provides

  • Entity linking (skills, models, agents and workflows are linked via association tables). This addresses multiple existing issues where changes to skills, models or agents do not propagate to workflows.
  • Supports for multiple database backend dialects supported in SQLAlchemy (SQLite, PostgreSQL, MySQL, Oracle, Microsoft SQL Server). The backend database can be specified a --database-uri argument when running the application. For example, autogenstudio ui --database-uri sqlite:///database.sqlite for SQLite and autogenstudio ui --database-uri postgresql://user:password@localhost/dbname for PostgreSQL.
  • Adds support for LocalCommandLineCodeExecutor and DockerCommandLineCodeExecutor introduced by @jackgerrits and @ekzhu

Note: This PR introduces significant changes to the database schema, incompatible with v< 0.0.56.

Testing

Install autogenstudio from autogenstudio-orm branch. Run autogenstudio ui

Related issue number

#1694 #2128

Checks

@victordibia victordibia requested a review from ekzhu April 17, 2024 23:45
async def create_skill(skill: Skill):
"""Create a new skill"""
filters = {"user_id": skill.user_id}
return create_entity(skill, Skill, filters=filters)

Check warning

Code scanning / CodeQL

Information exposure through an exception Medium

Stack trace information
flows to this location and may be exposed to an external user.
@api.post("/models")
async def create_model(model: Model):
"""Create a new model"""
return create_entity(model, Model)

Check warning

Code scanning / CodeQL

Information exposure through an exception Medium

Stack trace information
flows to this location and may be exposed to an external user.
@api.post("/workflows")
async def create_workflow(workflow: Workflow):
"""Create a new workflow"""
return create_entity(workflow, Workflow)

Check warning

Code scanning / CodeQL

Information exposure through an exception Medium

Stack trace information
flows to this location and may be exposed to an external user.
@api.post("/sessions")
async def create_session(session: Session):
"""Create a new session"""
return create_entity(session, Session)

Check warning

Code scanning / CodeQL

Information exposure through an exception Medium

Stack trace information
flows to this location and may be exposed to an external user.
@victordibia victordibia added the studio Related to AutoGen Studio. label Apr 17, 2024
@Knucklessg1
Copy link
Collaborator

Wow, this is fantastic! Happy to see support for more DBs!

@ekzhu
Copy link
Collaborator

ekzhu commented Apr 18, 2024

@ma-armenta I have invited you to join as a collaborator to review this issue.

@ekzhu
Copy link
Collaborator

ekzhu commented Apr 20, 2024

When I start AutoGen studio built from this branch, it shows multiple "Connection error 404 Not Found" in a banner error message.

@ShaneYuTH
Copy link

@victordibia The field "admin_name" is declared twice with the exact same specifications and default value from AgentConfig under datamodel.py. version: autogenstudio 0.0.56rc1

@victordibia
Copy link
Collaborator Author

@victordibia The field "admin_name" is declared twice with the exact same specifications and default value from AgentConfig under datamodel.py. version: autogenstudio 0.0.56rc1

Thanks. Fixed.

@ShaneYuTH
Copy link

@victordibia Unable to delete sessions, got
"The Session is linked to another entity and cannot be deleted".
version: 0.0.56rc3

@ShaneYuTH
Copy link

@victordibia Unable to delete sessions, got "The Session is linked to another entity and cannot be deleted". version: 0.0.56rc3

FYI: It violates foreign key constraint "message_session_id_fkey" on table "message"

@ma-armenta
Copy link
Collaborator

@victordibia I am building from source to test things out. Using postgresql and got ModuleNotFoundError: No module named 'psycopg2'. Lib is installed, however I have been using psycopg2-binary instead to avoid any compiler or external library requirements, if any, just some thoughts.

@victordibia victordibia marked this pull request as ready for review May 4, 2024 23:35
Copy link

gitguardian bot commented May 4, 2024

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
10404662 Triggered Generic CLI Secret 2448baa .github/workflows/dotnet-release.yml View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@victordibia victordibia enabled auto-merge May 9, 2024 19:20
@victordibia victordibia added this pull request to the merge queue May 11, 2024
Merged via the queue into main with commit d50f654 May 11, 2024
20 of 21 checks passed
@victordibia victordibia deleted the autogenstudio-orm branch May 11, 2024 14:35
jayralencar pushed a commit to jayralencar/autogen that referenced this pull request May 28, 2024
…2425)

* update orm branch + accesibility tweaks

* general file location refactor

* add support for LocalCommandLineCodeExecutor and DockerCommandLineCodeExecutor

* update code execution config laoding

* version update

* bump version rc1

* add model type selection (openai , gemini, azure)

* add ability to test workflow

* psycopg3 support

* add close logic to build tab pop ups, enable testing of workflows in build view

* updates to dbmanager, version bump

* add max_tokens default value

* ensure sessions are used correctly in dbmanager

* initial support for migrations

* update sessions/workflow api routing for clarity.

* general refactor, + add support for initial sample workflows

* orm branch updates

* Removed incorrect Git LFS files

* update git lfs tracking

---------

Co-authored-by: Audel Rouhi <knucklessg1@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
studio Related to AutoGen Studio.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants