Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add role to reflection with llm #2527

Merged
merged 16 commits into from
May 14, 2024

Conversation

MarianoMolina
Copy link
Contributor

@MarianoMolina MarianoMolina commented Apr 26, 2024

Use 'role' in summary_args to use as the role string of the prompt message used to summarize.

Why are these changes needed?

When creating a GroupChat summary, the prompt 'role' is hardcoded as "system". Idea is to enable the initiate_chat() flow to define this by declaring it in system_args. Issue is that there is already a 'role' argument that is passed:
(From the docstrings)

"role": role of the message, can be "assistant", "user", "function".
This field is only needed to distinguish between "function" or "assistant"/"user".

However, I don't see anywhere in the flow that this value is retrieved or used for anything. I'm not entirely sure what the intention was (potentially to distinguish function calls from conversations, but the flow makes that decision irrespective of the 'role' value), so we can either (1) repurpuse this value to use as the role string to pass in the message dict or (2) add a different argument (called 'summary_role' or something) to avoid potential issues.

Given that, to the best of my knowledge, the old 'role' value wasn't being used, I think the best approach is 1.

With these changes, the argument can still be used to distinguish between function and assistant, but now has a 'default' value of 'system', and is used as the string value for the 'role' attribute in the summary prompt message.

  • No doc changes needed

Related issue number

Closes #2492
#2644

Checks

… be able to pass the role for the summarizing prompt
… be able to pass the role for the summarizing prompt, minor docstring adjustments
@MarianoMolina MarianoMolina marked this pull request as ready for review April 26, 2024 16:54
@ekzhu ekzhu requested a review from qingyun-wu April 27, 2024 00:40
Copy link
Collaborator

@ekzhu ekzhu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a test case for this?

@ekzhu ekzhu requested a review from sonichi April 27, 2024 00:43
@MarianoMolina
Copy link
Contributor Author

Added a test on test_groupchat.

@sonichi sonichi enabled auto-merge April 29, 2024 15:44
Copy link

gitguardian bot commented Apr 29, 2024

⚠️ GitGuardian has uncovered 2 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
10404662 Triggered Generic CLI Secret eff19ac .github/workflows/dotnet-release.yml View secret
10404662 Triggered Generic CLI Secret e973ac3 .github/workflows/dotnet-release.yml View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secrets safely. Learn here the best practices.
  3. Revoke and rotate these secrets.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@sonichi
Copy link
Contributor

sonichi commented Apr 29, 2024

https://github.com/microsoft/autogen/actions/runs/8881467046/job/24383882709?pr=2527
The code formatting error can be fixed by pre-commit.

https://github.com/microsoft/autogen/actions/runs/8881467028/job/24383887017?pr=2527
The build error is due to the use of "OAI_CONFIG_LIST" in the added test. Could you use a mocked config list for this test which doesn't require real openai keys?

@ekzhu
Copy link
Collaborator

ekzhu commented Apr 30, 2024

The changes are fine. Please follow @sonichi 's comments. Thanks.

auto-merge was automatically disabled April 30, 2024 18:27

Head branch was pushed to by a user without write access

@MarianoMolina
Copy link
Contributor Author

Fixed both comments.

Copy link
Contributor

@sonichi sonichi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please fix the conflict and one minor comment? Thanks.

autogen/agentchat/conversable_agent.py Outdated Show resolved Hide resolved
@sonichi
Copy link
Contributor

sonichi commented May 2, 2024

The code formatting error can be fixed with pre-commit.

@MarianoMolina
Copy link
Contributor Author

I installed pre-commit, but since I am not using WSL, I had to uninstall to be able to commit.

No idea what this file was about
@ekzhu ekzhu enabled auto-merge May 13, 2024 10:34
@codecov-commenter
Copy link

codecov-commenter commented May 13, 2024

Codecov Report

Attention: Patch coverage is 0% with 6 lines in your changes are missing coverage. Please review.

Project coverage is 22.84%. Comparing base (372ac1e) to head (b23b4ba).
Report is 17 commits behind head on main.

Files Patch % Lines
autogen/agentchat/conversable_agent.py 0.00% 6 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##             main    #2527       +/-   ##
===========================================
- Coverage   33.11%   22.84%   -10.27%     
===========================================
  Files          86       86               
  Lines        9108     9243      +135     
  Branches     1938     2131      +193     
===========================================
- Hits         3016     2112      -904     
- Misses       5837     6888     +1051     
+ Partials      255      243       -12     
Flag Coverage Δ
unittests 22.78% <0.00%> (-10.33%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ekzhu
Copy link
Collaborator

ekzhu commented May 13, 2024

Please fix the CI error on formatting.

You can install pre-commit so this will be checked before commit. https://microsoft.github.io/autogen/docs/contributor-guide/pre-commit

auto-merge was automatically disabled May 13, 2024 22:13

Head branch was pushed to by a user without write access

@MarianoMolina
Copy link
Contributor Author

Not sure how or why, but seems the issue is with the test_groupchat file, which got an update in main that commented all functions and created this issue. This merge from main seems to be the cause this issue was present in this branch: b23b4ba

@sonichi sonichi enabled auto-merge May 14, 2024 06:42
@sonichi sonichi added this pull request to the merge queue May 14, 2024
Merged via the queue into microsoft:main with commit 4b747d7 May 14, 2024
74 of 88 checks passed
@MarianoMolina MarianoMolina deleted the add-role-to-reflection-with-llm branch May 14, 2024 17:30
jayralencar pushed a commit to jayralencar/autogen that referenced this pull request May 28, 2024
* Added 'role' as a summary_args and to the reflection_with_llm flow to be able to pass the role for the summarizing prompt

* Added 'role' as a summary_args and to the reflection_with_llm flow to be able to pass the role for the summarizing prompt, minor docstring adjustments

* Added test for summary prompt role assignment

* Fixed docstrings and mocked llm-config in the test

* Update autogen/agentchat/conversable_agent.py

Co-authored-by: Chi Wang <wang.chi@microsoft.com>

* ran pre-commit

* ran pre-commit2

* fixed old arg name

* Delete dasdaasd

No idea what this file was about

* Fixed incorrect merge update on test_groupchat

---------

Co-authored-by: Chi Wang <wang.chi@microsoft.com>
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>
victordibia pushed a commit that referenced this pull request Jul 30, 2024
* Added 'role' as a summary_args and to the reflection_with_llm flow to be able to pass the role for the summarizing prompt

* Added 'role' as a summary_args and to the reflection_with_llm flow to be able to pass the role for the summarizing prompt, minor docstring adjustments

* Added test for summary prompt role assignment

* Fixed docstrings and mocked llm-config in the test

* Update autogen/agentchat/conversable_agent.py

Co-authored-by: Chi Wang <wang.chi@microsoft.com>

* ran pre-commit

* ran pre-commit2

* fixed old arg name

* Delete dasdaasd

No idea what this file was about

* Fixed incorrect merge update on test_groupchat

---------

Co-authored-by: Chi Wang <wang.chi@microsoft.com>
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug]: reflection_with_llm issue with local server
4 participants