[Bug]: [AutogenStudio] Agents no longer have access to previous conversations within a Session

[dragosMC91]

Describe the bug

This started happenning after I upgraded AutogenStudio from 0.0.44a0 -> 0.0.47.
I then downgraded again, but the behavior persists.
Tried removing the current python virtual envs, and installing autogenstudio in new venv, but the bug persists.

Steps to reproduce

1. install autogenstudio pip install autogenstudio==0.0.44a0 (this was the last version which I know worked)
2. start the server via autogenstudio ui --port 8084 --appdir /path/to/ui (I made sure to clean the ui artifacts)
3. setup a basic workflow with 1 proxyuser and 1 primary assistant agent
4. in the Playground, create a new session with the workflow from step 3.
5. input a prompt which outputs a list (e.g give me 10 uses for ginger)
6. ask a follow up question on any of the 10 points, without specifying exactly the text from the point (e.g pleae elaborate on point 6)

Model Used

I tried gpt 3.5 turbo, 4 turbo, mistral medium, claude 3 opus

Expected Behavior

When I ask the follow up question, a request is made to openai with all previous messages from the conversation.
So considering the previous example, when I ask pleae elaborate on point 6, the model knows exactly which point I am refering to.

Screenshots and logs

userproxy (to primary_assistant):

give me 10 uses for ginger

---

primary_assistant (to userproxy):

To provide you with 10 uses for ginger, I will combine my knowledge with some research. Let's start by listing some common uses for ginger:

1. Culinary Purposes: Ginger is widely used in cooking and baking to add flavor to dishes such as stir-fries, curries, soups, and desserts.
2. Digestive Aid: Ginger is known for its ability to help with digestion and alleviate digestive issues like nausea, bloating, and indigestion.
3. Anti-Inflammatory Properties: Ginger contains compounds that have anti-inflammatory effects, which can help reduce inflammation in the body.
4. Immune System Booster: Ginger is rich in antioxidants and has immune-boosting properties that can help strengthen the immune system.
5. Relief from Motion Sickness: Ginger is often used to alleviate symptoms of motion sickness and nausea.
6. Sore Throat Relief: Ginger tea with honey can help soothe a sore throat and provide relief from coughing.
7. Pain Relief: Ginger has been used traditionally to help alleviate pain, including menstrual cramps and muscle soreness.
8. Blood Sugar Regulation: Some studies suggest that ginger may help regulate blood sugar levels.
9. Skin Benefits: Ginger has antioxidant properties that can help improve skin health and reduce signs of aging.
10. Hair Care: Ginger can be used in hair masks or shampoos to promote hair growth and improve scalp health.

Now, let's summarize these uses in a list format using Python:

---

userproxy (to primary_assistant):

TERMINATE

---

Modified files: 0
INFO: 127.0.0.1:55332 - "POST /api/messages HTTP/1.1" 200 OK
userproxy (to primary_assistant):

give me 10 uses for ginger

---

primary_assistant (to userproxy):

TERMINATE

---

userproxy (to primary_assistant):

pleae elaborate on point 6

---

primary_assistant (to userproxy):

TERMINATE

---

Modified files: 0
INFO: 127.0.0.1:55336 - "POST /api/messages HTTP/1.1" 200 OK
userproxy (to primary_assistant):

give me 10 uses for ginger

---

primary_assistant (to userproxy):

TERMINATE

---

userproxy (to primary_assistant):

pleae elaborate on point 6

---

primary_assistant (to userproxy):

TERMINATE

---

userproxy (to primary_assistant):

pleae elaborate on point 6

---

primary_assistant (to userproxy):

To elaborate on point 6, ginger can be used in various desserts such as gingerbread cookies, ginger snaps, ginger cake, ginger ice cream, ginger pudding, ginger scones, ginger muffins, ginger-flavored frosting, and ginger-infused custards. Ginger adds a warm and spicy flavor to these sweet treats, balancing out the sweetness and providing a unique taste profile. It can be used in both baked and frozen desserts to enhance the overall flavor and aroma. Ginger can be grated, minced, or powdered depending on the recipe requirements.

---

userproxy (to primary_assistant):

TERMINATE

---

Additional Information

AutoGenStudio Version: 0.0.44a0
Operating System: MacOS with M2 pro
Python Version: 3.10.12 / 3.11.8

[victordibia]

Hi @dragosMC91

Thanks for reporting this ...
The short answer is to consider:

• update autogenstudio pip install -U autogenstudio ..v0.0.48
• change the summary_method field in your agent workflow to llm instead of last.

Background

• The ability for agents to reference previous conversations is impacted by the type of information stored in the session history. In AGS, we store the final response from a workflow of agents as history ( not the agent conversations itself).

• The final response is determined by the summary_method field for a workflow configuration. Supported options include

  • last: The default is last which saves the last message in the agent conversation. The drawback of last is that if the last message does not contain the actual answer, then that information is not committed to history.
  • llm (recommended): This uses an LLM to summarize the agent conversation before it is added to history (recommended).
  • none: No final message is created.

In your case, the summary message is likely set to last hence the inability to reference previous messages.

I am happy to get some feedback here too.

summarymethod.mp4

[dragosMC91]

Hello @victordibia, thank you for the reply and apologies for the delayed response, I just got to testing this and indeed it works as expected.

The details you provided about summary_method are really useful, I initially thought the role of the attribute is just cosmetic. Maybe this should be mentioned in the official docs (https://autogen-studio.com/) I am sure other people had this problem as well.

Also, I now noticed with the latest version of autogenstudio (0.0.54), with summary_method == llm, that you get a neatly formatted summary of the complete answer. So, for this, an additional API call is made to the llm service, maybe this should also be explicitly mentioned in the docs because it will incur extra costs (I think). I am not sure which option is more expensive:

1. (from previous versions) commit the full response to the session history and thus submit more input tokens on next messages
2. (current versions) make 1 additional request to summarize the response and only commit that to the session history
   Not sure if the above is a valid concern and this is actually how the setup works but I thought 2 might be more expensive since output tokens are 3x (openai models) to 5x (anthropic models) more expensive than input tokens

Feedback on AGS as someone who has been fiddling with the ui tool since it's early days (autogen assistant 0.0.1x ui)

1. the ability to export user created resources (models, agents, workflows etc) is invaluable - it happened a few times after an AGS version update that I had to purge the DB because nothing was working anymore, now I can just export my resources and import them if this happens again. Suggestion: maybe have an export all resources feature, which downloads a small zip with all relevant folders and their corresponding resources
2. usability in general was greatly improved - with current versions I can’t think of any real issues i've encountered, I would like more control over how responses are displayed in the UI, maybe have this decoupled from 'summary_method' feature
3. I would like the ability to be able to customize the proxy user more, currently it executes code locally which is scary, I would like to tell it to use docker. Also when using docker, it would be nice to be able to execute all code within a target container (not specify the image and have the framework create the container) but I guess this part is not related to the AGS ui project, its related to the root project

Overall I think you guys are doing an incredible job and I love this project and will continue to use it

[victordibia]

Hi @dragosMC91 ,

Thanks for all of this feedback and suggestion!!
I am still thinking about the right was to store history - in addition to cost, full history can be noisy (imagine an agent conversation with 30 turns and many errors becoming part of the history which can degrade future performance). I am sure we will all learn and converge on some standard, or just make that an option that the user/dev can modify.

Thanks for the feedback on the value of being able to export entities. I like the idea of export all (possibly recreate the db sort of). There is currently some work to rewrite/improve the db capabilities in AGS #1694 . Once that is concluded, we should enable such a feature. We are also trying to improve the overall UI/UX and being able to specify docker container should be integrated here.

Finally, https://autogen-studio.com/ IS NOT OFFICIAL DOCUMENTATION.

@gagb @jackgerrits @ekzhu