Enhance Agent Logging: Comprehensive Step, Tool, and Memory Tracking in JSON format #615
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- added asdict to turn things into json compatible output
- adding updated log methods to run
- make the run update for tools
- edit "run" return type with condition on how to return if output type is JSON .
- add newer methods to the log_step_metadata for logging
- update tool usage fn to add and account for tool usage metadata.
- minor refactor
- update correct libraries & dependencies .
- create class to execute modular unittests - def setup for modular setup - objective to keep setup minimal , so that tests aren't bloated and fast to run - Since most param have a set default , init of necessary condition is a valid and supportive op wrt test speed .
- Implemented test_log_step_metadata_basic to verify the correct logging of step metadata including step_id, timestamp, tokens, and memory usage. - Confirmed that the token counts for total are accurately logged.
- Implemented test_log_step_metadata_no_long_term_memory to ensure that when long-term memory is None, the memory_usage for long_term is an empty dictionary in the log result.
- Implemented test_log_step_metadata_timestamp to verify that a timestamp is included in the log result when logging step metadata.
- Implemented test_token_counting_integration to verify the correct total token count when using a mocked tokenizer, ensuring that prompt and response token counts are accurately aggregated.
- Implemented test_agent_output_updating to verify that logging step metadata correctly updates the total token count and ensures that the agent's output steps are properly tracked confirming only one step is recorded.
- apart of modular and small unittests , its time to put to test for larger and more wholesome integration test - an end to end test , to check if this change in code causes any breaks in flow and normal functioning .
- add main to make this run
- Update the docs to return describe changes - Add lines specific to how to toggle between output types .
github-actions
bot
added
documentation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
The current Agent class lacks detailed logging of each step, tool usage, and memory consumption. This makes it difficult for users to debug and analyze the agent's behavior, especially when dealing with complex tasks.
Solution
We've implemented comprehensive logging for the Agent class, capturing detailed information about each step, tool usage, and memory consumption. The logging system now supports both string and JSON output formats, allowing users to choose the most suitable format for their needs.
Key changes include:
log_step_metadatamethodImpact
These changes significantly improve the transparency and debuggability of the Agent class. Users can now:
Final Results
With these changes, users can now:
The enhanced logging system provides valuable insights into the agent's decision-making process, making it easier for developers to optimize and troubleshoot their AI applications.
Videos And Screenshots
You can find the video of the code explaination , walkthrough , demos , unittests etc here : https://drive.google.com/file/d/1IsL288KWy5IyZ6bKJiEKpWyDAbNomFvH/view?usp=sharing
A small glimpse of all testcases successfully passing including the integration test is as follows :
The first 5 here are indications of successful unittests and the 6th is the successful integration test .
Additional Notes
📚 Documentation preview 📚: https://swarms--615.org.readthedocs.build/en/615/