Refactor transform_messages

[dkirsche]

Why are these changes needed?

Some small refactors to improve readabilty, simplify and optimize.

Improvements

• The refactored code reduces the overall complexity and the number of iterations over the messages.
• Consolidated Truncation Logic: The logic for truncating each message to max_tokens_per_message and trimming the list to max_messages is more straightforward and easier to follow. It consolidates several steps into fewer lines of code.
• Total tokens count excludes system to be apples to apples comparison to processed_messages_tokens
• truncate_str_to_tokens requires less iterations. Iterate over tokens instead of characters.

Related issue number

None

Checks

• [X ] I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
• I've added tests (if relevant) corresponding to the changes introduced in this PR.
• I've made sure all auto checks have passed.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (2750391) 39.33% compared to head (350aa4a) 61.97%.
Report is 1 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #1631       +/-   ##
===========================================
+ Coverage   39.33%   61.97%   +22.63%     
===========================================
  Files          57       57               
  Lines        6096     6093        -3     
  Branches     1365     1481      +116     
===========================================
+ Hits         2398     3776     +1378     
+ Misses       3502     1974     -1528     
- Partials      196      343      +147     

Flag	Coverage Δ
unittests	61.75% <100.00%> (+22.42%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dkirsche]

@dkirsche please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

• (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

• (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

@microsoft-github-policy-service agree

[dkirsche]

@dkirsche please look at the commits I submitted here: dkirsche#1

Especially pay attention to the second commit which added a new test case and fixed a bug. Lmk what you think!

Your changes made sense. I pulled them into the PR.

[gagb]

@dkirsche please look at the commits I submitted here: dkirsche#1
Especially pay attention to the second commit which added a new test case and fixed a bug. Lmk what you think!

Your changes made sense. I pulled them into the PR.

Thank you so much @dkirsche! Also, I think we need to improve this function to handle the case when the message history has tool_calls and responses. But we can fix this in a new PR.

[dkirsche]

Some of the checks are in a waiting state. @ekzhu I believe I need your review before proceeding.