[.Net] Support raw-data in ImageMessage #2552

DavidLuong98 · 2024-04-30T20:00:18Z

Why are these changes needed?

This PR supports data url for ImageMessage which will be sent to ChatCompletion API's (Azure OpenAi and Open AI )

Add BinaryData to ImageMessage and update callers that use Uri to consider BinaryData when necessary.
Introduce a simple example of the usage for ImageMessage using the BinaryData constructor overload.
Update unit tests to reflect the change.

Related issue number

Closes #1893

Checks

I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
I've added tests (if relevant) corresponding to the changes introduced in this PR.
I've made sure all auto checks have passed.

…tnet

[skip ci]

…tnet

* add discord link and update package description * Update getting-start.md

… sync (microsoft#2231) * update * rename RegisterPrintMessageHook to RegisterPrintMessage * update website * update update.md * fix link error

…nAIChatAgent (microsoft#2347) * update openai version && add sample for json output * add example in web * update update.md * update image url

* add mstral client * enable streaming support * add mistralClientAgent * add test for function call * add extension * add support for toolcall and toolcall result message * add support for aggregate message * implement streaming function call

* update existing examples to use messageCOnnector * add overview * add function call document * add example 14 * add mistral token count usage example * update version

gitguardian · 2024-04-30T22:19:02Z

⚠️ GitGuardian has uncovered 40 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
10404693	Triggered	Generic High Entropy Secret	`b0f1693`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`3f7c648`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`58e27f2`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`4d24b68`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`5903c9c`	test/oai/test_utils.py	View secret
10404662	Triggered	Generic CLI Secret	`d58f2c8`	.github/workflows/dotnet-release.yml	View secret
10404694	Triggered	Generic High Entropy Secret	`4d24b68`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`58e27f2`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`3f7c648`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`b0f1693`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`5903c9c`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`5903c9c`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`58e27f2`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`3f7c648`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`4d24b68`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`b0f1693`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`4d24b68`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`5903c9c`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`58e27f2`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`3f7c648`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`b0f1693`	test/oai/test_utils.py	View secret
10404662	Triggered	Generic CLI Secret	`43892f0`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`01b15c4`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`43892f0`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`4bb63f5`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`aa96c44`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`43892f0`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`01b15c4`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`01b15c4`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`2cf0768`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`aa96c44`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`2cf0768`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`1ca29af`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`4bb63f5`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`4bb63f5`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`43892f0`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`01b15c4`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`4bb63f5`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`1ca29af`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`0369ec7`	.github/workflows/dotnet-release.yml	View secret

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secrets safely. Learn here the best practices.
Revoke and rotate these secrets.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

* update * add sample project * revert notebook change back * update * update interactive version * add nuget package * refactor Message * update example * add azure nightly build pipeline * Set up CI with Azure Pipelines [skip ci] * Update nightly-build.yml for Azure Pipelines * add dotnet interactive package * add dotnet interactive package * update pipeline * add nuget feed back * remove dotnet-tool feed * remove dotnet-tool feed comment * update pipeline * update build name * Update nightly-build.yml * Delete .github/workflows/dotnet-ci.yml * update * add working_dir to use step * add initateChat api * update oai package * Update dotnet-build.yml * Update dotnet-run-openai-test-and-notebooks.yml * update build workflow * update build workflow * update nuget feed * update nuget feed * update aoai and sk version * Update InteractiveService.cs * add support for GPT 4V * add DalleAndGPT4V example * update example * add user proxy agent * add readme * bump version * update example * add dotnet interactive hook * update * udpate tests * add website * update index.md * add docs * update doc * move sk dependency out of core package * udpate doc * Update Use-function-call.md * add type safe function call document * update doc * update doc * add dock * Update Use-function-call.md * add GenerateReplyOptions * remove IChatLLM * update version * update doc * update website * add sample * fix link * add middleware agent * clean up doc * bump version * update doc * update * add Other Language * remove warnings * add sign.props * add sign step * fix pipelien * auth * real sign * disable PR trigger * update * disable PR trigger * use microbuild machine * update build pipeline to add publish to internal feed * add internal feed * fix build pipeline * add dotnet prefix * update ci * add build number * update run number * update source * update token * update * remove adding source * add publish to github package * try again * try again * ask for write pacakge * disable package when branch is not main * update * implement streaming agent * add test for streaming function call * update * fix microsoft#1588 * enable PR check for dotnet branch * add website readme * only publish to dotnet feed when pushing to dotnet branch * remove openai-test-and-notebooks workflow * update readme * update readme * update workflow * update getting-start * upgrade test and sample proejct to use .net 8 * fix global.json format && make loadFromConfig API internal only before implementing * update * add support for LM studio * add doc * Update README.md * add push and workflow_dispatch trigger * disable PR for main * add dotnet env * Update Installation.md * add nuget * refer to newtonsoft 13 * update branch to dotnet in docfx * Update Installation.md * pull out HumanInputMiddleware and FunctionCallMiddleware * fix tests * add link to sample folder * refactor message * refactor over IMessage * add more tests * add more test * fix build error * rename header * add semantic kernel project * update sk example * update dotnet version * add LMStudio function call example * rename LLaMAFunctin * remove dotnet run openai test and notebook workflow * add FunctionContract and test * update doc * add documents * add workflow * update * update sample * fix warning in test * reult length can be less then maximumOutputToKeep (microsoft#1804) * merge with main * add option to retrieve inner agent and middlewares from MiddlewareAgent * update doc * adjust namespace * update readme * fix test * use IMessage * more updates * update * fix test * add comments * use FunctionContract to replace FunctionDefinition * move AutoGen contrac to AutoGen.Core * update installation * refactor streamingAgent by adding StreamingMessage type * update sample * update samples * update * update * add test * fix test * bump version * add openaichat test * update * Update Example03_Agent_FunctionCall.cs * [.Net] improve docs (microsoft#1862) * add doc * add doc * add doc * add doc * add doc * add doc * update * fix test error * fix some error * fix test * fix test * add more tests * edits --------- Co-authored-by: ekzhu <ekzhu@users.noreply.github.com> * [.Net] Add fill form example (microsoft#1911) * add form filler example * update * fix ci error * [.Net] Add using AutoGen.Core in source generator (microsoft#1983) * fix using namespace bug in source generator * remove using in sourcegenerator test * disable PR test * Add .idea to .gitignore (microsoft#1988) * [.Net] publish to nuget.org feed (microsoft#1987) * publish to nuget * update ci * update dotnet-release * update release pipeline * add source * remove empty symbol package * update pipeline * remove tag * update installation guide * [.Net] Rename some classes && APIs based on doc review (microsoft#1980) * rename sequential group chat to round robin group chat * rename to sendInstruction * rename workflow to graph * rename some api * bump version * move Graph to GroupChat folder * rename fill application example * [.Net] Improve package description (microsoft#2161) * add discord link and update package description * Update getting-start.md * [.Net] Fix document comment from the most recent AutoGen.Net engineer sync (microsoft#2231) * update * rename RegisterPrintMessageHook to RegisterPrintMessage * update website * update update.md * fix link error * [.Net] Enable JsonMode and deterministic output in AutoGen.OpenAI OpenAIChatAgent (microsoft#2347) * update openai version && add sample for json output * add example in web * update update.md * update image url * [.Net] Add AutoGen.Mistral package (microsoft#2330) * add mstral client * enable streaming support * add mistralClientAgent * add test for function call * add extension * add support for toolcall and toolcall result message * add support for aggregate message * implement streaming function call * track (microsoft#2471) * [.Net] add mistral example (microsoft#2482) * update existing examples to use messageCOnnector * add overview * add function call document * add example 14 * add mistral token count usage example * update version * Update dotnet-release.yml (microsoft#2488) * update * revert gitattributes * WIP : Binary ImageMessage * WIP : Able to pass unit test * Add example, cover more usages * Rename File --------- Co-authored-by: XiaoYun Zhang <xiaoyuz@microsoft.com> Co-authored-by: Xiaoyun Zhang <bigmiao.zhang@gmail.com> Co-authored-by: mhensen <mh@webvize.nl> Co-authored-by: ekzhu <ekzhu@users.noreply.github.com> Co-authored-by: Krzysztof Kasprowicz <60486987+Krzysztof318@users.noreply.github.com> Co-authored-by: luongdavid <luongdavid@microsoft.com>

LittleLittleCloud and others added 30 commits December 8, 2023 11:17

update

521b76a

add sample project

5ad414c

revert notebook change back

75e4e69

update

56e5741

Merge branch 'main' into dotnet

cc254b4

update interactive version

ddb2d69

Merge branch 'dotnet' of https://github.com/microsoft/autogen into do…

e6b7417

…tnet

add nuget package

72f1778

refactor Message

cebc2ed

update example

0744f0e

Merge branch 'main' into dotnet

a1d98ab

Merge branch 'main' into dotnet

9538a0b

add azure nightly build pipeline

cb198ac

Merge branch 'dotnet' of https://github.com/microsoft/autogen into do…

36f09b1

…tnet

Set up CI with Azure Pipelines

0177888

[skip ci]

Update nightly-build.yml for Azure Pipelines

048ee55

add dotnet interactive package

3bbfee1

add dotnet interactive package

b27012c

Merge branch 'dotnet' of https://github.com/microsoft/autogen into do…

d33066d

…tnet

update pipeline

2f558af

add nuget feed back

b22c55b

remove dotnet-tool feed

9556c3e

remove dotnet-tool feed comment

970c6f1

update pipeline

63eec7d

update build name

638f37e

Update nightly-build.yml

0b5c1ad

Delete .github/workflows/dotnet-ci.yml

37da646

update

c05a1dd

Merge branch 'dotnet' of https://github.com/microsoft/autogen into do…

bc6f3ab

…tnet

Merge branch 'main' into dotnet

b011bbb

LittleLittleCloud and others added 18 commits March 27, 2024 04:15

[.Net] Improve package description (microsoft#2161)

025684d

* add discord link and update package description * Update getting-start.md

[.Net] Fix document comment from the most recent AutoGen.Net engineer…

8214a21

… sync (microsoft#2231) * update * rename RegisterPrintMessageHook to RegisterPrintMessage * update website * update update.md * fix link error

[.Net] Enable JsonMode and deterministic output in AutoGen.OpenAI Ope…

27f172f

…nAIChatAgent (microsoft#2347) * update openai version && add sample for json output * add example in web * update update.md * update image url

Merge branch 'main' into dotnet

e650cf0

track (microsoft#2471)

1a60660

[.Net] add mistral example (microsoft#2482)

7cce072

* update existing examples to use messageCOnnector * add overview * add function call document * add example 14 * add mistral token count usage example * update version

Merge branch 'main' into dotnet

7cadfb0

Update dotnet-release.yml (microsoft#2488)

09de3cc

Merge branch 'main' into dotnet

de2b3a6

Merge branch 'main' into dotnet

cc4564f

update

f34feb6

revert gitattributes

999371a

Merge branch 'main' into dotnet

3c61969

WIP : Binary ImageMessage

737b5c7

WIP : Able to pass unit test

c5f30ac

Add example, cover more usages

f5d8b39

Rename File

0eaa5b4

ekzhu requested a review from LittleLittleCloud April 30, 2024 20:52

Merge

311fe47

DavidLuong98 changed the base branch from dotnet to main April 30, 2024 22:17

LittleLittleCloud approved these changes May 1, 2024

View reviewed changes

Merge remote-tracking branch 'origin/main' into u/davidluong98/1893

0461cdd

sonichi added this pull request to the merge queue May 1, 2024

github-merge-queue bot removed this pull request from the merge queue due to no response for status checks May 2, 2024

LittleLittleCloud added this pull request to the merge queue May 2, 2024

Merged via the queue into microsoft:main with commit f4a07ff May 2, 2024
17 of 18 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[.Net] Support raw-data in ImageMessage #2552

[.Net] Support raw-data in ImageMessage #2552

DavidLuong98 commented Apr 30, 2024

gitguardian bot commented Apr 30, 2024 •

edited

Loading

[.Net] Support raw-data in ImageMessage #2552

[.Net] Support raw-data in ImageMessage #2552

Conversation

DavidLuong98 commented Apr 30, 2024

Why are these changes needed?

Related issue number

Checks

gitguardian bot commented Apr 30, 2024 • edited Loading

⚠️ GitGuardian has uncovered 40 secrets following the scan of your pull request.

gitguardian bot commented Apr 30, 2024 •

edited

Loading