Add code interpreter for GptAssistant Notebook.

[LeoLjl]

Why are these changes needed?

Add code interpreter for GptAssistant Notebook.

Related issue number

#602

Checks

• I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
• I've added tests (if relevant) corresponding to the changes introduced in this PR.
• I've made sure all auto checks have passed.

[LeoLjl]

@gagb @IANTHEREAL Is it possible to see how GPTAssistant is interacting with its code interpreter?

[codecov-commenter]

Codecov Report

Merging #644 (1b77e98) into main (c1d6a5e) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #644   +/-   ##
=======================================
  Coverage   28.72%   28.72%           
=======================================
  Files          27       27           
  Lines        3380     3380           
  Branches      760      760           
=======================================
  Hits          971      971           
  Misses       2338     2338           
  Partials       71       71           

Flag	Coverage Δ
unittests	28.66% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

[LeoLjl]

Also, if there is a better example of illustrating how code interpreter works, please feel free to comment.

[gagb]

Tried to run, but got the following error:

ValueError: Unexpected run status: failed. Full run info:

{
  "id": "run_wqG41r4RvPJ4RpLM9X8S3o8J",
  "assistant_id": "--",
  "cancelled_at": null,
  "completed_at": null,
  "created_at": 1699814958,
  "expires_at": null,
  "failed_at": 1699814959,
  "file_ids": [],
  "instructions": "You are an expert at solving math questions. Write code and run it to solve math problems. Reply TERMINATE when the task is solved and there is no problem.",
  "last_error": {
    "code": "server_error",
    "message": "Sorry, something went wrong."
  },
  "metadata": {},
  "model": "gpt-4-1106-preview",
  "object": "thread.run",
  "required_action": null,
  "started_at": 1699814958,
  "status": "failed",
  "thread_id": "thread_L0uUpMpz216Sjhwmxsfq2joz",
  "tools": [
    {
      "type": "code_interpreter"
    }
  ]
})

[sonichi]

Is the error caused by the openai version ? I bumped the openai version to the latest in #645

[gagb]

Is the error caused by the openai version ? I bumped the openai version to the latest in #645

It's working now.

[IANTHEREAL]

@gagb @IANTHEREAL Is it possible to see how GPTAssistant is interacting with its code interpreter?

Try to use code interpreter to generate a image, does it sound better? (such as, Draw a line chart to show the number of issues and pull requests for the repository 'microsoft/autogen' over the past 3 months)

After getting file id, then user proxy (maybe) can support to display it using code

from PIL import Image
import io
from IPython.display import display

api_response = oss_analyst._openai_client.files.with_raw_response.retrieve_content("file-sn9E3nGJRUlcHaimmtHMMuzz")

if api_response.status_code == 200:
    content = api_response.content
    image_data_bytes = io.BytesIO(content)
    image = Image.open(image_data_bytes)
    display(image)

[LeoLjl]

@gagb @IANTHEREAL Is it possible to see how GPTAssistant is interacting with its code interpreter?

Try to use code interpreter to generate a image, does it sound better? (such as, Draw a line chart to show the number of issues and pull requests for the repository 'microsoft/autogen' over the past 3 months)

After getting file id, then user proxy (maybe) can support to display it using code

from PIL import Image
import io
from IPython.display import display

api_response = oss_analyst._openai_client.files.with_raw_response.retrieve_content("file-sn9E3nGJRUlcHaimmtHMMuzz")

if api_response.status_code == 200:
    content = api_response.content
    image_data_bytes = io.BytesIO(content)
    image = Image.open(image_data_bytes)
    display(image)

It seems that gpt are not allowed to do web scraping. I asked it your question and it constantly asked me to provide issue and PR number data. So I uploaded an example with gpt synthesizing data.