Add LiteLLM+instructor (for structured output) backend for curator

[CharlieJCJ]

Closes: #74
Closes #179
Closes #164

Changes:

• Included two example scripts, illustrating litellm usage in prompting and structured output modes.
  /examples/litellm_recipe_prompting.py
/examples/litellm_recipe_structured_output.py (Note: need to put OpenAI and Anthropic keys in environment).
• Included a backend parameter when using prompter code link, right now defaults to OpenAI.
• Integrated with instructor for structure output support for a large coverage of models. For models that can't use litellm + instructor (structured output), used a try except block before dataset generation to check whether instructor works on a simple example code link.
• Added time and cost logging (cost using litellm.completion_cost, if model cost is in the community-maintained mapping here)
• Added estimate_total_tokens that includes estimate_output_tokens which derives from get_max_tokens that gets max output token of the specified model. code link
• Using the same async, retry strategy as OpenAI Online Request Processor
• Get litellm rate limit through hidden params dict, including x-ratelimit-limit-requests, and x-ratelimit-limit-tokens for rpm and tpm.
• litellm refactoring base online request processor #188
  • Includes the breakdown of the new abstract class, and instructions for any new inheritance of OnlineRequestProcessor.
• Implement a robust OpenAI online request processor's check_structured_output_support.
  • A naive check is to check Model Versions: Structured Outputs are supported by specific model versions: gpt-4o-mini version 2024-07-18 and later gpt-4o version 2024-08-06 and late
  • Closes openai online & batch only supported models are gpt4o and gpt4o-mini #164

Future Works:

• Support a baseline structured output strategy for models from inference platforms that do not support structured outputs from litellm. Right now, will directly prompt user that the current model doesn't support structured output. Note that litellm structured output model / provider coverage isn't good, link. Need additional research.
• More performance optimization for auto rate limiting / retry strategies, need battle testing & experiments and comparisons.

Example curator-viewer's view:

Tested on the following models, all working for litellm + instructor structured output.

"claude-3-5-sonnet-20240620", # https://docs.litellm.ai/docs/providers/anthropic # anthropic has a different hidden param tokens structure. 
"claude-3-5-haiku-20241022",
"claude-3-haiku-20240307",
"claude-3-opus-20240229",
"claude-3-sonnet-20240229",
"gpt-4o-mini", # https://docs.litellm.ai/docs/providers/openai
"gpt-4o-2024-08-06",
"gpt-4-0125-preview",
"gpt-3.5-turbo-1106",
"gemini/gemini-1.5-flash", # https://docs.litellm.ai/docs/providers/gemini; https://ai.google.dev/gemini-api/docs/models # 20-30 iter/s
"gemini/gemini-1.5-pro", # 20-30 iter/s
"together_ai/meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo", # https://docs.together.ai/docs/serverless-models
"together_ai/meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo",
"together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1",

Note that the following models does not support structured output (i.e. response_format in Prompter)

# "together_ai/nvidia/Llama-3.1-Nemotron-70B-Instruct-HF", # instructor not supported
# "deepinfra/nvidia/Llama-3.1-Nemotron-70B-Instruct" # instructor not supported

[CharlieJCJ]

Works for Claude model_name="claude-3-opus-20240229"

[CharlieJCJ]

[CharlieJCJ]

After #149's merge (that resolves #145), do performance comparison of litellm vs. OpenAI request processors

[CharlieJCJ]

#159 is been merged, now costs have been appropriately logged. litellm also supports cost logging now.

[CharlieJCJ]

TODO

• Need to investigate why same prompt yield different prompt token issue [litellm+instructor] same prompt, but different prompt tokens, why? #166

[CharlieJCJ]

Need to add a better default timeout.

[CharlieJCJ]

Request review / approval @RyanMarten @vutrung96

[RyanMarten]

Small changes - let me know when they are addressed and I'll do another review

[vutrung96]

LGTM!