Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validate the OpenAI API key format #1635

Merged
merged 5 commits into from
Feb 14, 2024

Conversation

gunnarku
Copy link
Contributor

Increase the amount of internal validation for OpenAI API keys. The intent is to shorten the debugging loop in case of invalid keys and typos. The changes do not add validation for Azure OpenAI API keys.

  • Add the validation in __init__ of OpenAIClient.

  • Introduce the MOCK_OPEN_AI_API_KEY constant for testing.

  • Add unit test coverage for the is_valid_api_key function.

Why are these changes needed?

The intent is to shorten the debugging loop in case of invalid keys and typos. In addition, we increase the correctness guarantees for OpenAIClient, i.e., the OpenAIClient instance will be created only with the API key in a valid format.

If anyone has a definition of constraints for Azure OpenAI API keys, then that will be a next step to increase the validation.

Topics for discussion:

  • Could logger.warning be more appropriate than raise ValueError(...)?
  • Currently there is no output of self._oai_client.api_key by design to avoid leaking API keys. Is this overly paranoid?

Related issue number

Checks

@gunnarku
Copy link
Contributor Author

@microsoft-github-policy-service agree

Copy link
Contributor

@sonichi sonichi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To help with debugging when a config_list is provided, we can add some info in the error message about the index of the config in the list which causes this error, by catching the error at line 359 of client.py.

@codecov-commenter
Copy link

codecov-commenter commented Feb 12, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (ad9e5cf) 37.80% compared to head (f6a0ab3) 57.42%.
Report is 2 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff             @@
##             main    #1635       +/-   ##
===========================================
+ Coverage   37.80%   57.42%   +19.61%     
===========================================
  Files          50       50               
  Lines        5726     5733        +7     
  Branches     1296     1406      +110     
===========================================
+ Hits         2165     3292     +1127     
+ Misses       3390     2179     -1211     
- Partials      171      262       +91     
Flag Coverage Δ
unittests 57.40% <100.00%> (+19.59%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@gunnarku
Copy link
Contributor Author

To help with debugging when a config_list is provided, we can add some info in the error message about the index of the config in the list which causes this error, by catching the error at line 359 of client.py.

@sonichi: Thank you for guidance! I will make two updates to the current set of changes:

  • Fix the root cause (missing api_key update in test_llava.py that I didn't catch locally. Output from the test log(s) was helpful.
  • Add a warning in client.py as you proposed.

Updates en route.

Increase the amount of internal validation for OpenAI API keys. The intent is
to shorten the debugging loop in case of typos. The changes do *not* add
validation for Azure OpenAI API keys.

* Add the validation in `__init__` of `OpenAIClient`. We'll log a
  warning when the OpenAI API key isn't valid.

* Introduce the `MOCK_OPEN_AI_API_KEY` constant for testing.

* Add unit test coverage for the `is_valid_api_key` function.
@sonichi sonichi added this pull request to the merge queue Feb 14, 2024
Merged via the queue into microsoft:main with commit f68c09b Feb 14, 2024
50 of 57 checks passed
whiskyboy pushed a commit to whiskyboy/autogen that referenced this pull request Apr 17, 2024
* Validate the OpenAI API key format

Increase the amount of internal validation for OpenAI API keys. The intent is
to shorten the debugging loop in case of typos. The changes do *not* add
validation for Azure OpenAI API keys.

* Add the validation in `__init__` of `OpenAIClient`.

* Introduce the `MOCK_OPEN_AI_API_KEY` constant for testing.

*  Add unit test coverage for the `is_valid_api_key` function.

* Validate the OpenAI API key format

Increase the amount of internal validation for OpenAI API keys. The intent is
to shorten the debugging loop in case of typos. The changes do *not* add
validation for Azure OpenAI API keys.

* Add the validation in `__init__` of `OpenAIClient`.

* Introduce the `MOCK_OPEN_AI_API_KEY` constant for testing.

*Add unit test coverage for the `is_valid_api_key` function.

* Log a warning when register a default client fails.

* Validate the OpenAI API key format

Increase the amount of internal validation for OpenAI API keys. The intent is
to shorten the debugging loop in case of typos. The changes do *not* add
validation for Azure OpenAI API keys.

* Add the validation in `__init__` of `OpenAIClient`. We'll log a
  warning when the OpenAI API key isn't valid.

* Introduce the `MOCK_OPEN_AI_API_KEY` constant for testing.

* Add unit test coverage for the `is_valid_api_key` function.

* Check for OpenAI base_url before API key validation

---------

Co-authored-by: Chi Wang <wang.chi@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants