Added OpenRouter support and model testing #1058
Open
+94
−1
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In response to Issue #991:
This PR adds support for OpenRouter integration to enable testing across multiple visual models. By implementing this, users can now configure and use OpenRouter models to handle various tasks and evaluate model performance on different difficulty levels.
Key Changes:
OpenRouter API Integration:
Added functions to handle requests to OpenRouter’s /chat/completions endpoint.
Updated configuration to support API key input for OpenRouter during setup.
Included environment variable handling for OPENROUTER_API_KEY and default model options.
Model Testing Support:
Implemented test_openrouter_model() function, which tests OpenRouter’s visual models by sending sample payloads and returning structured responses.
Enhanced configuration to allow users to select specific OpenRouter models (e.g., ANTHROPIC/CLAUDE-3.5-SONNET, meta-llama/llama-3.2-90b-vision-instruct, google/gemini-flash-1.5-8b) during setup.
Testing and Validation:
Functional Testing: Successfully tested the get_openrouter_response() and test_openrouter_model() functions to verify model response and error handling.
Pre-commit Hooks: Code was tested to ensure it complies with pre-commit requirements. Please review for MyPy and Alembic issues.
Important
Added OpenRouter support with API integration, model testing, and setup configuration.
get_openrouter_response()
inopenrouter.py
to handle requests to OpenRouter's/chat/completions
endpoint.setup.sh
to include OpenRouter API key configuration and model options.test_openrouter_model()
inmodel_testing.py
to test OpenRouter models with sample payloads.setup.sh
to support OpenRouter API key input and model selection during setup.get_openrouter_response()
andtest_openrouter_model()
.This description was created by for 8ddd110. It will automatically update as commits are pushed.