Query vLLM OpenAI /models endpoint to get model name and context window

[simon376]

Description

Similar to #755 , the OpenAI-compatible API server by vLLM exposes both the model name via id and the context length via max_model_len, which can be used to automatically setup the context length of the deployed LLM.

I'm not sure how to best check if the endpoint should be queried, e.g. by checking the API Base URL, or introducing a new apiType besides Azure, etc.

Checklist

• [x ] The base branch of this PR is dev, rather than main it's preview as in your contributing guidelines
• The relevant docs, if any, have been updated or created

References

• vLLM PR adding max_model_len
• Query server for model availability of OpenAI-compatible servers #609 Looks like this is closely related to the "AUTODETECT" functionality. however, that only uses the model names, not additional parameters as far as I am aware of (ref. load.ts:244). So it may be better to adjust the listModels() interface to include model parameters in its response

[netlify]

❌ Deploy Preview for continuedev failed.

Name	Link
🔨 Latest commit	ecb530e
🔍 Latest deploy log	https://app.netlify.com/sites/continuedev/deploys/669f7d4ca57f240008992099

[simon376]

sorry I messed something up, I only intended to include my one change in the OpenAI.ts file

[sestinj]

@simon376 I think this might be a good enough reason to make a subclass of the OpenAI class (we do this with Deepseek, Groq, and others for example). Everything can be the same except that there doesn't need to be a URL check to find out whether it is an instance of vLLM

[Patrick-Erichsen]

Hey @simon376 , any progress on this? Would love to get it shipped!

[simon376]

@sestinj I've created a new vLLM subclass and add documentation for it. Sorry for the whitespace changes in OpenAI.ts, couldn't figure out how to get rid of them

[sestinj]

Great, now worries on the whitespace this looks perfect

[agm-eratosth]

Hi @sestinj what version of the extension has this feature been integrated into?