OpenRouter support for Langsmith

[kvyb]

Feature request

OpenRouter is very popular as a provider for LLM inference, and should be available for use in playground with all of its models, next to OpenAI etc.

Motivation

Its a pity that it is not available in LangSmith Playground. Meaning I can't really do experiments with traces since all of our production inference is done on OpenRouter LLMs, and some of these LLMs are not even available with other providers.

[hinthornw]

Will add to backlog, and if this ticket gets a lot of demand we can re-prioritize

[kvyb]

Thanks

[hahaCrazy]

Some providers of openrouter are very unstable, and problems often occur in the model. If you do not specify a provider, when calling the api, the providers will be sorted by openrouter's default order. This order changes dynamically. Therefore, it is often easy for the model to fail to work properly.

The order in which langchain adds providers to openrouter is really important!

[arvin8]

Measuring feature demend based on github tickets can't be seen as a good product practice. There should be demand since openrouter is a very popular service and is the only service. But the demend for it might not be visible on this ticket which is open on a development platform like github. Demend here will come from developers but choice of tool will rarely come from the coding developer but higher managment.

[winglian]

Is it not sufficient to simply change the api base to point to openrouter for langsmith integration to work?

[hinthornw]

If it is openai compatible than it would be

[kvyb]

It seems to be in fact openai compatible

[hahaCrazy]

Some providers of openrouter are very unstable, and problems often occur in the model. If you do not specify a provider, when calling the api, the providers will be sorted by openrouter's default order. This order changes dynamically. Therefore, it is often easy for the model to fail to work properly.

The order in which langchain adds providers to openrouter is really important!

[arvin8]

need to have an option to define the provider as well.