OpenAI Prompt Caching: Add cached_token parameter in usage_metadata of AI response

[ShubhamMaddhashiya-bidgely]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

OpenAI recently introduced a prompt caching feature that is automatically available for newer models. With this update, OpenAI's responses now include details on how many cached tokens were used for a request.

The current response schema includes the following structure:

{
  "completion_tokens": 93,
  "prompt_tokens": 14155,
  "total_tokens": 14248,
  "completion_tokens_details": {
    "audio_tokens": null,
    "reasoning_tokens": 0
  },
  "prompt_tokens_details": {
    "audio_tokens": null,
    "cached_tokens": 4480
  }
}

Adding support for these new fields, specifically the cached_tokens parameter, would be highly beneficial for users. It would allow them to monitor caching hit rates and optimize token usage more effectively.

This enhancement would align LangChain's capabilities with OpenAI's latest API features, providing users with greater visibility into prompt efficiency and performance.

Motivation

Adding support for these new fields, specifically the cached_tokens parameter, would be highly beneficial for users. It would allow them to monitor caching hit rates and optimize token usage more effectively.

Proposal (If applicable)

I'm not sure if this enhancement is already in development, but if not, I would love to contribute to implementing this enhancement.

[sagaruprety]

I too would like to see this feature in Langchain, if not already existing.

[ShubhamMaddhashiya-bidgely]

@sagaruprety This feature is now available in langchain-core 0.3.15

[sagaruprety]

Amazing, thanks @ShubhamMaddhashiya-bidgely. Can you point me to the documentation for this, as I can't find it rightaway.

[RafaelMCarvalho]

Awesome.
@sagaruprety, you don't really need to setup anything. The messages have the UsageMetada where you can find the cache hit info in the input_token_details key (you can get it when using the include_raw option for example).

[RafaelMCarvalho]

@ShubhamMaddhashiya-bidgely, I wonder if LangSmith will take the cache hit into consideration. The token usage and cost calculation on LangSmith are not considering the cache hit so far. Do you have any info about it?

[lolbus]

Seems like the new prompt caching weren;t supported to date.. (2 months since the release)

time to move on to a new orchestrator?