Caching for chat based models.

[gd1m3y]

Is there any docs or related issues for caching the Azure chat open ai responses i cannot find one.

[CelestialCoder03]

Is there any docs or related issues for caching the Azure chat open ai responses i cannot find one.

I found a possible solution from the release note of gptcache. It worked with ChatOpenAI class, not so sure about Azure though.

from langchain.chat_models import ChatOpenAI
from gptcache import cache
from gptcache.adapter.langchain_models import LangChainChat

def get_msg(data, **_):
    return data.get("messages")[-1].content


cache.init(
    pre_embedding_func=get_msg,
)

chat = LangChainChat(chat=ChatOpenAI(temperature=0))

[PhilipMay]

I can confirm, that AzureChatOpenAI does not use caching at the moment.

Cache init by:

from langchain.cache import SQLiteCache
import langchain
langchain.llm_cache = SQLiteCache(database_path="./langchain-cache.db")

LangChain Version 0.0.166

Would be nice if this can be added. Thanks!

[jakobsa]

relates (#1644)

[iljoong]

Is there any docs or related issues for caching the Azure chat open ai responses i cannot find one.

I found a possible solution from the release note of gptcache. It worked with ChatOpenAI class, not so sure about Azure though.

from langchain.chat_models import ChatOpenAI
from gptcache import cache
from gptcache.adapter.langchain_models import LangChainChat

def get_msg(data, **_):
    return data.get("messages")[-1].content


cache.init(
    pre_embedding_func=get_msg,
)

chat = LangChainChat(chat=ChatOpenAI(temperature=0))

I have tested on Azure OpenAI and it works!

from langchain.chat_models import AzureChatOpenAI
from gptcache.adapter.langchain_models import LangChainChat
from gptcache import cache

def get_msg(data, **_):
    return data.get("messages")[-1].content


cache.init(
    pre_embedding_func=get_msg,
)

chat = LangChainChat(chat=AzureChatOpenAI(temperature=0.0, deployment_name='gpt-35-turbo'))

from langchain.schema import (
    AIMessage,
    HumanMessage,
    SystemMessage
)

messages = [HumanMessage(content="Tell me a very long version of joke")]
--
%%time 
chat(messages)
> CPU times: user 15.1 ms, sys: 0 ns, total: 15.1 ms
> Wall time: 16.7 s
--
%%time
chat(messages)
> CPU times: user 238 µs, sys: 74 µs, total: 312 µs
> Wall time: 313 µs

[Praj-17]

@czuo0303 I tried the same code for using ChatOpenAI but I am facing the following error, which version of langchain and gptcache are you on ? Also, can someone, please help me figure out this error?

My Code

from langchain.chat_models import ChatOpenAI
from langchain.schema import HumanMessage, SystemMessage
from gptcache import cache
from gptcache.adapter.langchain_models import LangChainChat,LangChainLLMs
from gptcache.embedding import  Onnx
from gptcache.manager import CacheBase, VectorBase, get_data_manager
from gptcache.similarity_evaluation.distance import SearchDistanceEvaluation
onnx = Onnx()

data_manager = get_data_manager(CacheBase("sqlite"), VectorBase("faiss", dimension=onnx.dimension))
cache.init(
    embedding_func=onnx.to_embeddings,
    data_manager=data_manager,
    similarity_evaluation=SearchDistanceEvaluation(),
    )
cache.set_openai_key()

chat = LangChainChat(chat=ChatOpenAI(temperature=0))

This gives me the following error

Traceback (most recent call last): File "e:/CODING PLAYGROUND/CODE/YoZu/Updated Backend/Chat/actions/langchain_chat_completion.py", line 20, in <module> chat = LangChainChat(chat=ChatOpenAI(temperature=0)) File "C:\Users\pwayk\anaconda3\envs\yozu2\lib\site-packages\langchain\load\serializable.py", line 74, in __init__ super().__init__(**kwargs) File "pydantic\main.py", line 340, in pydantic.main.BaseModel.__init__ File "pydantic\main.py", line 1066, in pydantic.main.validate_model File "pydantic\fields.py", line 439, in pydantic.fields.ModelField.get_default File "pydantic\utils.py", line 695, in pydantic.utils.smart_deepcopy File "C:\Users\pwayk\anaconda3\envs\yozu2\lib\copy.py", line 172, in deepcopy y = _reconstruct(x, memo, *rv) File "C:\Users\pwayk\anaconda3\envs\yozu2\lib\copy.py", line 270, in _reconstruct state = deepcopy(state, memo) File "C:\Users\pwayk\anaconda3\envs\yozu2\lib\copy.py", line 146, in deepcopy y = copier(x, memo) File "C:\Users\pwayk\anaconda3\envs\yozu2\lib\copy.py", line 230, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "C:\Users\pwayk\anaconda3\envs\yozu2\lib\copy.py", line 146, in deepcopy y = copier(x, memo) File "C:\Users\pwayk\anaconda3\envs\yozu2\lib\copy.py", line 237, in _deepcopy_method return type(x)(x.__func__, deepcopy(x.__self__, memo)) File "C:\Users\pwayk\anaconda3\envs\yozu2\lib\copy.py", line 172, in deepcopy y = _reconstruct(x, memo, *rv) File "C:\Users\pwayk\anaconda3\envs\yozu2\lib\copy.py", line 270, in _reconstruct state = deepcopy(state, memo) File "C:\Users\pwayk\anaconda3\envs\yozu2\lib\copy.py", line 146, in deepcopy y = copier(x, memo) File "C:\Users\pwayk\anaconda3\envs\yozu2\lib\copy.py", line 230, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "C:\Users\pwayk\anaconda3\envs\yozu2\lib\copy.py", line 172, in deepcopy y = _reconstruct(x, memo, *rv) File "C:\Users\pwayk\anaconda3\envs\yozu2\lib\copy.py", line 270, in _reconstruct state = deepcopy(state, memo) File "C:\Users\pwayk\anaconda3\envs\yozu2\lib\copy.py", line 146, in deepcopy y = copier(x, memo) File "C:\Users\pwayk\anaconda3\envs\yozu2\lib\copy.py", line 230, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "C:\Users\pwayk\anaconda3\envs\yozu2\lib\copy.py", line 161, in deepcopy rv = reductor(4) TypeError: cannot pickle 'onnxruntime.capi.onnxruntime_pybind11_state.InferenceSession' object

I am using langchain-0.0.263 and gptcache-0.1.39.1

[etlevents]

me too

[SimFG]

it will be fixed, detail: zilliztech/GPTCache#538

[dosubot]

Hi, @gd1m3y,

I'm helping the LangChain team manage their backlog and am marking this issue as stale. From what I understand, you were looking for documentation or related issues on caching Azure chat OpenAI responses, and there have been several comments discussing potential solutions and related issues. One user confirmed that AzureChatOpenAI does not currently use caching, while another user provided a possible solution using gptcache. Additionally, there were discussions around code shared by a user and an error encountered when trying to use ChatOpenAI. Lastly, a user mentioned that the issue will be fixed and provided a link to a pull request for more details.

Could you please confirm if this issue is still relevant to the latest version of the LangChain repository? If it is, please let the LangChain team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or the issue will be automatically closed in 7 days.

Thank you for your understanding and cooperation.