Retrieve prompts based on versioning rather than times for caching

[ajram23]

Describe the feature or potential improvement

Prompts don't change in seconds or minutes. And I don't want to fetching the prompt from your server for every call. What would be be better is for the prompts to be cached and provide a mechanism for looking for updates(for newer versions) and updating the cache. And not make this part of the execution pipeline.

Additional information

Submitting as requested by Clemens from Langfuse based on our conversation on Intercom!

[marcklingen]

Thanks for the suggestion!

Current behavior: prompts are refetched (blocking) after the configured cache TTL, they are not fetched on every call

Suggested behavior: prompt are refetched after configured cache ttl but stale cache is served. I'd probably rather not have prompts refetch after first use on scheduled interval in background, this has the risk of refetching prompts which are not frequently used

What do you think about this @ajram23?

[ajram23]

@marcklingen To be honest, cache seems like a hack when you have versions. Curious why would you refetch prompts which are not frequently used if their versions don't change? May be I am missing something.

[marcklingen]

awesome! let me know if the new prompt caching works as expected in your environment, until now, I got a lot of positive feedback on this

[ajram23]

@marcklingen So far so good. Glad that you are! I must be submitting good feature requests then ;)

[marcklingen]

I must be submitting good feature requests then ;)

yes, thank you!

offtopic: do you have a new public demo of your application by the way? loved the demo you once showed me

[ajram23]

@marcklingen not yet, packaging on a Mac sucks! Fighting that battle now, will reach out once its done!

[marcklingen]

best of luck, i can imagine