Upcoming architecture changes for Langfuse 3.0 (self-hosted)

[maxdeichmann]

Hi all,

Langfuse is growing a lot, both in feature scope as well as in usage on single instances. Thus we plan for a couple of changes that will be released in Langfuse v3.

We are currently required to mature our architecture as we are working on the following challenges:
✅ Building model-based evals, which requires us to run asynchronous tasks, rate limited, with failover capabilities.
🧑‍🍳 Improve performance as instances scale out.

I wanted to give you a heads up on upcoming changes which are required to make these features work. Currently, Langfuse contains a single Docker container, which takes care of everything we do. This was fast to set up Langfuse initially, but we need more technical capabilities now. In addition to the existing components (Docker container + Postgres database), we will add the following:

• Asynchronous processing in a dedicated container
• Redis container (for queues)
• OLAP storage (likely Clickhouse) to enable lower latencies on analytical queries and scale out storage for large and multi modal inputs outputs

If you self-host Langfuse, this means that we will likely advise to change to the following setup to be able to benefit from new infra changes easily. We are happy to hear your thoughts on this:

• Migrate from a container service (e.g. GCP Cloud Run) to a Virtual Machine to run docker-compose from there. Docker-compose helps to quickly bootstrap the new components.
• Either provide connection strings to the docker-compose for Redis, Postgres, and Clickhouse for managed databases or run all of them on the virtual machine as well.

Feel free to share your thoughts below on these topics:

• Tool choice: do you have experience with the tools mentioned above and have advise for us?
• Do you have ideas on how to make self-hosting easier in a multi-container, multi-database setup than docker compose?
• Do you have ideas on how a great migration path could look like?

Find more context in the last Langfuse Townhall meeting. We will provide an easy to follow upgrade path for self-hosters once v3 is generally available. The infrastructure change does not affect public APIs, thus, users of Langfuse Cloud will not be affected by this change. Currently we pilot the async container & queue for the evals feature which is currently in public beta on Langfuse Cloud.

UPDATE JULY 22nd

A more detailled overview of planned changes:

Architectures v2 vs. v3

• Containers
  • web container: hosts public api, and all resources for the user interface
  • worker container: asynchronous processes, no exposed ports
• Databases
  • Redis used as cache and queue
  • Postgres stores transactional data such as projects or API keys
  • Clickhouse stores tracing data generated by the SDKs. This database will do most of the processing as our server will insert all the SDK data and read it for tables and dashboards.

Next to the core application, an application load balancer for TLS termination and routing of requests to the Web container is necessary. We use nginx but you can also use e.g. the fully managed AWS load balancer.

Upgrade path from v2 to v3

Thousands of teams run on Langfuse (~400k docker pulls)
→ we aim to offer the easiest migration experience that is automated and documented

• Guidance on scalability of dockerized datastores, when are fully-managed datastores necessary, including at-rest encryption with custom keys
• Guidance of reliable auto-scaling configuration of the application and databases
• Migration script to move data from Postgres to Clickhouse, and/or to Clickhouse and a new Postgres instance

Application deployment

You will be able to deploy the containers kia Kubernetes or your own Container Deployment Service (such as Google Cloud Run) or via docker compose on a virtual machine. In either case, you will also be able to use dockerized databases or you can provide us with connection strings for managed databases.

For low-volume/non-production deployments, dockerized DBs + docker compose is a sensible option to keep complexity low. We will publish guidance on when options 3 and 4 are necessary.

DB deployment

Databases (see above): redis, postgres, clickhouse

Low-volume

• All datastores are available as single docker containers
• Can be bundled in docker compose or EKS/ECS/K8S deployment

High-volume / fully-managed → databases external of application cluster

• Redis
  • No special requirements
  • AWS: we chose ElastiCache (Redis OSS) for a fully managed Redis instance
• Postgres
  • No special requirements compared to other applications.
  • AWS: we chose RDS for a fully-managed Postgres instance
• Clickhouse
  • Main data store, thus this database will scale the most.
  • AWS:
    • There is no RDS-equivalent for Clickhouse on AWS
    • We use Clickhouse Cloud which is the fully managed Clickhouse database managed by the Clickhouse team. This can be run on an AWS region of your choice with VPC peering for private access and data security.
    • There are potentially additional vendors for managed Clickhouse such as DoubleCloud or Altinity.
    • Alternatively, you can run Clickhouse yourself in ECS, e.g. by using this template.

We will provide guidance at which scale high-volume/fully-managed clickhouse is necessary. On hosted Langfuse, we are currently in the process of migrating to arrive at a scalable architecture. Once we are done with the migration, we will release 3.0. We will keep you posted here on updates regarding the migration.

FAQ

• Will v3 introduce any non-OSS components that require additional licensing? No, all new infrastructure components (Clickhouse, Redis/Valkey, S3/minio) can be self-hosted without any non-FOSS dependency or licensing requirements.
• How long will there be security updates for v2? We will support security updates for V2 until at least end of the year 2024. If you require extended support, please reach out to us.

Timeline

We currently test many of the v3 infrastructure pieces on Langfuse Cloud. We will release v3 once all of these changes are "battle-tested" to be sure that this is a smooth transition for everyone self-hosting Langfuse without uncertainties. While there is no strict timeline, we aim for a release in late November.

[MarkEdmondson1234]

As requested from Discord, my comment:

I really do not want to move off serverless infra to a dedicated VM, I'd say a major reason I chose Langfuse was its Cloud Run deployment I could couple with my existing AlloyDB. And perhaps AlloyDB is quick enough it doesn't need help with analytical queries.

Cloud Run recently introduced side car containers so perhaps that is an option?
https://cloud.google.com/blog/products/serverless/cloud-run-now-supports-multi-container-deployments

There is a managed Redis option too but its a bit more pricy.

My serverless deployment at the moment I don't pay for Langfuse until I'm browsing the UI or its capturing traces, aside the already sunk cost of the database.

[noorvir]

The current docs advise not to use Docker Compose for production. I'm guessing that will change for v3? I would be great to have a ready to deploy docker compose that just needs an env file to get started. I'd use the cloud hosted service but have relatively strict data privacy requirements.

[marcklingen]

awesome, thank you @mautini 🙏 We are currently finalizing many infra pieces, will definitely reach out!

[benobytes]

Our current stack uses langfuse in Kubernetes so I will be happy to assist as well!

[verdverm]

We use k8s operators for the redis, postgres, clickhouse components. Would want a way to have develop vs production setups, which is probably easy from the chart perspective, more about docs / examples

[marcklingen]

I think it's sensible to do this similar as the current chart, i.e. provide two options:

1. dev/staging, low-volume prod: deploy all components within k8s
2. high-volume prod: add api keys for dedicated/managed redis, clickhouse, postgres

[DevBey]

big up vote, k8s is the way to go.

[Manouchehri]

I'm hesitant on a more complicated docker-compose setup. One of the reasons we were able to open to using Langfuse to begin with, was how easy it was to deploy on a serverless platform.

[nikcaryo-super]

copying from discord, GCP offers: https://cloud.google.com/blog/products/serverless/cloud-run-now-supports-multi-container-deployments but I'm not sure exactly how those would work for asynchronous tasks (e.g. does the sidecar running an async task continue running if the request finishes and the cpu is unallocated?)

maybe there's an architecture where async tasks are just sent as webhooks to whatever "langfuse-worker" url people have setup? and then GCP people can just have a langfuse-worker serverless instance / pubsub task queue or whatever.

[MarkEdmondson1234]

That’s how the rest of my infra works, pubsub events to Cloud Run scale to 0 workers

[marcklingen]

agree, would love to figure out together how to best run this on Google Cloud Run to potentially scale to 0. The worker is necessary to power async workloads (evals, tokenization) but could also scale to zero if the queue could be in pub/sub (GCP specific). I/m unsure how to find a generalizable solution here but we'll figure it out

[fernandocamargoai]

It feels weird to deploy using docker-compose. For me, it's always have been a good tool to use during development, but not really for production.

My team is currently planning to deploy it. We haven't decided if we'll deploy to kubernetes or Cloud Run though. We'll stay tuned to see what will work best for us.

[Manouchehri]

As a team with k8s experience.. we try to avoid it at all costs. 😬 We like Langfuse enough that we'd probably cave and do it if needed, but not our first choice.

[marcklingen]

Agree that we’d want to make it work on GCP/CloudRun and comparable solutions as it is just much easier. There’s no way really around it as without an additional container, it’s seems close to impossible to run async workloads

[MarkEdmondson1234]

There are a few options for async tasks, like Batch, cloud run jobs, workflows? GCP specific though. k8s is most generic but yeah, I’d prefer that only as a last option

[zoltan-fedor]

We do run Langfuse today on k8s (our team has lots of k8s experience - we run everything there with the exception of databases).
I wrote our own Helm chart for Langfuse (includes HPA (autoscaling), ingress, etc), which we use for about 1/2 a year with no issues both in prod and preprod.
I would advise that Langfuse includes a Helm chart for production-level deployments next to the docker compose option, especially once it become multi-container with v3.0

Also Redis should be allowed to be external (running Redis in a container in K8s is not really a good architecture) and connection to it parametrized just like Postgres is. I am sure I will be able to make it so in our Helm charts, but it could be useful for those who are less experienced. We already run Redis external to K8s.
We have no Clickhouse, but I am sure we will be able to set that up on K8s if required.

EDIT: I just saw that further own @marcklingen has already described basically the same I was recommending here (external database connection option + official helm chart offered by Langfuse). See https://github.com/orgs/langfuse/discussions/1902#discussioncomment-9729809

[marcklingen]

thanks for sharing you thoughts @zoltan-fedor 🙏

[chai3]

I want Radis and OLAP Clickhouse to be keep opt-in.

I am deploying LangFuse on AWS with a simple architecture.
Just AWS AppRunner and RDS.
AWS AppRunner only supports a single Docker image and cannot use docker compose and sidecar.
ECS, EC2, Radis, and ALB are expensive, but AppRunner only needs to pay the CPU cost when there is a request by user, and if there is no request, it is very cheap at 0.007 USD / GB-hour(If 2GB memory, 5 USD/month)

https://github.com/AI4Organization/langfuse-ecr-ecs-deployment-cdk/blob/f21d237324f9c5727227915a57523a7ead779720/lib/langfuse-ecr-apprunner-deployment-cdk-stack.ts

[chai3]

I found a similar discussion and v2 implementation with AWS CDK

https://github.com/orgs/langfuse/discussions/2863

https://github.com/aaronsu11/langfuse-on-aws

[qymab]

Any specific reason why not providing arm based images ?

[marcklingen]

See self-hosting docs

Are there prebuilt ARM images available? No, currently we do not publish official ARM images. However, you can build your own ARM images using the Dockerfile in the Langfuse repository.

No specific reason, open for contributions on this!

[marcklingen]

First version to include arm image was just released: https://github.com/langfuse/langfuse/releases/tag/v2.58.0
Thanks again @rennokki for the contribution

[newcworld]

when clickhouse, I think it's great

[aiakubovich]

I'm not a big fan of docker-compose and we unable to use docker-compose files for deployments. Our requirements dictate that we first build an image using an ARM template, scan it with an infosec tool like Aqua in the CI/CD pipeline, and then deploy it. It would be great if there is option to deploy each container as stand along app.

[marcklingen]

makes sense, thanks for sharing!

[tiagoantao]

We are currently very happy users of Langfuse 2 (massive internal adoption success).
There are a few of issues for us with the new architecture:

• Maintaining a new OLAP solution. We would prefer not to have it or to be able to integrate it with our existing OLAP solution
• We would prefer a container only solution vs a VM based solution. This is purely because our current deployment is container based and we have no VMs
• Better yet, containers without nothing attached (e.g. Clickhouse would require attaching volumes).

Thank you for considering our requirements

[marcklingen]

Thanks for you feedback here!

The v3 infrastructure will most likely include clickhouse as it is a very solid yet OSS OLAP db that works well at high throughput and size yet has support for upserts which are required in some scenarios.

we are currently aiming for these 4 deployment options:

data stores (postgres, redis, clickhouse)	docker compose (vm)	helm chart (k8s)
dockerized	1, low volume	2, low-mid volume
external, connection provided via env	3, low-mid volume	4, high volume

We'd encourage deployments with external data stores (3&4) for improved scalability and less ops overhead (no additional storage attached to containers) as there are managed services for all 3 data stores, postgres/redis offered by all major cloud vendors, and clickhouse e.g. via Clickhouse Cloud. At low volume, deployment via docker should work as well and we cross-checked this with many teams who run low-volume clickhouse deployments in docker.

The only alternative would be to maintain an additional postgres-only flavor of Langfuse. However, we do not want to go this route as this would slow down development a lot and would make the two versions drift in performance. Context: many analytics use cases right now are blocked by this change as the postgres instance is limited by IOPS for many teams who deploy langfuse at scale.

[verdverm]

I'd love to see this integrate with my existing log/mon stack (grafana lgtm, but there are others)

Having to fuse traces & logs across them is not ideal

[mrout94]

Will there be a way to migrate existing traces, prompts and datasets from v2 to v3?

We have massive internal success and adoption rate for this and there are a lot of data, prompts and traces stored in the containers. We are really excited about v3 but also do not want to redo these steps again.

Wishing the entire team good luck for the v3 release. Looking forward to this.

[marcklingen]

There will be a migration guide and script that will bring the migration effort down to a minimum. We are very much aware that you and many other teams depend on Langfuse for their workflows.

[snikch]

Hey team. Thanks for the updates in the newsletter.

I agree on the general comments that docker compose isn't a great production solution, however I generally see docker compose as a great way to document the required setup for self hosters. It's simple to grok the required services and interactions / configuration.

We will continue to use a k8s deployment, and are happy to integrate into our existing Redis and Clickhouse services / spin up new instances. Bringing up Clickhouse and Redis shouldn't be considered a large operational hurdle IMO for people wanting to self host. With the reality of services becoming more complex over time, and features / scale are added, it is a reasonable expectation (Posthog is a good example).

As long as the migration path is documented and clear this all sounds great! 👏

[marcklingen]

Thanks for sharing and your trust in this general path! We’ll follow up with drafts of these docs in this thread when we have a stable end-to-end deployment running across all features in order to get feedback and help folks test v3 ahead of a stable release.

We talked a lot about this and are convinced that this change is necessary to serve all teams that want to scale with Langfuse. If your product is doing well and Langfuse breaks as the infra was kept as simple as possible, that’s not great. Agree that Posthog is a great example as some other OSS analytics projects deliberately run on Postgres and thus can’t keep up with Posthog from my point of view.

[marcklingen]

Update to the above: we plan to release v3 in July (no strict ETA yet) as we are currently going though many optimization steps to make the new setup as performant as possible. We will post an update here once there is documentation and a pre-release version to try.

[4t8dd]

Looking forward to the migration to clickhouse.

[marcklingen]

me too!

[nonlocalStream]

Excited for this! any updated timelines?

[marcklingen]

currently running parts of this in beta on Langfuse Cloud. No strict ETA yet, will release once it is stable and a simple one-way migration for everyone

[pamelafox]

I've received an inquiry as to how we'd be able to deploy your changes to Azure:
Azure-Samples/langfuse-on-azure#14

We can set up a Redis connection to Azure Cache for Redis. I'm not familiar with Clickhouse. Do you already have an idea about how we could get that working on Azure?

[marcklingen]

The architecture is not yet finalized but I'd suggest to deploy as mentioned here via either fully-managed services (as the one for Redis) or dockerized versions (e.g. Clickhouse): https://github.com/orgs/langfuse/discussions/1902#discussioncomment-9729809

This should work on Azure as will work on other cloud providers as well and we will share drafts of the docs on this ahead of the v3 release here. Would love to chat about potential adaptations to the Azure example once this is finalized.

[indranilr]

Would one then need to use managed services from AWS for Postgres, Redis and Clickhouse if hosting Langfuse as ECS containers ?

[marcklingen]

You will not need to use managed services. For low-mid volume deployments, you cam use dockerized versions of Redis, Postgres and Clickhouse or run them all on a single VM via docker compose. See this comment for more details: https://github.com/orgs/langfuse/discussions/1902#discussioncomment-9729809

[indranilr]

Any specific benchmarks/guidance around s to what is the criteria (numbers range) to classify as low, mid and high volume (I assuming this is Query TPS) ?

[marcklingen]

Not yet, we aim to provide guidance together with the v3 migration docs.

Core metrics:

• TPS of API
• Overall ingestion volume and order of magnitude of objects stored -> relevant for read-heavy APIs, can be approximated by #(Traces/Observations/Scores)/Timeframe

[luciowu]

Happy to see Langfuse is keep evolving!
We deployed Langfuse in k8s cluster in GCP. And in order to meet our OLAP requirements, we build our own ETL pipeline (PG -> BigQuery) to extract the data to BigQuery.
Hope to make Clickhouse as a plugin so that we could keep our current deployment architecture without change.

[clemra]

We've just updated the initial post with more information and guidance on the envisaged changes. Here attached in PDF format.
Langfuse_V3_Architecture.pdf

[abdullah-retorio]

Hello, AFAIK Redis is not open-source anymore. Does this mean we need a prod-licence of Redis to self-host on our VMs or can we still use their docker containers freely?

[clemra]

Hi @abdullah-retorio -- great point.

We've pinned the Redis version to ensure it's open source. We will swap it out with Valkey soon.

[zxkxyz]

Any update on the timeline for V3 release? We're looking to deploy self-hosted Langfuse but are hesitant to proceed with V2 given V3 is right around the corner...

[marcklingen]

Thanks for asking, currently we run large pieces of the v3 infrastructure on our cloud service already to test the setup and provide sufficient guidance once rolling it out to everyone self-hosting with v3. I'd recommend to get started with v2 today as the transition will be simple and might take another 1-2 months on our side. I have updated this posted accordingly.

[Freezaa9]

Hello,
So the low volume langfuse V3 we just be a container upgrade in a case of an existing deployment on GCP using Cloud run + Postgres. No additional Redis or clickhouse needed ?

If needed can we keep:
Langfuse in cloud run
Postgres in GCP Cloud SQL:

and add click house and redis on 2 separate cloud run container

In large organization, it is a pain (sometime not even possible) to deploy new service like that, and we don't need high volume yet. Some organizations don't have dedicated teams to manage a K8S and all new services need to pass by multiple committee (security ...). And VM are not authorized because it is hard to managed and serverless solution are prefered.

Thanks in advance !

[marcklingen]

Thanks for sharing your thoughts!

We aim to facilitate a straightforward low-volume deployment of Langfuse. Currently, we assume this will be based on Docker Compose, which may not be suitable for you since it cannot run serverless. We will do more research on potential options, considering these constraints. However, we prefer not to have multiple distributions of the applications targeting different datastores, as this would slow down the project. We want to stay focused on executing the overall roadmap as the market and requirements evolve quickly.

[chai3]

We will support security updates for V2 until at least end of the year 2024.

Please extend V2 security updates. For example, a few months after V3 GA.

[timoschd]

Hi, will caching of llm outputs like Helicone offers be included in v3 or is it in any way planned ? Your planned architecture is going to look quite similar to theirs.

[marcklingen]

As Langfuse is focused on async observability that is not in the critical path and does not add latency, caching is out of scope for now. I know some teams who use LiteLLM for this alongside Langfuse, especially for CI runs who benefit the most from caching.

[timoschd]

Thanks for your reply and hinting at LiteLLM. The path forward for Langfuse is now clearer for me.

[marcklingen]

Yes, highly recommended if you like the proxy approach to extend llm tracing/logging to many projects within your org. Usually projects then shift to async logging via the langfuse integrations over time for more in-depth tracing capabilities

[ninjaa]

Hi, I work with a leading US enterprise AI consultancy.

We love Langfuse V2 because:

• Data model really matches Metrics, Logging, Tracing & Evals excellently
• Open source containerized approach means we don't need to go through complex procurement workflows when presenting PoCs and prototyping
• Langfuse UX is really strong given the early stage - works for us for now
• Any data we can't get out of the frontend we feel comfortable fetching from Postgres directly using SQL as a last resort
• docker compose has us up and running locally and in stage on a VM in minutes. we were able to do an initial deploy on prod this way too, but expect to have to migrate to k8s.

Regarding the upgrade path, we have the following concerns:

• Development environment: We run Langfuse locally during development. Currently V2 only requires PostgreSQL and Langfuse which is ideal in terms of resource usage while developing. An analysis of the memory footprint and processing needs of V2 vs V3 would be very helpful. Rapid local development is a really important when prototyping AI apps.
• Relative unfamiliarity with ClickHouse vs Postgres: ClickHouse seems to offer an SQL layer for querying and a JDBC endpoint for conecting to it, which should make data access straightforward with standard DB browsers & from the cli. Not tried it yet, but hopefully is straightforward enough. An analysis of the two schemas V2 & V3 and what it would take to get data out of Clickhouse would be useful.

Generally we support the evolution of Langfuse in the direction mentioned. It makes sense to add Redis and ClickHouse for latency and improved reporting and petabyte analytics scale. We are very excited to see what you guys have cooked up, but do keep our requirements in mind. We are really pleased with the small and nimble footprint of V2 for now and haven't run into any scaling issues yet.

Keep up the great work. Excited to be part of your user community!

[marcklingen]

Thanks for taking the time to share this!

Development environment: We run Langfuse locally during development. Currently V2 only requires PostgreSQL and Langfuse which is ideal in terms of resource usage while developing. An analysis of the memory footprint and processing needs of V2 vs V3 would be very helpful. Rapid local development is a really important when prototyping AI apps.

Agree! Docker compose is super important for quick POCs and in environments where a production/staging deployments needs too much time due to restrictions within an organization. We will include guidance on this in the v3 documentation.

Relative unfamiliarity with ClickHouse vs Postgres: ClickHouse seems to offer an SQL layer for querying and a JDBC endpoint for conecting to it, which should make data access straightforward with standard DB browsers & from the cli. Not tried it yet, but hopefully is straightforward enough. An analysis of the two schemas V2 & V3 and what it would take to get data out of Clickhouse would be useful.

Clickhouse should be relatively easy to query from any db viewer/interface and the core db-schema stays relatively similar to the current one (which is close to open telemetry). I currently use tableplus to run tests on clickhouse and it works flawlessly.

Generally we support the evolution of Langfuse in the direction mentioned. It makes sense to add Redis and ClickHouse for latency and improved reporting and petabyte analytics scale. We are very excited to see what you guys have cooked up, but do keep our requirements in mind. We are really pleased with the small and nimble footprint of V2 for now and haven't run into any scaling issues yet.

Thank you! Both being easy to get started and the right solution when projects scale is important to us. Currently Langfuse is doing very well on the former and v3 will solve for the latter.

[secsilm]

Any update on the timeline for V3 release?

[marcklingen]

thanks for the ping on this, we currently plan with an initial release in late November.