-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Asynchronous Multi Site Ingester Replication #4600
Comments
I don't know whether Loki should support so much high-level reliability but I'll comment my just idea as a user. If I were you, I should use nginx request mirroing like the following image.
|
First of all thank you for the help!
We are definitely considering a solution in that direction. If we were to execute the solution you offered how would we replicate the data that was written to Loki when one site was down to that site the moment he is back on?
With that said I do understand that our use case is probably unique, and maybe Loki moving forward better prioritize adding support and improving in different things, however I think a lot of users would appreciate if this subject would be addressed in the documentation. Again, thanks for the help we don't take it for granted. |
I'm just a one of the Loki's users but your topic is interesting to me so let me discuss. You mean that a site failure causes "all of ingesters down in the site as same as other servers" so you can't see logs on them while the outage, right? |
The end goal is to keep Loki up 100% of the time. Loki is hosted in a single datacenter and receives logs originating from the datacenter he is hosted in and others. If the site hosting Loki fails, Loki is unavailable across our entire ecosystem. The solution is to deploy a backup Loki in a different site, when the main Loki site fails we will move to using the backup Loki. Loki stores data in ingesters and in long term storage. Making the data in the long term storage available in 2 sites is easy, we use Cassandra and we can set up a multisite Cassandra cluster and replicate data to the other site. However, the data in the ingesters won't be available in the backup site because it gets replicated to the other site only after it is flushed to the backend storage. Ingester data missing during a site failure means in our eyes that Loki is fairly limited in his multi-site resilience capabilities and it is the problem we are trying to solve or at least mitigate. A side note... I mentioned WAL to emphasize that we understand data won't be lost forever, just unavailable during the site downtime:
Again, thanks for the help. |
Thank you for clarification. My idea is the following image.
|
Thanks for the detailed response. Your idea is definitely something we are heavily considering. However in this architecture, since there is no replication mechanism between the two Loki clusters, all the logs written to the backup site when the master site is down won't be available in the master site when he comes back up. This creates a "hole" in the data available in one site so both clusters potentially won't really be fully synchronized. This fact is significant to us since our master site has significantly better hardware and we want to be using it in every second he is up. This means we want to provide our Loki clients with a single Loki data source pointing to the master site and when the master site failed the Loki DNS address will change and point to the backup site. In fact we want users to have a single Grafana data source for reasons other then the one I just mentioned, we actually prioritized a single Grafana data source as the main attribute we want to maintain in the multi site architecture. I guess if we were to use nginx mirroring, since both Loki data sources will be identical it is not the same use-case like we had with Prometheus but we actually prioritize having a single Loki data source higher than maintaining ingester data during a site failure. |
I see.
This architecture don't create "hole" because Cassandra replication is enabled and data source is only one, I think. If you need more complex routing in LB side for failover, you could use envoy instead of Nginx |
Interesting, this solution definitely answers our resilience and convenient needs. We had a similar one come up in our discussions. This solution bares a significant drawback, since the ingesters are separate they have no way of knowing if an ingester in a different site has written down to Cassandra the chunks it holds in memory. This means that each line of log will be written to Cassandra twice. Aside from the additional storing costs we fear our querying performance would take a significant hit since double the data would be pulled in each query. |
Right. |
We are leaning to accept the fact that ingester data will be unavailable at site downtime, we think it is probably the best option out of the bunch. This feature request was open in hope of an addressing of this topic in future releases. |
Hey yall, great discussion going on in here. I suspect we'll introduce zone-aware replication at some point in the future, meaning that we could host ingesters across multiple P.S. I'd highly advise moving away from Cassandra and towards a single-store configuration using the boltdb-shipper unless you have very intentional reasons for choosing Cassandra. It's where our current and future development efforts are focused and is a fast, cost-effective, and easier to operate in my opinion. |
Thank you so much!!! It is definitely reassuring that a feature of this kind is in the works. Should I keep the issue open or close it knowing something is coming in the future?
We are operating on a completely private network, this means the Object Storage we use is ours and it still heavily uses HDD disks. The solution was Cassnadra for us, working from a block-storage based database made the speed data was pulled from the backend storage around 15 times faster. |
I'm fine leaving this issue open, especially so folk can 👍 it :). There's already a lot of code for this in Cortex and I suspect it'd just be a matter of prioritization on our side. |
maybe we can already use |
Hi! This issue has been automatically marked as stale because it has not had any We use a stalebot among other tools to help manage the state of issues in this project. Stalebots are also emotionless and cruel and can close issues which are still very relevant. If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry. We regularly sort for closed issues which have a We may also:
We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task, |
We are trying to deploy on-premise Loki architecture in 2 data centers and we want our deployment to be resilient to site-level malfunctions.
We configured our Promtail agents across all data centers to send logs to a single centralized Loki deployment hosted in a single datacenter.
We want to deploy a backup Loki on a different site that we will use instead of the main one at the time of an outage. We use Cassandra for both chunk and index store and we can easily replicate Cassandra across 2 data centers so that data it is available in the backup site when needed, however data in the "main" Loki stored in the ingesters won't be available to the backup site until it is flushed to the backend storage.
Now we understand that the data won't truly be lost due to the WAL, but it can lead to the lost of important log lines, originating from the site and going into the site, that might be extremely useful when figuring out the root cause of the outage and might hold back application hosted on the failed datacenter from getting a clear picture of the state of their application. Potentially a few hours of logs might be unavailable to us at the time of an outage despite the effort we put into setting up a backup deployment.
We haven't fully thought about the technical details of the solution, but we think that providing a set of ingesters a way to "fetch" for themselves chunks stored on a remote set of ingesters can increase Loki's resilience. We think that a mechanism to prevent ingesters from flushing the same set of data to the backend storage when the sites are up and especially when the master site is down is key to this solution and would be challenging to execute.
An alternative solution would be to provide a set of ingesters with some kind of a site identifier. In that case queriers would turn to multiple sites to fetch logs, depending on the query, and a single site outage would lead to the lose of logs originating only from that site. We realize this solution introduces some complications, a set of ingesters and distributors would have to be deployed in each site and fetching logs might be performed in inconsistent latencies.
Any help or feedback would be greatly appreciated.
The text was updated successfully, but these errors were encountered: