Delete log streams via API

[cyriltovena]

Is your feature request related to a problem? Please describe.

When the users will want to tweak Promtail scrape configuration (relabeling, pipeline, discovery) to work with their own deployments, they'll probably generate some unused or unwanted Loki streams.

It would be nice if we could search/filter through streams and ordered them by last chunk received, throughput and total size. Then be able to deleted those we don't want anymore.

This could also help users to create some sort of cron job to apply retention. @sandlis

We could use prometheus to scrape promtail and who creates those metrics. Loki could then use this to query for use within the API.

Describe the solution you'd like

1 - List all streams ordered by last data seen and throughout over the last 5minutes, we should be able provide label matcher {app="mongod" cluster="us-east1"} to narrow down the search, this should be a query limited by 100 results by default.
2 - Delete a list of stream via their labelset and may be from a specific time.

I feel like this could be useful to have in Grafana some sort of stream management UI @davkal ?

[tomwilkie]

Lets try and match the Prometheus API here: https://prometheus.io/docs/prometheus/latest/querying/api/#delete-series

[cyriltovena]

We should start with GET, https://prometheus.io/docs/prometheus/latest/querying/api/#finding-series-by-label-matchers

/cc @davkal

[slim-bean]

related to #113

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[blocke]

This ticket was closed but #113 only covers the GET portion. I'm hoping the DELETE portion gets included too as I just spent a couple hours fighting an incorrect timestamp parsing pipeline. Should a reminder for DELETE be added to #113 or can this ticket be saved from the stale bot?

Thanks.

[bastjan]

I've hacked together a rough prototype of the delete API after seeing cortexproject/cortex#2103 and blowing up my index with an erroneously parsed timestamp.

Is anyone working on the delete API on the Loki team? I could finish up my prototype over the next 2-3 weeks if needed.

[sandeepsukhani]

Not yet. I appreciate your efforts! We would like to get it working for Cortex and then reuse most of the code in Loki.
Also out of curiosity, was the issue that you were facing introduced by code from that Cortex PR? I would like to fix it if it is.

[bastjan]

My issue was unrelated. I parsed the "time since boot" counter from the linux kernel log as label by accident.

The cortex PR allowed me to use the new storage.DeleteSeriesIDs. Which made it quite easy to implement the remainder :)

[daveram]

Is there any way to manually go in and delete all instances of a label? We're running into the scenario where we started collecting the wrong logs under the wrong label name, promtail reconfig was nice and easy but now we have the old labels hanging out with invalid logs. Would like to clean up, and since the API doesn't support this like Prometheus yet, hoping there's a more manual way.

We're using two different setups, one with filesystem and boltdb for chunks/index, the other with s3 and cassandra. Both have the same scenario happening

[ayZagen]

We also need this also. It would be great to have something like in prometheus. link

[tibin-mfl]

Do we have any update on this, Is delete api available on Loki ?
What can we do if we want to put some on demand logs to loki for some some issue debugging and later delete that log stream.

For Example :
We push DB logs to loki for analysing some production issue after issue debugging I don't want the logs so I need to delete that logs to reduce unnecessary storage.

[cyriltovena]

It's actively being worked on by @sandeepsukhani

Feel free to give him support, it's OpenSource !

[Tschis]

@cyriltovena Hi! Thanks for the info! 💪

But while it is not done, do you know of a workaround that can be used?
Due to some circumstances we created labels on a field that went above 100,000 entries. Now we are stuck with an error message cardinality limit exceeded for logs{log_code}; 123474 entries, more than limit of 100000.

Since we hold our logs for many days, is there currently any way I can manually delete these label values?

[diemus]

I have to delete all data to solve the "cardinality limit exceeded for logs ....more than limit of 100000" problem. and I have no idea which label is high cardinality. is there a tool to see this? I just found the index size grow fast these days, but can not find the cause

[cyriltovena]

We recently released a way to see that with logcli. (aka --analyze-labels)

see:

• https://grafana.com/blog/2020/08/13/loki-1.6.0-released-metric-query-performance-up-to-10x-faster-push-logs-from-any-client-to-promtail-query-language-and-logcli-enhancements-and-more/
• logcli: adds --analyize-labels to logcli series command and changes how labels are provided to the command #2497

It does exactly what you want ✌️

[AMoghrabi]

@cyriltovena Thanks for the info. Are you aware of a way to actually delete the labels? I couldn't find information on that in the PR or blog. Thanks!

[tmikaeld]

Same problem here, need to clear the logs due to GDPR request.

Seems silly to wipe all logs, when there are only a few labels we need to clear.

[sandstrom]

On the topic of log deletion, I think it would be even more useful if we could configure retention. E.g. keep everything for X days/months/years.

Or even better, since logs are usually stored on S3 (or similar storage), and loki is append-only (mostly?), maybe we could simply configure our S3 bucket to delete all files older than X days.

That would also work nicely with protecting logs, since we could use e.g. S3 object lock to prevent deletion of logs until X days have passed.

If you do add support for deleting streams via the API, don't enable it by default. For security, it's great that log shippers (anyone sending data to loki) can't delete it). For security, I think most setups will want to handle retention in loki or on the underlying storage (e.g. S3 bucket deletion rules).

[justinmchase]

I need this also, its crazy how easy it is to add a bad label and then have it totally break your entire log database to the point where you need to wipe out all of your log data to repair it.

This should be a pretty high priority to fix imo.

[michaelkebe]

Version 2.3.0 has this feature now.

https://grafana.com/docs/loki/latest/operations/storage/logs-deletion/

[09jvilla]

Closing this issue now that the log deletion api has been released.

[pgassmann]

The documentation for Log deletion API has been moved here:
https://grafana.com/docs/grafana-cloud/data-configuration/logs/delete-log-lines/
https://grafana.com/docs/loki/latest/api/#request-log-deletion