Add an option to delete logs from the Logs page

[anna-geller]

Feature description

Currently, you can delete logs from the Executions page.

We should add the same functionality from the Logs page.

[MilosPaunovic]

@anna-geller I think I would need BE change for this to be feasible, as the endpoint which is doing the log removal from Executions page needs executionId and taskRunId parameters, which are not present in the response of the endpoint which is currently listing logs.

[aku]

As I've mentioned in this issue #3961 there might be a problem when flow\executions are deleted but their logs remain. Please make sure it is possible to delete logs even when corresponding flow\task\execution has been deleted.

[anna-geller]

don't delete executions @aku :) it's better to keep them for audit trail/history. we recommend to only delete those that are far away in the past to avoid such issues

[aku]

don't delete executions @aku :) it's better to keep them for audit trail/history. we recommend to only delete those that are far away in the past to avoid such issues

don't delete executions @aku :) it's better to keep them for audit trail/history. we recommend to only delete those that are far away in the past to avoid such issues

Unfortunately, storage space is not unlimited. In my case I had thousands or executions per day. It makes not sense to keep them for a long time. I've planned to purge them every few days to keep my S3 storage happy

[anna-geller]

Technically speaking, Executions are stored in the database, so they shouldn't affect S3 except for the data (e.g. outputs) processed in executions. Would you like to talk to @Ben8t about your use case? I think we can find a way to manage it efficiently depending on the design of the flows and triggers

[aku]

Technically speaking, Executions are stored in the database, so they shouldn't affect S3 except for the data (e.g. outputs) processed in executions. Would you like to talk to @Ben8t about your use case? I think we can find a way to manage it efficiently depending on the design of the flows and triggers

I had sub-optimal solution using EachItem task to process a lot of data coming from kafka trigger. The flow generated thousands of child flows that stored some data (execution results?) in S3. I lost 300Gb of space in just a few hours.

Now I use different strategy - processing everything in a single flow. It has helped to reduce a number of executions so S3 is not an issue now. However, I still have thousands of logs entries that I do not need. Especially, after I purged executions via built-in Purge task.

[anna-geller]

Interesting. In that case, perhaps it is worth iterating directly with the database logs table? you can add a Postgres SQL query task to remove the logs you don't need

[aku]

Interesting. In that case, perhaps it is worth iterating directly with the database logs table? you can add a Postgres SQL query task to remove the logs you don't need

I do not feel comfortable to mess directly with Kestra's DB. If it is the only option to purge leftover logs from deleted flows\executions could you please provide me with a query? I'll give it a try. Ideally, I'd like to delete all flows\tasks logs older than N days. Ability to specify log level will be a bonus.

Honestly, I was hoping to get some built-in functionality to delete logs based on some criteria (level, age, flowid etc.)
I've tried to use logs suppression like you recommended but in some cases it is useful to keep logs for some period like 24 hours.

[loicmathieu]

Logs didn't have a unique identifier, to be able to delete logs safely from a filter on the logs page, we would need to add an identifier (for ex autoincrement id) or duplicate the delete logic with the search logic (which is kind of strange deleting with a fulltext and a page but why not).

[loicmathieu]

@MilosPaunovic

Finally we decided to allow deleting logs from the Flow -> Logs tab and from the Triggers -> Log tab.
I created a new endpoint for that, you can use the branch #4438
It adds an endpoint DELETE /api/v1/logs/namespace/flowId?triggerId=triggerId, trigerId being optional.

[MilosPaunovic]

Closed with #4438.