Rest api to reset deprecation indexing cache

[pgomulka]

Deprecation indexing is using RateLimitingFilter to throttle duplicated
deprecations.
This commit adds REST API and transport actions to reset the LRU cache in this filter.

This will not affect the log4j filter used for throttling file appenders.
usage is very simple. Just call DELETE on _logging/deprecation_cache

curl --request DELETE \
  --url http://localhost:9200/_logging/deprecation_cache \
  --header 'Authorization: Basic ZWxhc3RpYzpwYXNzd29yZA=='

closes #78134

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[pgomulka]

I wasn't sure I extended the right classes in DeprecationCacheResetAction and TransportDeprecationCacheResetAction
My goal was to keep it as simple as possible.
Is there a generic "empty request" somewhere in our codebase already?
Also do we have a way to make sure that a response (even empty) is returned to a rest client only when operations on every node succeeded?

[rjernst]

I think this needs to be a TransportNodesAction. HandledTransportAction would only be handled by a single node, but TransportNodesAction fans out to all nodes (you'll need to create an appropriate single node request/response as well).

[rjernst]

Since resetting the cache is not deprecated, I don’t think it should be logged as such. You could log as debug?

[pgomulka]

we can log as debug. But this would mean that the log line will be in server log files only. It would not appear in deprecation data stream.
I agree that logging as warn deprecation might be confusing (I wanted to avoid this by introducing a new category).
but then again, maybe I am overdoing this and we don't need that information in deprecation data stream at all.

@cjcenizal do you know how the Upgrade Assitant UI is planning to use this API and then present fresh deprecations? Is the "marker" indicating a reset being performed needed at all?

[cjcenizal]

@pgomulka We expect the following workflow:

1. Users will enable deprecation logging
2. As applications send requests using deprecated API features, they'll see a log counter increase
3. They'll review the logs in an analysis UI, such as Discover, to identify the requests at fault
4. They'll fix and redeploy an application (or all of them)
5. They'll reset the counter, and wait for the count to go back up
6. If it goes up, they'll continue the process at step 3, and if it doesn't go up then they'll have confidence their fixes were effective

Does this make sense?

[cjcenizal]

BTW you can test this out yourself by spinning up a fresh 7.16 deployment on Cloud.

[cjcenizal]

@pgomulka Oh sorry I missed your question.

how the Upgrade Assitant UI is planning to use this API

I expect that when the user resets the counter (step 5), we'll send a request to this API to also reset the cache.

[pugnascotia]

The word "counter" makes me worry here - we use a rate limiting mechanism so that we don't keep logging the same message over and over, which means counting the occurrences of each message could be wildly inaccurate.

[cjcenizal]

@pugnascotia I think it all depends on how we present it to the user. For example, we could present it as "a count of unique logs". But I think you and anyone else should try out the UX and see how it feels.

[pgomulka]

so there is a counter. I suspect it counts how many documents are there in deprecation log data stream

As @pugnascotia mentioned reseting the indexing cache means that deprecations that already were indexed can be indexed again (duplicated).
In step 5:

They'll reset the counter, and wait for the count to go back up

I feel that by reseting the counter you mean removing the documents from deprecation log index? Data streams are append only, so I think it is not possible. The count on the UI would not go down, unless you remember what was the previous count.

With a marker we could query for the last reset cache. Take the timestamp. And then query for documents with timestamp greater then the fetched latest reset cache

[cjcenizal]

This is the deprecation log counter. It's just below the UI in your screenshot.

I feel that by reseting the counter you mean removing the documents from deprecation log index?

No. The way we've implemented this counter is we store a timestamp when the user clicks "Reset counter" and count all new docs that have been indexed since that timestamp.

[pgomulka]

No. The way we've implemented this counter is we store a timestamp when the user clicks "Reset counter" and count all new docs that have been indexed since that timestamp.

so that solves it all :) I will remove the redundant log indicating a reset
thanks @cjcenizal !

[cjcenizal]

Thanks @pgomulka!

[cjcenizal]

@pgomulka I'd like to open an issue in the Kibana repo and link to some docs. Do we have docs for this API? Or could you add a couple examples to the PR description and I'll just cross-link here?

[pgomulka]

@cjcenizal good question. I am not sure if this should be treated more like internal API or we should publicly document this. I don't mind either approach.
I will add a usage to the PR description.