Replace dgraph-io/badger cache storage with etcd-io/bbolt

[stefans-elastic]

Proposed commit message

Replacing dgraph-io/badger persistent storage for key-value cache with etcd-io/bbolt. Originally it was meant to just get rid of go.opencensus.io dependency which is introduced by badger (please see parent issue for more details). After it got evident that this won't erase go.opencensus.io dependency it was decided that this work still should be done since etcd-io/bbolt is already used elsewhere in the project and it isn't a good thing to have multiple storages for cache (again, please see parent issue for more details (in comments)).

Implementation should be fairly straight-forward but I would like to clarify one thing - since bolt doesn't support value expiration the expiration time (and TTL) are stored as metadata of the value. Upon value retrieval it is checked for expiration and if it is expired then nil is returned and value gets deleted from bolt DB.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Closes Drop dbadger-io dependency #42443

Use cases

Screenshots

Logs

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @stefans-elastic? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[botelastic]

This pull request doesn't have a Team:<team> label.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[AndersonQ]

[Suggestion]
Shouldn't the error be logged?
I see the original code also ignored the error, but I wonder if it's the best approach.
I know it's a cache and by definition it should not be a fatal error not being able to set a value in the cache, but if for some reason it keeps happening, it might impact the overall performance of whatever is using the cache. Besides if it fails often, there might be an issue that needs to be investigated.

[stefans-elastic]

i'm not sure, I wasn't intending to change the code (had to due to linter complains). Do you want me to add logging of the error?

[AndersonQ]

i'm not sure, I wasn't intending to change the code (had to due to linter complains).

I thought so.

However, this error breaks the contract, as it's clearly states the last access time of the element is updated. I only see one package using the cache. To avoid much changes, what do you think of the following. If you can confirm there is not "keyNotFoundError", you could log it as debug level or warning. regardless, would you mind updating the doc to say it might fail to update the last access time and still return a nil error?

[stefans-elastic]

so currently (at least how I've implemented this) if key doesn't exist the store will return nil error and nil value (as is indicated by this test - https://github.com/stefans-elastic/beats/blob/drop-dbadger-io/x-pack/libbeat/kv/bbolt/bbolt_test.go#L100-L107). So i wonder - do I need to return an error in case of key not being find in store?

[stefans-elastic]

(also added a debug log)

[stefans-elastic]

(and updated godoc comment (i hope that's what you meant by "would you mind updating the doc to say it might fail to update the last access time and still return a nil error?"))

[AndersonQ]

Sorry, but could you log the error instead of saying the key was not found?

[stefans-elastic]

sure, done
please check it out

[stefans-elastic]

/test

[stefans-elastic]

@elastic/beats-tech-leads / @VihasMakwana could you review this PR?

[cmacknz]

Looks like the persistent cache has a few uses related to cloudfoundry including the add_cloudfoundry_metadata process via the newClientCacheWrap function.

rg newClientCacheWrap
x-pack/libbeat/common/cloudfoundry/hub.go
82:func (h *Hub) ClientWithCache() (Client, error) {

x-pack/libbeat/common/cloudfoundry/cache_integration_test.go
41:             client, err := hub.ClientWithCache()
52:             client, err := hub.ClientWithCache()

x-pack/libbeat/processors/add_cloudfoundry_metadata/add_cloudfoundry_metadata.go
51:     client, err := hub.ClientWithCache()

1. Let's indicate in the changelog that this change only impacts Cloudfoundry related functionality. It looks like the impact would be we essentially clear the cache and start from scratch, which doesn't seem breaking to me.
2. Do we have any way to sanity check check any of this running on Cloudfoundry itself?