Cache rendered centrals

[SimonBaeumer]

Description

This PR adds a cache for the rendered Centrals in the GitOps service.

Checklist (Definition of Done)

• Unit and integration tests added
• Added test description under Test manual
• Documentation added if necessary (i.e. changes to dev setup, test execution, ...)
• CI and all relevant tests are passing
• Add the ticket number to the PR title if available, i.e. ROX-12345: ...
• Discussed security and business related topics privately. Will move any security and business related topics that arise to private communication channel.
• Add secret to app-interface Vault or Secrets Manager if necessary
• RDS changes were e2e tested manually
• Check AWS limits are reasonable for changes provisioning new resources

Test manual

• Unit tests
• Manual
  • Deploy Fleet-Manager
  • Delete fleetshard-sync deployment (to prevent scheduling Centrals)
  • Execute to create Centrals: for i in {1..1500}; do ./scripts/create-central.sh "test-central-${i}"; done
  • Export your OCM token (export OCM_TOKEN=$(ocm token)
  • Run ./scripts/fmcurl rhacs/v1/agent-clusters/1234567890abcdef1234567890abcdef/centrals
    • Expect long response time
  • Run ./scripts/fmcurl rhacs/v1/agent-clusters/1234567890abcdef1234567890abcdef/centrals
    • Run again, faster response time

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: SimonBaeumer

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [SimonBaeumer]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ludydoo]

Example of keyed mutex for locking cache keys

Suggested change

	var km = keyedMutex{locks: map[string]*sync.Mutex{}}
	type keyedMutex struct {
	locks map[string]*sync.Mutex
	mLock sync.Mutex
	}
	func (k keyedMutex) getLockBy(key string) *sync.Mutex {
	k.mLock.Lock()
	defer k.mLock.Unlock()
	if ret, ok := k.locks[key]; ok {
	return ret
	}
	lock := &sync.Mutex{}
	k.locks[key] = lock
	return lock
	}
	func (k keyedMutex) Lock(key string) {
	k.getLockBy(key).Lock()
	}
	func (k keyedMutex) Unlock(key string) {
	k.getLockBy(key).Unlock()
	}

[ludydoo]

I think you need to lock the cache key

Suggested change

	result, err := readFromCache(params, cfg)
	if err != nil {
	return "", err
	}
	if result != "" {
	return result, nil
	}
	centralCR, err := renderCentral(params, cfg)
	if err != nil {
	return "", err
	}
	centralYaml, err := yaml.Marshal(centralCR)
	if err != nil {
	return "", errors.Wrap(err, "failed to marshal Central CR")
	}
	centralYAMLString := string(centralYaml)
	err = addCache(params, cfg, centralYAMLString)
	if err != nil {
	return "", err
	}
	return centralYAMLString, nil
	cacheKey, err := getCacheKey(params, cfg)
	if err != nil {
	return "", err
	}
	km.Lock(cacheKey)
	defer km.Unlock(cacheKey)
	if result := readFromCache(cacheKey); result != "" {
	return result, nil
	}
	centralCR, err := renderCentral(params, cfg)
	if err != nil {
	return "", err
	}
	centralYaml, err := yaml.Marshal(centralCR)
	if err != nil {
	return "", errors.Wrap(err, "failed to marshal Central CR")
	}
	centralYAMLString := string(centralYaml)
	addCache(cacheKey, centralYAMLString)
	return centralYAMLString, nil

[ludydoo]

Here you need to RLock the cache + passing the cache key directly

Suggested change

	func readFromCache(params CentralParams, config Config) (string, error) {
	cacheKey, err := getCacheKey(params, config)
	if err != nil {
	return "", fmt.Errorf("Could not get cache key: %w", err)
	}
	if val, ok := centralCRYAMLCache[cacheKey]; ok {
	func readFromCache(cacheKey string) (string) {
	centralCRYAMLCache.RLock()
	defer centralCRYamlCache.RUnlock()
	if val, ok := centralCRYAMLCache[cacheKey]; ok {

[ludydoo]

passing the cache key directly

Suggested change

	func addCache(params CentralParams, config Config, centralYAML string) error {
	centralCacheMutex.Lock()
	defer centralCacheMutex.Unlock()
	key, err := getCacheKey(params, config)
	if err != nil {
	return fmt.Errorf("could not add to cache: %w", err)
	}
	centralCRYAMLCache[key] = centralYAML
	return nil
	func addCache(cacheKey string, centralYAML string) {
	centralCacheMutex.Lock()
	defer centralCacheMutex.Unlock()
	centralCRYAMLCache[key] = centralYAML