Do Konnect and Gateway reconciliation in different goroutines

[lahabana]

Is there an existing issue for this?

• I have searched the existing issues

Does this enhancement require public documentation?

• I have added an Acceptance Criteria item for adding and/or adjusting public documentation (if applicable)

Problem Statement

When we update the config:

1. we push config to all gateways in parallel, waiting for all to return
2. then we push to Konnect.

See:

kubernetes-ingress-controller/internal/dataplane/kong_client.go

Lines 473 to 474 in 95af584

	shas, gatewaysSyncErr := c.sendOutToGatewayClients(ctx, parsingResult.KongState, c.kongConfig, isFallback)
	konnectSyncErr := c.maybeSendOutToKonnectClient(ctx, parsingResult.KongState, c.kongConfig, isFallback)

This happens in a loop that IIUC is run every 3s:

kubernetes-ingress-controller/internal/dataplane/synchronizer.go

Lines 166 to 190 in 2cbf631

	for {
	select {
	case <-ctx.Done():
	p.logger.Info("Context done: shutting down the proxy update server")
	if err := ctx.Err(); err != nil && !errors.Is(err, context.Canceled) {
	p.logger.Error(err, "Context completed with error")
	}
	p.syncTicker.Stop()
	p.lock.Lock()
	defer p.lock.Unlock()
	p.isServerRunning = false
	p.configApplied = false
	return
	case <-p.syncTicker.C:
	if err := p.dataplaneClient.Update(ctx); err != nil {
	p.logger.Error(err, "Could not update kong admin")
	continue
	}
	initialConfig.Do(p.markConfigApplied)
	}
	}
	}

So if Konnect sync takes too long, this may negatively impact dataplanes as they may not receive up to date configs.

Proposed Solution

• Run Konnect sync in a separate goroutine to avoid slowing down reconciliation if Konnect sync is taking too long
• Ensure that Konnect sync can skip ticks if things take too long

Additional information

No response

Acceptance Criteria

• Konnect sync that takes over 3s won't block dataplane reconciliation

[lahabana]

Replaced by #6325