Fix NetworkPolicy span calculation

[tnqn]

A NetworkPolicy's span is calculated in internalNetworkPolicyWorker, based on the span of the AppliedToGroups it refers to, while the span of AppliedToGroup is calculated in appliedToGroupWorker which runs in parallel with internalNetworkPolicyWorker. It could happen that the calcuated span is out of date if AppliedToGroups' span is updated after internalNetworkPolicyWorker calculates a NetworkPolicy's span, and the NetworkPolicy wouldn't be enqueued for another sync if it's not committed to the storage yet.

On the other hand, if we commit the NetworkPolicy to the storage before calculating the NetworkPolicy's span, it would have to use a stale span first and might need to update the NetworkPolicy twice and generate two update events in one sync.

To fix the issue without generating extra events, we introduce a separate subscription mechanism that allows subscribing to update of AppliedToGroup for NetworkPolicy. With the subscription, we can still calculate the NetworkPolicy's span first, then commit it to the storage. If any of the subscribed AppliedToGroups are updated, the NetworkPolicy will be notified and resynced.

Fixes #5553

[tnqn]

/test-all
/test-ipv6-all
/test-ipv6-only-all
/test-windows-all

[antoninbas]

@tnqn you marked this as as draft, so you no longer need an immediate review?

[tnqn]

@tnqn you marked this as as draft, so you no longer need an immediate review?

Yes, I'm considering another fix, I will mark it as ready after I finalize the solution.

[tnqn]

/test-all
/test-ipv6-all
/test-ipv6-only-all

[tnqn]

@antoninbas it's ready for review now.

[antoninbas]

LGTM

[jianjuns]

Question - can we add a flag to a NP object in storage to indicate the span is calculated or not (and store the object before span calculation)?

[jianjuns]

object -> objects

[tnqn]

done

[jianjuns]

Should we call it something like subscriptionService?

[tnqn]

I feel it's more common in Go to name objects with an -er suffix, but I understand subscriber is somewhat confusing. I have renamed it to notifier, it should make sense to subscribe to a notifier and send a message via a notifier, what do you think?

[Dyanngg]

Change LGTM. I'm assuming by the definition of the issue that this only happens for new internalNPs? For internalNPs update events this won't happen as they are already in store

[tnqn]

Question - can we add a flag to a NP object in storage to indicate the span is calculated or not (and store the object before span calculation)?

I considered adding a flag but think it is more complicated and risky than decoupling notification mechanism from the storage:

1. The issue doesn't just happen when processing a new NP, it could also happen when an existing NP is changed to use another AppliedToGroup. It commits an intermediate state of a NP to the storage, how to handle the object for the storage's readers, i.e. the watchers, should it be sent to a node based on the span or not? If yes, a NP may be sent to a Node it shouldn't do because the span may be outdated. If no, a NP may be deleted from a Node when it happens to reconnect the controller.
2. Currently the span calculation of NP doesn't need the internalNetworkPolicyMutex, which is used to ensure NetworkPolicy's indices and AppliedToGroups/AddressGroups are updated atomically. If we store the object first, then calculate the span, then store it again, we have to acquire the mutex twice, or include a CPU-intense process in the critical section.

[tnqn]

Change LGTM. I'm assuming by the definition of the issue that this only happens for new internalNPs? For internalNPs update events this won't happen as they are already in store

It could also happen if a NP is updated to use a new AppliedToGroup.

[jianjuns]

Your points make sense. @tnqn

[tnqn]

/test-all