Fix missing call of removing groupDb cache when deleting OVS group #4592

ceclinux · 2023-02-01T10:16:06Z

Fix missing call of removing groupDb cache when deleting OVS group

The old group will be reused unexpectedly if groupDb cache managed
by OFBridge is not cleared for that group, which causes a new group
claims to have a different group type acquiring the old group type.

Fixes #4575

ceclinux · 2023-02-01T10:20:34Z

/test-multicast-e2e

codecov · 2023-02-01T10:44:58Z

Codecov Report

Merging #4592 (a9a1292) into main (482fc93) will increase coverage by 0.12%.
The diff coverage is 71.42%.

@@            Coverage Diff             @@
##             main    #4592      +/-   ##
==========================================
+ Coverage   69.75%   69.88%   +0.12%     
==========================================
  Files         400      403       +3     
  Lines       59450    60115     +665     
==========================================
+ Hits        41472    42010     +538     
- Misses      15178    15283     +105     
- Partials     2800     2822      +22

Flag	Coverage Δ		*Carryforward flag
e2e-tests	`38.34% <ø> (+<0.01%)`	⬆️	Carriedforward from 482fc93
integration-tests	`34.40% <7.14%> (-0.15%)`	⬇️
kind-e2e-tests	`47.64% <7.14%> (+1.35%)`	⬆️
unit-tests	`59.96% <71.42%> (+0.21%)`	⬆️

*This pull request uses carry forward flags. Click here to find out more.

Impacted Files	Coverage Δ
pkg/agent/openflow/client.go	`88.74% <50.00%> (-0.05%)`	⬇️
pkg/ovs/openflow/ofctrl_bridge.go	`80.07% <100.00%> (+2.52%)`	⬆️
pkg/agent/proxy/topology.go	`72.72% <0.00%> (-9.10%)`	⬇️
pkg/agent/multicluster/mc_route_controller.go	`47.00% <0.00%> (-8.55%)`	⬇️
...nt/apiserver/handlers/serviceexternalip/handler.go	`29.62% <0.00%> (-7.41%)`	⬇️
...agent/flowexporter/connections/deny_connections.go	`84.94% <0.00%> (-5.38%)`	⬇️
pkg/agent/route/route_linux.go	`66.19% <0.00%> (-5.30%)`	⬇️
pkg/agent/controller/egress/egress_controller.go	`82.37% <0.00%> (-2.86%)`	⬇️
...rs/multicluster/member/serviceexport_controller.go	`76.33% <0.00%> (-2.80%)`	⬇️
pkg/ovs/ovsctl/ofctl.go	`52.05% <0.00%> (-2.06%)`	⬇️
... and 32 more

ceclinux · 2023-02-01T14:47:48Z

/test-multicast-e2e

wenyingd · 2023-02-02T08:23:58Z

pkg/ovs/openflow/ofctrl_bridge.go

@@ -229,10 +229,6 @@ func (b *OFBridge) DeleteGroup(id GroupIDType) error {
 	if ofctrlGroup == nil {
 		return nil
 	}
-	g := &ofGroup{bridge: b, ofctrl: ofctrlGroup}


Why remove line 232 to 235, but not directly call OFBridge.DeleteGroup(gID) in the caller?

This change might cause group still exists in OVS when calling OFBridge.DeleteGroup but not sending another separate OF message to remove group.

Updated.

OFBridge.DeleteGroup is not called here directly because DeleteOFEntries will callmetrics.OVSFlowOpsErrorCount.WithLabelValues metrics.OVSFlowOpsCount.WithLabelValues for prometheus integration while DeleteGroup don't(and it shouldn't). I have changed DeleteGroup to allow users to decide whether to do flow deletion and call DeleteOFEntries and DeleteGroup both here.

ceclinux · 2023-02-02T13:35:57Z

/test-multicast-e2e

wenyingd

Code change looks good to me, but it looks the UT coverage does not satisfy the requirement.

ceclinux · 2023-02-16T08:41:50Z

Updated. UT coverage requirement satisfied @wenyingd

wenyingd · 2023-02-23T09:08:55Z

pkg/agent/openflow/client.go

-			return fmt.Errorf("error when deleting Service Endpoints Group %d: %w", groupID, err)
+			return fmt.Errorf("error when deleting Openflow entries for Service Endpoints Group %d: %w", groupID, err)
+		}
+		if err := c.featureService.bridge.DeleteGroup(groupID, false); err != nil {


Suggested change

if err := c.featureService.bridge.DeleteGroup(groupID, false); err != nil {

if err := c.bridge.DeleteGroup(groupID, false); err != nil {

updated. Thanks

wenyingd · 2023-02-23T09:09:09Z

pkg/agent/openflow/client.go

-			return fmt.Errorf("error when deleting Multicast receiver Group %d: %w", groupID, err)
+			return fmt.Errorf("error when deleting Openflow entries for Multicast receiver Group %d: %w", groupID, err)
+		}
+		if err := c.featureMulticast.bridge.DeleteGroup(groupID, false); err != nil {


Suggested change

if err := c.featureMulticast.bridge.DeleteGroup(groupID, false); err != nil {

if err := c.bridge.DeleteGroup(groupID, false); err != nil {

updated. Thanks

wenyingd

LGTM

wenyingd · 2023-02-28T11:04:47Z

/test-all
/test-windows-all

pkg/agent/openflow/client_test.go

hongliangl · 2023-02-28T11:51:34Z

pkg/ovs/openflow/ofctrl_bridge.go

@@ -224,14 +224,16 @@ func (b *OFBridge) createGroupWithType(id GroupIDType, groupType ofctrl.GroupTyp
 	return g
 }

-func (b *OFBridge) DeleteGroup(id GroupIDType) error {
+func (b *OFBridge) DeleteGroup(id GroupIDType, deleteFlows bool) error {


Is it used anywhere that deleteFlows is true? Another question: why name it deleteFlows? I didn't see that this is related to any flows.

It is related to flows. In deleteFlows, g.Delete() is called, which calls AddOFEntriesInBundle.

deleteFlows = true is not used in other places. Keeping it as a parameter because I think it is reasonable that flow transactions should be kept at this level. Users can disable it explicitly, not implicitly if the flow transaction is not wanted.

How about 'deleteGroupOnOVS'? This parameter is used to decide whether to delete the group on OVS.

hongliangl · 2023-02-28T11:52:49Z

pkg/ovs/openflow/ofctrl_bridge.go

-	g := &ofGroup{bridge: b, ofctrl: ofctrlGroup}
-	if err := g.Delete(); err != nil {
-		return fmt.Errorf("failed to delete the group: %w", err)
+	if deleteFlows {


Could you add more comments or PR description? I didn't get that why the issue can be fixed by bypassing this part.

hongliangl · 2023-03-01T03:43:23Z

I synced with @wenyingd offline about the issue #4575. The root cause is that:

Create a group for AntreaProxy with CreateGroup, then a group object is return, and the group object will be also cached groupDB in ofnet. Note that, the group type will be select.
Sync the group to OVS with bundle.
Delete the group in OVS with bundle, and the method DeleteGroup is not called, as a result, cached group object in groupDB will not be deleted.
If the group ID is reallocated to Multicast, then the cached group object will be reused, and the group type is still select, not all.

I'm ok with this fix, but could you add more comments about adding parameter deleteFlows to DeleteGroup?

ceclinux · 2023-03-01T09:00:04Z

Thanks for the review. Your understanding is correct. The root cause for the contrived code fix is coming from Prometheus metrics count. Please check #4592 (comment) and the updated comment for detail to see if it is reasonable to you.

pkg/ovs/openflow/ofctrl_bridge.go

pkg/ovs/openflow/interfaces.go

tnqn

The seems a serious issue when Multicast is enabled. Please backport it after merged.

tnqn · 2023-03-08T04:03:53Z

pkg/ovs/openflow/ofctrl_bridge.go

+// DeleteGroup deletes a specified group in groupDb. If deleteGroupOnOVS sets to true, the group deletion transaction
+// will be synced to OVS. Note that to record OVS operation in Prometheus, openFlowClient.DeleteOFEntries([]binding.OFEntry{groupCache})
+// should be called in advance, and deleteGroupOnOVS is unnecessary.
+func (b *OFBridge) DeleteGroup(id GroupIDType, deleteGroupOnOVS bool) error {


I think the methods is a little confusing as CreateGroup only construct group in memory while DeleteGroup could delete the group in OVS. Methods should be symmetrical so clients can use similar methods to manage the same resource.
Since there isn't even an actual case needing deleteGroupOnOVS to be true, I think we should just make CreateGroup and DeleteGroup symmetrical.

pkg/ovs/openflow/ofctrl_bridge.go

tnqn · 2023-03-09T14:36:31Z

pkg/ovs/openflow/ofctrl_bridge.go

@@ -224,15 +224,13 @@ func (b *OFBridge) createGroupWithType(id GroupIDType, groupType ofctrl.GroupTyp
 	return g
 }

+// DeleteGroup deletes a specified group in groupDb. Note that to record OVS operations in Prometheus, openFlowClient.DeleteOFEntries([]binding.OFEntry{groupCache})


update comment

tnqn · 2023-03-09T14:37:31Z

pkg/ovs/openflow/ofctrl_bridge_test.go

+			name:           "delete existed group without flow",
+			existedGroupID: 20,


Suggested change

name: "delete existed group without flow",

existedGroupID: 20,

name: "delete existing group without flow",

existingGroupID: 20,

tnqn · 2023-03-09T14:37:41Z

pkg/ovs/openflow/ofctrl_bridge_test.go

+			err:            nil,
+		},
+		{
+			name:           "delete non-existed group",


tnqn · 2023-03-09T14:38:41Z

pkg/ovs/openflow/ofctrl_bridge_test.go

+	} {
+		t.Run(m.name, func(t *testing.T) {
+			b.ofSwitch = newFakeOFSwitch(b)
+			b.ofSwitch.NewGroup(uint32(m.existedGroupID), ofctrl.GroupAll)


why it doesn't use b.CreateGroup to be symmetrical

The old group will be reused unexpectedly if groupDb cache managed by OFBridge is not cleared for that group, which causes a new group claims to have a different group type acquiring the old group type. Fixes antrea-io#4575 Signed-off-by: ceclinux <src655@gmail.com>

tnqn

LGTM

tnqn · 2023-03-10T12:30:16Z

/test-all

tnqn · 2023-03-15T12:16:14Z

@ceclinux could you backport the PR to applicable releases up to 1.7.

…ntrea-io#4592) The old group will be reused unexpectedly if groupDb cache managed by OFBridge is not cleared for that group, which causes a new group claims to have a different group type acquiring the old group type. Fixes antrea-io#4575 Signed-off-by: ceclinux <src655@gmail.com>

ceclinux force-pushed the fix-delete-multicast-group branch from effaf08 to fb1e07e Compare February 1, 2023 10:19

ceclinux force-pushed the fix-delete-multicast-group branch from fb1e07e to 15fd036 Compare February 1, 2023 14:47

ceclinux changed the title ~~fix missing call of removing groupDb cache for OVS group when deletin…~~ Fix missing call of removing groupDb cache when deleting OVS group Feb 2, 2023

ceclinux requested a review from wenyingd February 2, 2023 08:02

wenyingd reviewed Feb 2, 2023

View reviewed changes

ceclinux force-pushed the fix-delete-multicast-group branch from 15fd036 to f8faceb Compare February 2, 2023 13:27

wenyingd reviewed Feb 6, 2023

View reviewed changes

ceclinux force-pushed the fix-delete-multicast-group branch 7 times, most recently from 778bfc4 to bb275da Compare February 15, 2023 06:38

wenyingd reviewed Feb 23, 2023

View reviewed changes

ceclinux force-pushed the fix-delete-multicast-group branch from bb275da to a9a1292 Compare February 26, 2023 09:52

wenyingd previously approved these changes Feb 28, 2023

View reviewed changes

wenyingd requested a review from hongliangl February 28, 2023 11:04

hongliangl reviewed Feb 28, 2023

View reviewed changes

ceclinux dismissed wenyingd’s stale review via ea6d558 March 1, 2023 08:59

ceclinux force-pushed the fix-delete-multicast-group branch from a9a1292 to ea6d558 Compare March 1, 2023 08:59

hongliangl reviewed Mar 1, 2023

View reviewed changes

pkg/ovs/openflow/ofctrl_bridge.go Outdated Show resolved Hide resolved

hongliangl reviewed Mar 1, 2023

View reviewed changes

pkg/ovs/openflow/interfaces.go Outdated Show resolved Hide resolved

ceclinux force-pushed the fix-delete-multicast-group branch from ea6d558 to 3d4e1d5 Compare March 2, 2023 07:04

wenyingd added this to the Antrea v1.11 release milestone Mar 2, 2023

hongliangl previously approved these changes Mar 7, 2023

View reviewed changes

tnqn reviewed Mar 8, 2023

View reviewed changes

jianjuns reviewed Mar 8, 2023

View reviewed changes

pkg/ovs/openflow/ofctrl_bridge.go Outdated Show resolved Hide resolved

pkg/ovs/openflow/ofctrl_bridge.go Outdated Show resolved Hide resolved

ceclinux dismissed hongliangl’s stale review via 5c3a140 March 9, 2023 14:34

ceclinux force-pushed the fix-delete-multicast-group branch from 3d4e1d5 to 5c3a140 Compare March 9, 2023 14:34

tnqn reviewed Mar 9, 2023

View reviewed changes

ceclinux force-pushed the fix-delete-multicast-group branch 2 times, most recently from 912b9b5 to e514b98 Compare March 9, 2023 16:29

ceclinux force-pushed the fix-delete-multicast-group branch from e514b98 to 4a094af Compare March 9, 2023 16:31

tnqn approved these changes Mar 10, 2023

View reviewed changes

tnqn merged commit 9b6319d into antrea-io:main Mar 10, 2023

tnqn added action/backport Indicates a PR that requires backports. action/release-note Indicates a PR that should be included in release notes. labels Mar 10, 2023

This was referenced Mar 16, 2023

Automated cherry pick of #4592: Fix missing call of removing groupDb cache when deleting OVS #4706

Merged

Automated cherry pick of #4592: Fix missing call of removing groupDb cache when deleting OVS #4712

Merged

luolanzone mentioned this pull request Mar 24, 2023

Automated cherry pick of #4592: Fix missing call of removing groupDb cache when deleting OVS #4758

Closed

ceclinux mentioned this pull request Mar 28, 2023

Automated cherry pick of #4592: Fix missing call of removing groupDb cache when deleting OVS #4771

Closed

	if err := c.featureService.bridge.DeleteGroup(groupID, false); err != nil {
	if err := c.bridge.DeleteGroup(groupID, false); err != nil {

	if err := c.featureMulticast.bridge.DeleteGroup(groupID, false); err != nil {
	if err := c.bridge.DeleteGroup(groupID, false); err != nil {

		name: "delete existed group without flow",
		existedGroupID: 20,

Fix missing call of removing groupDb cache when deleting OVS group #4592

Fix missing call of removing groupDb cache when deleting OVS group #4592

Conversation

ceclinux commented Feb 1, 2023 • edited Loading

ceclinux commented Feb 1, 2023

codecov bot commented Feb 1, 2023 • edited Loading

Codecov Report

ceclinux commented Feb 1, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ceclinux commented Feb 2, 2023

wenyingd left a comment

Choose a reason for hiding this comment

ceclinux commented Feb 16, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wenyingd left a comment

Choose a reason for hiding this comment

wenyingd commented Feb 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hongliangl commented Mar 1, 2023

ceclinux commented Mar 1, 2023

tnqn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tnqn left a comment

Choose a reason for hiding this comment

tnqn commented Mar 10, 2023

tnqn commented Mar 15, 2023

ceclinux commented Feb 1, 2023 •

edited

Loading

codecov bot commented Feb 1, 2023 •

edited

Loading

ceclinux commented Feb 16, 2023 •

edited

Loading