Avoid possible race condition using locks #6850

Pushpalanka · 2024-07-04T16:02:01Z

Why the changes in this PR are needed?

to address #6849

What are the changes in this PR?

Add locks when reading and writing the bundle plugin configuration.

No tests are added, as not sure exactly on how to write a test for this. The issue is experienced when checking for race conditions while using bundle plugin status listeners while using OPA as a library.

mjungsbluth · 2024-07-04T18:19:29Z

plugins/bundle/plugin.go

@@ -157,7 +157,10 @@ func (p *Plugin) Reconfigure(ctx context.Context, config interface{}) {
 	// Look for any bundles that have had their config changed, are new, or have been removed
 	newConfig := config.(*Config)
 	newBundles, updatedBundles, deletedBundles := p.configDelta(newConfig)
+
+	p.mtx.Lock()


There already is a p.cfgMtx that is held for this purpose further above

Yes, thanks for pointing out.

ashutosh-narkar · 2024-07-12T16:38:24Z

plugins/bundle/plugin.go

@@ -62,7 +62,7 @@ type Plugin struct {
 	downloaders       map[string]Loader
 	logger            logging.Logger
 	mtx               sync.Mutex
-	cfgMtx            sync.Mutex
+	cfgMtx            sync.RWMutex


We need to hold the write lock while updating the config. Also if you could look at the test failures that would be great. Thanks.

Thanks. I have addressed it.
Could you please have a look?

ashutosh-narkar

@Pushpalanka the changes seem ok. There are places where previously atomic operations are no longer atomic. So we need to be careful about the impact of the change.

ashutosh-narkar · 2024-07-18T17:59:06Z

plugins/bundle/plugin.go

+	p.cfgMtx.Lock()
+	p.config = *parsedConfig
+	p.cfgMtx.Unlock()
+


Why do need to hold the lock during plugin instantiation?

Thanks for raising this! Was carried away by the line 74 parsedConfig.Bundles as it reads the Bundles from the config. But yes we don't need this at initiation.

ashutosh-narkar · 2024-07-18T18:16:16Z

plugins/bundle/plugin.go

+	p.cfgMtx.RLock()
+	bundles := p.config.Bundles
+	p.cfgMtx.RUnlock()
+


Currently the below operation is performed atomically. Do we know the consequences of changing that behavior?

This needs to be changed (be atomic for the iteration).

netlify · 2024-07-21T23:33:07Z

✅ Deploy Preview for openpolicyagent ready!

Name	Link
🔨 Latest commit	`9b945de`
🔍 Latest deploy log	https://app.netlify.com/sites/openpolicyagent/deploys/669e991efa74c8000924ee67
😎 Deploy Preview	https://deploy-preview-6850--openpolicyagent.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

ashutosh-narkar · 2024-07-22T16:49:34Z

plugins/bundle/plugin.go

@@ -310,10 +313,14 @@ func (p *Plugin) UnregisterBulkListener(name interface{}) {

 // Config returns the plugins current configuration
 func (p *Plugin) Config() *Config {
+	p.cfgMtx.RLock()
+	defer p.cfgMtx.RUnlock()


We aren't making a copy of the config, so what benefit is the lock giving us here?

I am thinking we still need to protect against reading while writes are in-progress.

Just understood the point. Will correct.

Signed-off-by: Pushpalanka Jayawardhana <pushpalanka.jayawardhana@zalando.de>

ashutosh-narkar · 2024-07-24T00:44:01Z

plugins/bundle/plugin.go


 	// Look for any bundles that have had their config changed, are new, or have been removed
 	newConfig := config.(*Config)
 	newBundles, updatedBundles, deletedBundles := p.configDelta(newConfig)
 	p.config = *newConfig
+	p.cfgMtx.Unlock()


Have you considered the scenario of concurrent calls to Reconfigure? Could it break the following code flow?

What if we keep the existing code here?

ashutosh-narkar · 2024-07-24T00:59:17Z

plugins/bundle/plugin.go

+		p.cfgMtx.RLock()
+		src := p.config.Bundles[name]
+		p.cfgMtx.RUnlock()
+


If the write lock on the config was acquired at this point, won't it affect persistBundle?

I couldn't see a concern with persistBundle as src is read to a new variable and passed. However if we acquire the write lock for the whole method of Reconfigure as in previous comment, then this read lock will be a problem.
Not sure if I understood your concern here.

ashutosh-narkar · 2024-07-24T01:02:51Z

@Pushpalanka thanks for the back and forth on this and also for addressing my comments. Dealing with mutexes is tricky and sometimes it's hard to tell if a change can have unintended consequences. So I appreciate your work on this.

Pushpalanka · 2024-08-01T11:38:37Z

@Pushpalanka thanks for the back and forth on this and also for addressing my comments. Dealing with mutexes is tricky and sometimes it's hard to tell if a change can have unintended consequences. So I appreciate your work on this.

Thanks @ashutosh-narkar for your patience and support so far. I am surfing unknown waters here, yet understand this is a tricky and critical code segments I am touching. Feel free to take over as well if this becoming less efficient. :)

ashutosh-narkar · 2024-08-05T19:04:08Z

@Pushpalanka thanks for the back and forth on this and also for addressing my comments. Dealing with mutexes is tricky and sometimes it's hard to tell if a change can have unintended consequences. So I appreciate your work on this.

Thanks @ashutosh-narkar for your patience and support so far. I am surfing unknown waters here, yet understand this is a tricky and critical code segments I am touching. Feel free to take over as well if this becoming less efficient. :)

I'll spend sometime looking into this. Thanks for reporting the issue and working on a fix.

ashutosh-narkar · 2024-08-08T21:35:57Z

Closing in favor of #6921. Thanks for working on this @Pushpalanka!

Pushpalanka · 2024-08-08T23:19:44Z

Closing in favor of #6921. Thanks for working on this @Pushpalanka!

Thanks a lot @ashutosh-narkar 👍

Pushpalanka force-pushed the race-condition branch from 35ab4b2 to 58bea07 Compare July 4, 2024 16:13

mjungsbluth reviewed Jul 4, 2024

View reviewed changes

Pushpalanka closed this Jul 5, 2024

Pushpalanka reopened this Jul 11, 2024

Pushpalanka force-pushed the race-condition branch from 1a67ff8 to c3689e8 Compare July 11, 2024 09:24

Pushpalanka marked this pull request as draft July 11, 2024 09:28

Pushpalanka force-pushed the race-condition branch 2 times, most recently from 57f8f05 to 5c620d6 Compare July 11, 2024 09:37

Pushpalanka marked this pull request as ready for review July 11, 2024 15:16

Pushpalanka force-pushed the race-condition branch from 0603dc8 to 291775a Compare July 11, 2024 15:20

ashutosh-narkar reviewed Jul 12, 2024

View reviewed changes

Pushpalanka force-pushed the race-condition branch 4 times, most recently from ae231b0 to 8d9ff08 Compare July 17, 2024 18:50

Pushpalanka requested a review from ashutosh-narkar July 17, 2024 21:15

ashutosh-narkar reviewed Jul 18, 2024

View reviewed changes

Pushpalanka force-pushed the race-condition branch 4 times, most recently from fc89516 to 927bf03 Compare July 22, 2024 14:31

ashutosh-narkar reviewed Jul 22, 2024

View reviewed changes

Pushpalanka added 7 commits July 22, 2024 19:38

Add locks for plugin config to avoid possible race condition.

b03f1c1

Signed-off-by: Pushpalanka Jayawardhana <pushpalanka.jayawardhana@zalando.de>

Modify lock to be RW.

640e7e6

Signed-off-by: Pushpalanka Jayawardhana <pushpalanka.jayawardhana@zalando.de>

Modify to read lock for config copy.

69b9a54

Signed-off-by: Pushpalanka Jayawardhana <pushpalanka.jayawardhana@zalando.de>

Modify reconfigure back to write lock.

2ea4aeb

Signed-off-by: Pushpalanka Jayawardhana <pushpalanka.jayawardhana@zalando.de>

Remove redundant read lock.

56e6560

Signed-off-by: Pushpalanka Jayawardhana <pushpalanka.jayawardhana@zalando.de>

Add back read lock.

902fada

Signed-off-by: Pushpalanka Jayawardhana <pushpalanka.jayawardhana@zalando.de>

Add read locks.

07ce3f5

Signed-off-by: Pushpalanka Jayawardhana <pushpalanka.jayawardhana@zalando.de>

Pushpalanka added 6 commits July 22, 2024 19:38

Remove read lock acquiring is block is already under a write lock.

f362405

Signed-off-by: Pushpalanka Jayawardhana <pushpalanka.jayawardhana@zalando.de>

Remove read lock acquiring is block is already under a write lock.

7f1cb5d

Signed-off-by: Pushpalanka Jayawardhana <pushpalanka.jayawardhana@zalando.de>

Avoid recursive read locking.

0d6beb9

Signed-off-by: Pushpalanka Jayawardhana <pushpalanka.jayawardhana@zalando.de>

Address review comments.

bf880ae

Signed-off-by: Pushpalanka Jayawardhana <pushpalanka.jayawardhana@zalando.de>

Remove the need for config lock at persistBundle

65fe4f5

Signed-off-by: Pushpalanka Jayawardhana <pushpalanka.jayawardhana@zalando.de>

Address review comments

9b945de

Signed-off-by: Pushpalanka Jayawardhana <pushpalanka.jayawardhana@zalando.de>

Pushpalanka force-pushed the race-condition branch from e28d88c to 9b945de Compare July 22, 2024 17:38

ashutosh-narkar reviewed Jul 24, 2024

View reviewed changes

ashutosh-narkar closed this Aug 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Avoid possible race condition using locks #6850

Avoid possible race condition using locks #6850

Pushpalanka commented Jul 4, 2024

mjungsbluth Jul 4, 2024

Pushpalanka Jul 5, 2024

ashutosh-narkar Jul 12, 2024

Pushpalanka Jul 17, 2024

ashutosh-narkar left a comment

ashutosh-narkar Jul 18, 2024

Pushpalanka Jul 21, 2024 •

edited

Loading

ashutosh-narkar Jul 18, 2024

mjungsbluth Jul 19, 2024 •

edited

Loading

netlify bot commented Jul 21, 2024 •

edited

Loading

ashutosh-narkar Jul 22, 2024

Pushpalanka Jul 22, 2024

Pushpalanka Jul 22, 2024

ashutosh-narkar Jul 24, 2024

ashutosh-narkar Jul 24, 2024

Pushpalanka Aug 1, 2024

ashutosh-narkar commented Jul 24, 2024

Pushpalanka commented Aug 1, 2024

ashutosh-narkar commented Aug 5, 2024

ashutosh-narkar commented Aug 8, 2024

Pushpalanka commented Aug 8, 2024

Avoid possible race condition using locks #6850

Avoid possible race condition using locks #6850

Conversation

Pushpalanka commented Jul 4, 2024

Why the changes in this PR are needed?

What are the changes in this PR?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ashutosh-narkar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Pushpalanka Jul 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mjungsbluth Jul 19, 2024 • edited Loading

Choose a reason for hiding this comment

netlify bot commented Jul 21, 2024 • edited Loading

✅ Deploy Preview for openpolicyagent ready!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ashutosh-narkar commented Jul 24, 2024

Pushpalanka commented Aug 1, 2024

ashutosh-narkar commented Aug 5, 2024

ashutosh-narkar commented Aug 8, 2024

Pushpalanka commented Aug 8, 2024

Pushpalanka Jul 21, 2024 •

edited

Loading

mjungsbluth Jul 19, 2024 •

edited

Loading

netlify bot commented Jul 21, 2024 •

edited

Loading