feat: Use fsnotify to detect model/policy files change in casbin plugin #614

lyt122 · 2024-06-27T09:38:19Z

To fix #556

codecov · 2024-06-27T09:46:42Z

Codecov Report

Attention: Patch coverage is 72.83951% with 22 lines in your changes missing coverage. Please review.

Project coverage is 88.19%. Comparing base (e27e2f7) to head (ac85bd9).
Report is 7 commits behind head on main.

Files	Patch %	Lines
plugins/pkg/file/fs.go	75.51%	7 Missing and 5 partials ⚠️
plugins/plugins/casbin/config.go	70.96%	5 Missing and 4 partials ⚠️
plugins/plugins/casbin/filter.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #614      +/-   ##
==========================================
- Coverage   88.36%   88.19%   -0.17%     
==========================================
  Files         115      115              
  Lines        5456     5457       +1     
==========================================
- Hits         4821     4813       -8     
- Misses        454      461       +7     
- Partials      181      183       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

spacewander

Could you update

htnn/plugins/tests/integration/casbin_test.go

Line 118 in 0483a64

    
           time.Sleep(10 * time.Second) // TODO remove this once we switch the file change detector to inotify

to see if the change is detected?

spacewander · 2024-06-28T01:57:56Z

plugins/pkg/file/fs.go

+	fs = &Fsnotify{
+		Watcher: watcher,
+	}
+	return


We don't need this return?

spacewander · 2024-06-28T02:02:18Z

plugins/pkg/file/fs.go

+		return
+	}
+
+	watcher := newFsnotify().Watcher


Why don't we use defaultFsnotify?

plugins/pkg/file/fs.go

spacewander · 2024-06-28T02:11:11Z

plugins/plugins/casbin/filter.go

+					f.reloadEnforcer()
+				}, conf.modelFile, conf.policyFile)
+				if err != nil {
+					api.LogErrorf("failed to update Enforcer: %v", err)


The msg is not precise. The code above doesn't update Enforcer

plugins/pkg/file/fs.go

spacewander · 2024-06-28T02:25:34Z

plugins/pkg/file/fs.go

+			if !ok {
+				return
+			}
+			logger.Info(fmt.Sprintf("event: %v", event))


Just put the event as an argument is enough, we don't need to use Sprintf

When I use logger.Info("event: %v", event), an error is occurred:
DPANIC file file/fs.go:100 odd number of arguments passed as key-value pairs for logging {"ignored key": "WRITE \"example3760250304\""}
so I use Sprintf.

As the error message indicated, the arguments should be key-value pairs, which have even numbers, for example:

htnn/api/pkg/plugins/plugins.go

Line 73 in 1d5db16

logger.Info("register plugin type", "name", name)

spacewander · 2024-06-28T02:27:26Z

plugins/pkg/file/fs.go

+			}
+			logger.Info(fmt.Sprintf("event: %v", event))
+			onChange()
+			return


We should keep watching, and should not exit after the first event. This is a waste of goroutine and watcher, and force the user to rewatch each time.

spacewander · 2024-06-28T02:51:32Z

plugins/pkg/file/fs_test.go

-	tmpfile.Close()
-	assert.False(t, IsChanged(f))
+	tmpfile.Sync()
+	mu.Lock()


Does the lock here guarantee the callback in WatchFiles run before assertion? The assertion may get the lock first.

spacewander · 2024-06-28T03:04:17Z

plugins/pkg/file/fs.go

+			logger.Error(err, "failed to close fsnotify watcher")
+		}
+	}(w)
+	err := w.Add(files.Name)


Consider a case: people remove the file and then create a new file with the same name. The fsnotify won't be triggered in this case. What about watching the file's directory and filtering according to the name?

spacewander · 2024-06-28T05:09:38Z

I think about it again. Using global fsnotify is tough for a beginner. We can implement a per-file fsnotify version first.

spacewander · 2024-06-28T05:13:15Z

plugins/pkg/file/fs.go

 )

-func IsChanged(files ...*File) bool {
+func Update(onChange func(), files ...*File) (err error) {


The Update is redundant, use WatchFiles is enough.

spacewander · 2024-07-04T06:06:49Z

@lyt122
I converted this PR to draft, feel free to click the "Ready for review" button or @ me if you are ready.

lyt122 · 2024-07-11T04:50:04Z

@spacewander

spacewander · 2024-07-11T09:24:27Z

plugins/plugins/casbin/filter.go

-			}
-		}()
+	if getChanged() {
+		conf.reloadEnforcer()


Why don't we call this inside file.WatchFiles(func() {?

because if reloadenforcer called in file.WatchFiles(), the config won't be changed. So I need the flag

I wonder why. Could you explain it?

I don't know. In fact, this problem bother me for a long time.

spacewander · 2024-07-11T09:25:41Z

plugins/plugins/casbin/filter.go

@@ -35,33 +35,27 @@ type filter struct {
 	config    *config
 }

+var (
+	Changed   = false


We don't need this flag if we can do reloadEnforcer in the fsnotify's callback

spacewander · 2024-07-11T09:26:43Z

plugins/plugins/casbin/filter.go

+	ChangedMu.Unlock()
+	role, _ := headers.Get(conf.Token.Name) // role can be ""
+	url := headers.Url()
+	err := file.WatchFiles(func() {


Why can't we move this part inside config.Init?

spacewander · 2024-07-11T09:27:05Z

plugins/tests/integration/casbin_test.go

 	// configuration is not changed, but file changed
 	err = os.WriteFile(policyFile2.Name(), []byte(policy), 0755)
 	require.Nil(t, err)
+
+	//wait to run reloadEnforcer
+	time.Sleep(5 * time.Second)


Ping @lyt122
Don't miss #614 (comment)

spacewander · 2024-07-11T09:27:50Z

plugins/tests/integration/casbin_test.go

 	hdr := http.Header{}
 	hdr.Set("customer", "alice")

 	assert.Eventually(t, func() bool {
 		resp, _ := dp.Post("/echo", hdr, strings.NewReader("any"))
 		return resp != nil && resp.StatusCode == 200
-	}, 1*time.Second, 10*time.Millisecond)
+	}, 3*time.Second, 1*time.Second)


Suggested change

}, 3*time.Second, 1*time.Second)

}, 3*time.Second, 100*time.Millisecond)

Better to use shorter interval to reduce test time

spacewander · 2024-07-11T09:29:54Z

plugins/plugins/casbin/filter_test.go

@@ -88,6 +88,7 @@ func TestCasbin(t *testing.T) {
 						assert.Equal(t, tt.status, 0)
 					} else {
 						assert.Equal(t, tt.status, lr.Code)
+						assert.False(t, Changed)


When Changed is removed, we can compare the field in conf directly for test

spacewander · 2024-07-11T09:40:16Z

plugins/pkg/file/fs.go

+func watchFiles(onChanged func(), file *File) {
+	dir := filepath.Dir(file.Name)
+	defer func() {
+		storeWatchedFiles.lock.Lock()


The global storeWatchedFiles makes things complex. And the current impl is incorrect if multiple config watch the same file.

We can improve it like this:

create a watcher which wraps the fsnotify when initing the config

add files into the watcher

start the watcher (remember to stop the goroutine once the watcher is closed)

lyt122 · 2024-07-12T17:40:26Z

@spacewander ready to review

spacewander · 2024-07-13T05:09:30Z

plugins/plugins/casbin/config.go

+	runtime.SetFinalizer(conf, func(conf *config) {
+		err := conf.watcher.Stop()
+		if err != nil {
+			api.LogErrorf("failed to close watcher, err: %v", err)


Suggested change

api.LogErrorf("failed to close watcher, err: %v", err)

api.LogErrorf("failed to stop watcher, err: %v", err)

spacewander · 2024-07-13T05:10:02Z

plugins/plugins/casbin/config.go

-	e, err := casbin.NewEnforcer(conf.Rule.Model, conf.Rule.Policy)
+	conf.watcher = watcher
+
+	err = conf.watcher.AddFile(conf.modelFile, conf.policyFile)


Better to name it AddFiles if this method accept multiple files

spacewander · 2024-07-13T05:12:53Z

plugins/pkg/file/fs.go

-	return defaultFs.Stat(path)
+func (w *Watcher) Start(onChanged func()) {
+	go func() {
+		logger.Info("start watch files")


Suggested change

logger.Info("start watch files")

logger.Info("start watching files")

plugins/pkg/file/fs.go

plugins/pkg/file/fs_test.go

spacewander · 2024-07-13T05:30:24Z

plugins/pkg/file/fs.go

+		for {
+			select {
+			case event, ok := <-w.watcher.Events:
+				if !ok {


There is no need to add this branch as the watcher doesn't close the Events.

spacewander · 2024-07-13T05:30:47Z

plugins/pkg/file/fs.go

+				logger.Info("file changed: ", "event", event)
+				onChanged()
+			case err, ok := <-w.watcher.Errors:
+				if !ok {


lyt122 · 2024-07-14T13:40:40Z

@spacewander ready to review

spacewander · 2024-07-15T04:02:27Z

plugins/pkg/file/fs_test.go

-	assert.False(t, IsChanged(f))
-	time.Sleep(1000 * time.Millisecond)
+
+	tmpfile, _ := os.CreateTemp("./", "example")


Suggested change

tmpfile, _ := os.CreateTemp("./", "example")

tmpfile, _ := os.CreateTemp("", "example")

would be better so that the test case won't pollute the git repo.

plugins/pkg/file/fs.go

spacewander · 2024-07-15T04:09:08Z

plugins/pkg/file/fs.go

+		for {
+			select {
+			case event := <-w.watcher.Events:
+				if _, exists := w.files[event.Name]; exists {


The test case doesn't work on Mac. I spent some time investigating it and found that the event.Name is example1104322652 and the map is map[./example1104322652:true]. Let's convert the input file name to absolute path.

lyt122 · 2024-07-15T05:12:53Z

@spacewander

spacewander · 2024-07-15T08:28:40Z

plugins/pkg/file/fs.go

+		if _, exists := w.files[file.Name]; !exists {
+			w.files[file.Name] = true
+		}
+		dir, err := filepath.Abs(file.Name)


Missing a Dir call?

spacewander · 2024-07-15T08:36:29Z

plugins/pkg/file/fs.go

+		for {
+			select {
+			case event := <-w.watcher.Events:
+				if event.Op&fsnotify.Chmod == fsnotify.Chmod {


If the Op is Chmod | SomethingElse, the SomethingElse event will be ignored.

lyt122 · 2024-07-15T12:22:28Z

@spacewander

spacewander · 2024-07-16T02:09:10Z

plugins/pkg/file/fs.go

+		for {
+			select {
+			case event := <-w.watcher.Events:
+				if event.Op.Has(fsnotify.Chmod) {


Does this solve the issue that the Op both has Chmod and SomethingElse flags? In this case, the SomethingElse event will be dropped.

lyt122 · 2024-07-16T02:50:53Z

@spacewander

spacewander · 2024-07-16T08:43:13Z

plugins/pkg/file/fs.go

+				if event.Op&fsnotify.Chmod != 0 {
+					event.Op &= ^fsnotify.Chmod // Remove the Chmod bit
+					if event.Op == 0 {
+						continue // Skip if it was only a Chmod event


This code works but why don't we just use event.Op == fsnotify.Chmod?

spacewander · 2024-07-17T02:47:35Z

Merged. Thanks!

lyt122 added 3 commits June 27, 2024 15:56

feat: Use fsnotify to detect model/policy files change in casbin plugin

66d0cb7

feat: Use fsnotify to detect model/policy files change in casbin plugin

fe5182e

feat: Use fsnotify to detect model/policy files change in casbin plugin

1bfb1d4

spacewander reviewed Jun 27, 2024

View reviewed changes

lyt122 added 3 commits June 27, 2024 23:27

Merge branch 'mosn:main' into main

3120753

feat: Use fsnotify to detect model/policy files change in casbin plugin

7bf2834

Merge branch 'main' of github.com:lyt122/htnn

d0dc175

spacewander reviewed Jun 28, 2024

View reviewed changes

lyt122 added 2 commits June 30, 2024 23:50

feat: Use fsnotify to detect model/policy files change in casbin plugin

17ed368

format go

99bf29d

lyt122 changed the title ~~feat: Use fsnotify to detect model/policy files change in casbin plugin~~ [WIP]feat: Use fsnotify to detect model/policy files change in casbin plugin Jun 30, 2024

Merge branch 'mosn:main' into main

57329f4

github-actions bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jul 3, 2024

spacewander marked this pull request as draft July 4, 2024 06:05

lyt122 added 10 commits July 6, 2024 19:57

refactor codes

6eb6b03

refactor codes

5c99c69

refactor codes

846ff5b

refactor codes

f330ec6

add test codes

29822b8

add test codes

a7bf59d

add test codes

6719586

add test codes

daf2c24

add test codes

7120888

fix: avoid data race

c2ffc2e

lyt122 marked this pull request as ready for review July 8, 2024 14:14

lyt122 changed the title ~~[WIP]feat: Use fsnotify to detect model/policy files change in casbin plugin~~ feat: Use fsnotify to detect model/policy files change in casbin plugin Jul 8, 2024

Merge branch 'main' of github.com:lyt122/htnn

e34511d

lyt122 reopened this Jul 10, 2024

github-actions bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jul 10, 2024

refactor codes

7a1551f

spacewander reviewed Jul 11, 2024

View reviewed changes

lyt122 added 3 commits July 13, 2024 01:03

refactor codes

bbefd66

refactor codes

f239cf5

refactor codes

75d00da

spacewander reviewed Jul 13, 2024

View reviewed changes

lyt122 added 2 commits July 14, 2024 21:16

refactor: watch the dir of file

b9d7d06

refactor codes

0b03da5

spacewander reviewed Jul 15, 2024

View reviewed changes

lyt122 added 2 commits July 15, 2024 13:00

use file's abspath

8ca375a

refactor codes

363dd1d

spacewander reviewed Jul 15, 2024

View reviewed changes

refactor codes

b81b325

spacewander reviewed Jul 16, 2024

View reviewed changes

refactor codes

c879870

spacewander approved these changes Jul 16, 2024

View reviewed changes

refactor codes

ac85bd9

spacewander approved these changes Jul 17, 2024

View reviewed changes

spacewander merged commit f9cec7a into mosn:main Jul 17, 2024
13 checks passed

	}, 3time.Second, 1time.Second)
	}, 3time.Second, 100time.Millisecond)

	api.LogErrorf("failed to close watcher, err: %v", err)
	api.LogErrorf("failed to stop watcher, err: %v", err)

	logger.Info("start watch files")
	logger.Info("start watching files")

	tmpfile, _ := os.CreateTemp("./", "example")
	tmpfile, _ := os.CreateTemp("", "example")

feat: Use fsnotify to detect model/policy files change in casbin plugin #614

feat: Use fsnotify to detect model/policy files change in casbin plugin #614

Conversation

lyt122 commented Jun 27, 2024

codecov bot commented Jun 27, 2024 • edited Loading

Codecov Report

spacewander left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spacewander commented Jun 28, 2024

Choose a reason for hiding this comment

spacewander commented Jul 4, 2024

lyt122 commented Jul 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spacewander Jul 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lyt122 commented Jul 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lyt122 commented Jul 14, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lyt122 commented Jul 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lyt122 commented Jul 15, 2024

Choose a reason for hiding this comment

lyt122 commented Jul 16, 2024

Choose a reason for hiding this comment

spacewander commented Jul 17, 2024

codecov bot commented Jun 27, 2024 •

edited

Loading

spacewander Jul 11, 2024 •

edited

Loading