🐛 Recreate watcher if the file unlinked and replaced

[m-messiah]

The current certwatcher misses some events, so if the certificate is maintained in a failsafe-backup way it cannot be watched and fails to get any updates.

For example, when it started with certwartcher.New("app.crt", "app.key") and someone does:

curl -O app.crt.new example.com/cert
curl -O app.key.new example.com/key
diff app.crt.new app.crt || (echo "No change"; exit 0)
ln app.crt app.crt.old
ln app.key app.key.old
mv app.crt.new app.crt
mv app.key.new app.key

The FD of the file gets these inotify events and stops being watched.

# Diff
app.crt OPEN
app.crt CLOSE_NOWRITE,CLOSE
# Move
app.crt ATTRIB
app.crt MOVE_SELF
app.crt ATTRIB
app.crt DELETE_SELF

Please take a look at the added test for the implementation of failsafe cert replacement.
The problem is that in this stage because of hardlink the certwatcher continues to watch app.crt.old and app.key.old inodes until they are removed. If they are removed properly - it gets a REMOVE fsnotify event and recreates the watch, but if they use something like unlink and still have open FD or another watch - they would be watched forever.

The case is already mentioned in fsnotify library comments: https://github.com/fsnotify/fsnotify/blob/c1467c02fba575afdb5f4201072ab8403bbf00f4/fsnotify.go#L40-L48

// # Linux notes
//
// When a file is removed a Remove event won't be emitted until all file
// descriptors are closed, and deletes will always emit a Chmod. For example:
//
//      fp := os.Open("file")
//      os.Remove("file")        // Triggers Chmod
//      fp.Close()               // Triggers Remove
//

and https://github.com/fsnotify/fsnotify/blob/c1467c02fba575afdb5f4201072ab8403bbf00f4/fsnotify.go#L135-L139

        //   fsnotify.Chmod     Attributes were changed. On Linux this is also sent
        //                      when a file is removed (or more accurately, when a
        //                      link to an inode is removed). On kqueue it's sent
        //                      when a file is truncated. On Windows it's never
        //                      sent.

So the PR adds fsnotify.Chmod event to be treated equally to fsnotify.Remove in case someone still looks at the old inode, because for certiwatcher the important data are in the filename, not inode.

Note, there is no harm in recreating the watch for a file that is already monitored (in case Chmod was thrown for any other reason) as watcher would just reuse the existing: https://github.com/fsnotify/fsnotify/blob/c1467c02fba575afdb5f4201072ab8403bbf00f4/fsnotify.go#L277-L312

// A path can only be watched once; watching it more than once is a no-op and will
// not return an error. Paths that do not yet exist on the filesystem cannot be
// watched.
//
// A watch will be automatically removed if the watched path is deleted or
// renamed. The exception is the Windows backend, which doesn't remove the
// watcher on renames.

[vincepri]

/ok-to-test

[vincepri]

/approve
/assign @sbueringer

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: m-messiah, vincepri

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [vincepri]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sbueringer]

/lgtm
/hold

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: b9055ed877ccf5bafe16326c138eddb0e39f675f

[alvaroaleman]

/hold cancel
Thank you!

[vincepri]

/cherry-pick release-1.8

[k8s-infra-cherrypick-robot]

@vincepri: cannot checkout release-1.8: error checking out "release-1.8": exit status 1 error: pathspec 'release-1.8' did not match any file(s) known to git

In response to this:

/cherry-pick release-1.8

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[vincepri]

/cherry-pick release-0.18

[vincepri]

/cherry-pick release-0.17

[k8s-infra-cherrypick-robot]

@vincepri: new pull request created: #2919

In response to this:

/cherry-pick release-0.18

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[k8s-infra-cherrypick-robot]

@vincepri: new pull request created: #2920

In response to this:

/cherry-pick release-0.17

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.