fix(clustering/rpc): sync retry timeout due to block

[StarlightIbuki]

Summary

timeout should not be considered a failure trial

Checklist

• The Pull Request has tests
• A changelog file has been created under changelog/unreleased/kong or skip-changelog label added on PR if changelog is unnecessary. README.md
• There is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - PUT DOCS PR HERE

Issue reference

KAG-6224

[ADD-SP]

    kong/clustering/services/sync/rpc.lua:380:16: unused function start_sync_once_timer

Could you fix the linting?

[StarlightIbuki]

    kong/clustering/services/sync/rpc.lua:380:16: unused function start_sync_once_timer

Could you fix the linting?

done

[chronolaw]

Now timeout option is 0:

local SYNC_MUTEX_OPTS = { name = "get_delta", timeout = 0, }

So how the error timeout will happen? Or it will always be timeout?

[StarlightIbuki]

@chronolaw This is a locked operation, where timeout = 0 means it will fail instantly if the lock can not be acquired. By tolerating timeout in this case, we allow sync_once to actually retry instead of giving up when another coroutine holds the lock

[chronolaw]

Do you mean that it will always get a timeout error?

[StarlightIbuki]

@chronolaw No. When the callback fails or errors out it will emit a different error. That is what we count for a real error

[StarlightIbuki]

I.e, if we have no error but the version still does not match, we count it as 1 trial; if it's a real failure (internal or external), we still count it as 1 trial. This way we prevent it from looping forever; but meanwhile we do not want a running coroutine to block the sync_once call and make it quickly exhaust all the chances, so we do not count timeouts(lock failure)

[chronolaw]

@StarlightIbuki do not forget cherry-pick.