Plugin manager/bundle activation race condition #2343
Assignees
Labels
Comments
tsandall
added a commit
to tsandall/opa
that referenced
this issue
Apr 27, 2020
This change fixes a race condition in the manager that was caused by registering the storage trigger _after_ the plugins had been started. The problem was that if the bundle plugin was able to download and activate before the trigger registration in the manager went through, the store and the manager would be out-of-sync after startup. The bundle would activate successfully but the plugin manager would not see the change. This meant that the server health check, status plugin, etc. would report successful activation and clients using either of those APIs for synchronization could start querying. If they executed a query within this window, virtual docs would not be visible because the plugin manager would not yet have a compiler to return to the server. Similarly, if clients queried the v1/policies API they would see the raw policy contents but no AST (since the latter is retrieved from the compiler.) To remove the race condition the plugin manager simply registers the trigger before starting any of the plugins. This ensures that it sees all changes made by any of the plugins. Fixes open-policy-agent#2343 Signed-off-by: Torin Sandall <torinsandall@gmail.com>
patrick-east
pushed a commit
to patrick-east/opa
that referenced
this issue
Apr 27, 2020
This change fixes a race condition in the manager that was caused by registering the storage trigger _after_ the plugins had been started. The problem was that if the bundle plugin was able to download and activate before the trigger registration in the manager went through, the store and the manager would be out-of-sync after startup. The bundle would activate successfully but the plugin manager would not see the change. This meant that the server health check, status plugin, etc. would report successful activation and clients using either of those APIs for synchronization could start querying. If they executed a query within this window, virtual docs would not be visible because the plugin manager would not yet have a compiler to return to the server. Similarly, if clients queried the v1/policies API they would see the raw policy contents but no AST (since the latter is retrieved from the compiler.) To remove the race condition the plugin manager simply registers the trigger before starting any of the plugins. This ensures that it sees all changes made by any of the plugins. Fixes open-policy-agent#2343 Signed-off-by: Torin Sandall <torinsandall@gmail.com>
patrick-east
pushed a commit
that referenced
this issue
Apr 27, 2020
This change fixes a race condition in the manager that was caused by registering the storage trigger _after_ the plugins had been started. The problem was that if the bundle plugin was able to download and activate before the trigger registration in the manager went through, the store and the manager would be out-of-sync after startup. The bundle would activate successfully but the plugin manager would not see the change. This meant that the server health check, status plugin, etc. would report successful activation and clients using either of those APIs for synchronization could start querying. If they executed a query within this window, virtual docs would not be visible because the plugin manager would not yet have a compiler to return to the server. Similarly, if clients queried the v1/policies API they would see the raw policy contents but no AST (since the latter is retrieved from the compiler.) To remove the race condition the plugin manager simply registers the trigger before starting any of the plugins. This ensures that it sees all changes made by any of the plugins. Fixes #2343 Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
There is a race condition in the plugin manager whereby the bundle plugin can be started and finish activating the first bundle before the plugin manager registers a trigger on the store. If this happens then the manager does not see the first bundle activation and then subsequent queries can see inconsistent state (e.g., API calls that do not use the compiler but only access the store will return expected results, but API calls that use the compiler will see incorrect/inconsistent results.)
Thanks to @istalker2 for reporting and providing a script to easily repro the problem.
The text was updated successfully, but these errors were encountered: