Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-execute auth policies on conflict #4102

Merged
merged 18 commits into from
Aug 11, 2022

Conversation

eddyashton
Copy link
Member

Includes #4101, so keeping as a draft until that has merged. This will also need to be manually merged with #4093, since they touch the same lines of auth code.

The auth policies were previously executed once, at the start of process_command(), and were not re-executed on Tx conflicts, meaning they could break serializability. The fix is simple (moving a block of code in frontend.h), but it's non-trivial to test. I added a TestManualConflictsFrontend that uses condition_variable wait points to insert arbitrary execution (including committing other transactions) during an endpoint's execution. I've also added a bunch of other unit tests with this, to confirm operations have the expected dependencies. The relevant unit test, testing the auth policy issue that we're actually looking at, is the last one ("Removed caller ident post-read").

@ghost
Copy link

ghost commented Aug 4, 2022

auth_transactionality@48498 aka 20220811.6 vs main ewma over 20 builds from 48239 to 48486

Click to see table

main

build_id build_number tpcc_sgx_cft^ tpcc_sgx_cft_mem ls_sgx_cft^ ls_sgx_cft_mem ls_jwt_sgx_cft^ ls_jwt_sgx_cft_mem ls_js_sgx_cft^ ls_js_sgx_cft_mem ls_v8_sgx_cft^ ls_v8_sgx_cft_mem ls_full_js_sgx_cft^ ls_full_js_sgx_cft_mem ls_full_v8_sgx_cft^ ls_full_v8_sgx_cft_mem ls_js_jwt_sgx_cft^ ls_js_jwt_sgx_cft_mem hist_sgx_cft^ RB put (/s)^ CHAMP put (/s)^ RB get (/s)^ CHAMP get (/s)^
48239 20220802.2 6341.07 8.32349e+07 20109.1 1.66503e+07 5683.14 1.56017e+07 2583.4 1.00967e+07 1609.45 1.66597e+08 2124.37 9.31027e+06 1433.98 9.79149e+07 1933.24 9.04813e+06 20349.6 891366 1.37368e+06 9.39881e+06 3.58048e+07
48248 20220802.6 6346.87 8.32349e+07 20263.5 1.6126e+07 5553.34 1.53396e+07 2552.54 1.00967e+07 1590.59 1.66597e+08 2115.99 9.31027e+06 1419.19 9.79149e+07 1937.45 8.78598e+06 18446.5 909334 1.39206e+06 9.28798e+06 3.5189e+07
48267 20220803.3 6373.45 8.45456e+07 20471.7 1.58639e+07 5651.34 1.53396e+07 2611.14 9.83456e+06 1583.33 1.66597e+08 2114.15 9.31027e+06 1395.8 9.81771e+07 1929.23 9.04813e+06 20297.9 902400 1.3769e+06 9.23354e+06 3.56174e+07
48290 20220803.14 6381.82 8.3497e+07 20458.1 1.58639e+07 5640.15 1.53396e+07 2540.33 1.00967e+07 1592.15 1.66597e+08 2120.64 9.31027e+06 1422.34 9.87014e+07 1924.29 9.04813e+06 18289 876257 1.36151e+06 9.20445e+06 3.58669e+07
48304 20220804.1 6283.98 8.37592e+07 20390.7 1.58639e+07 5621.96 1.56017e+07 2545.3 1.00967e+07 1617.06 1.64762e+08 2117.04 9.31027e+06 1438.5 9.84392e+07 2016.09 8.78598e+06 20426.3 908481 1.35574e+06 9.19206e+06 3.55556e+07
48323 20220804.8 6308.84 8.32349e+07 20224.2 1.69124e+07 5631.75 1.53396e+07 2557.44 9.83456e+06 1617.92 1.66859e+08 2117.06 9.31027e+06 1364.2 9.76528e+07 2009.73 8.78598e+06 21082.9 893076 1.36898e+06 9.30064e+06 3.57417e+07
48348 20220804.16 6470.12 8.40213e+07 20162.3 1.63882e+07 5611.59 1.56017e+07 2610.84 1.00967e+07 1603.88 1.66859e+08 2133.3 9.31027e+06 1408.7 9.87014e+07 1936.8 9.04813e+06 20767.6 874353 1.31602e+06 8.95888e+06 3.50685e+07
48375 20220805.3 6581.93 8.40213e+07 20244.8 1.58639e+07 5588.85 1.53396e+07 2598.55 1.00967e+07 1590.43 1.66335e+08 2110.71 9.31027e+06 1437.79 9.79149e+07 1965.96 9.04813e+06 20769.3 901323 1.3709e+06 9.34298e+06 3.58669e+07
48376 20220805.4 6575.88 8.40213e+07 20602.8 1.58639e+07 5680.24 1.53396e+07 2480.28 1.00967e+07 1649.69 1.66859e+08 2122.21 9.31027e+06 1442.64 9.81771e+07 1929.88 9.04813e+06 20495.5 911473 1.37026e+06 9.41172e+06 3.63121e+07
48391 20220805.11 6446.79 8.32349e+07 20014.6 1.6126e+07 5628.3 1.58639e+07 2594.33 1.00967e+07 1603.05 1.66597e+08 2073.94 9.31027e+06 1415.05 9.63421e+07 1920.2 9.04813e+06 18389.9 909972 1.37044e+06 9.18377e+06 3.59298e+07
48399 20220808.2 6499.33 8.42835e+07 20415.4 1.66503e+07 5670.06 1.53396e+07 2554.96 1.00967e+07 1630.43 1.645e+08 2118.44 9.31027e+06 1412.02 9.81771e+07 1932.81 9.04813e+06 20513.5 871966 1.36106e+06 9.20859e+06 3.56168e+07
48407 20220808.5 6438.4 8.40213e+07 20249.2 1.6126e+07 5638.84 1.53396e+07 2548 1.00967e+07 1595.51 1.67121e+08 2108.24 9.31027e+06 1430.87 9.81771e+07 1927.33 8.78598e+06 19704.5 888306 1.36669e+06 9.19206e+06 3.58669e+07
48422 20220808.12 6398.02 8.3497e+07 20137.7 1.63882e+07 5695.03 1.53396e+07 2625.18 9.83456e+06 1582.61 1.66859e+08 2120.63 8.78598e+06 1404 9.73907e+07 1935.04 9.04813e+06 19485.4 902483 1.39112e+06 9.26278e+06 3.59298e+07
48427 20220808.14 6230.03 8.40213e+07 19679.8 1.58639e+07 5547.25 1.53396e+07 2417.7 1.03588e+07 1550.66 1.645e+08 2072.83 9.04813e+06 1369.48 9.76528e+07 1959.08 9.04813e+06 17632.4 920524 1.40023e+06 9.3901e+06 3.63114e+07
48435 20220808.17 6396.62 8.37592e+07 20212.4 1.58639e+07 5676.31 1.56017e+07 2492.05 1.00967e+07 1582.1 1.66859e+08 2084.38 9.31027e+06 1401.8 9.79149e+07 1936.03 9.04813e+06 22698 907599 1.38341e+06 9.30486e+06 3.55556e+07
48452 20220809.2 6397.96 8.37592e+07 20177.9 1.63882e+07 5695.08 1.56017e+07 2498.61 1.00967e+07 1602.55 1.66597e+08 2093.11 9.31027e+06 1399.63 9.81771e+07 1931.37 8.78598e+06 20235.1 903311 1.37495e+06 9.21688e+06 3.54939e+07
48461 20220809.7 6585.9 8.3497e+07 20158 1.63882e+07 5669.79 1.53396e+07 2557.5 1.00967e+07 1626.24 1.645e+08 2121.12 9.31027e+06 1443.42 9.84392e+07 1986.42 9.04813e+06 18271.3 886722 1.41396e+06 9.41163e+06 3.59292e+07
48466 20220809.9 6444.69 8.37592e+07 20005.2 1.56017e+07 5605.49 1.56017e+07 2541.33 1.00967e+07 1618.96 1.66859e+08 2115.35 9.31027e+06 1450.97 9.81771e+07 2014.27 9.04813e+06 20835.2 901011 1.37201e+06 9.22938e+06 3.58669e+07
48469 20220810.1 6186.62 8.42835e+07 20033.2 1.58639e+07 5604.55 1.53396e+07 2556.71 9.83456e+06 1587.94 1.64762e+08 2082.75 9.04813e+06 1421.15 9.84392e+07 1944.56 9.04813e+06 22527.5 906189 1.38172e+06 9.21684e+06 3.58669e+07
48486 20220811.1 6270.83 8.40213e+07 19935.8 1.58639e+07 5612.96 1.53396e+07 2499.35 1.2456e+07 1583.15 1.66859e+08 2112.67 9.31027e+06 1405.59 9.81771e+07 1923.87 9.04813e+06 23657.6 917679 1.38602e+06 9.28794e+06 3.56788e+07

auth_transactionality

build_id build_number tpcc_sgx_cft^ tpcc_sgx_cft_mem ls_sgx_cft^ ls_sgx_cft_mem ls_jwt_sgx_cft^ ls_jwt_sgx_cft_mem ls_js_sgx_cft^ ls_js_sgx_cft_mem ls_v8_sgx_cft^ ls_v8_sgx_cft_mem ls_full_js_sgx_cft^ ls_full_js_sgx_cft_mem ls_full_v8_sgx_cft^ ls_full_v8_sgx_cft_mem ls_js_jwt_sgx_cft^ ls_js_jwt_sgx_cft_mem hist_sgx_cft^ RB put (/s)^ CHAMP put (/s)^ RB get (/s)^ CHAMP get (/s)^
48358 20220804.20 6277.95 8.3497e+07 20149.1 1.58639e+07 5625.77 1.53396e+07 2542.22 1.00967e+07 1595.73 1.66597e+08 2081.43 9.31027e+06 1400.42 9.76528e+07 1874.82 1.08831e+07 17398.1 888919 1.38209e+06 9.10619e+06 3.57411e+07
48477 20220810.4 6447.08 8.37592e+07 20106.9 1.74367e+07 5637.85 1.58639e+07 2551.35 1.00967e+07 1625.99 1.66597e+08 2125.65 9.04813e+06 1401.9 9.73907e+07 1896.46 8.78598e+06 20439.9 901276 1.38311e+06 9.33868e+06 3.58663e+07
48498 20220811.6 6277.07 8.45456e+07 20017.3 1.63882e+07 5617.61 1.53396e+07 2477.3 1.00967e+07 1597.23 1.66597e+08 2146.64 9.31027e+06 1402.7 9.81771e+07 1933.44 9.04813e+06 20504.6 903870 1.37958e+06 9.39445e+06 3.59298e+07

images

Copy link
Member

@MahatiC MahatiC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm pending rebase

@eddyashton eddyashton marked this pull request as ready for review August 10, 2022 09:33
@eddyashton eddyashton requested a review from a team August 10, 2022 09:33
@eddyashton eddyashton added 2.x-todo PRs which should be backported to 2.x auto-backport Automatically backport this PR to LTS branch labels Aug 10, 2022
@eddyashton eddyashton merged commit 33b9f4c into microsoft:main Aug 11, 2022
@github-actions
Copy link

💔 All backports failed

Status Branch Result
release/2.x Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

backport --pr 4102

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details

eddyashton added a commit to eddyashton/CCF that referenced this pull request Aug 11, 2022
@eddyashton eddyashton added the backported This PR was successfully backported to LTS branch label Aug 11, 2022
MahatiC pushed a commit to MahatiC/CCF that referenced this pull request Aug 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.x-todo PRs which should be backported to 2.x auto-backport Automatically backport this PR to LTS branch backported This PR was successfully backported to LTS branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants