[YSQL] YSQL upgrade unit test fails when using 2.20.1.3 snapshot #23182
Assignees
Labels
Comments
yugabyte-ci
added
kind/bug
myang2021
added a commit
that referenced
this issue
Jul 15, 2024
Summary: The following upgrade test that uses a 2.20.3.1 initdb snapshot currently fails: ./yb_build.sh release --java-test 'org.yb.pgsql.TestYsqlUpgrade#upgradeIsIdempotent' --java-test-args '-Dysql_sys_catalog_snapshot_path=/tmp/yugabyte-2.20.3.1/share/initial_sys_catalog_snapshot' The error looks like an expected diff found in pg_rewrite table: ``` Expected row: ...... :constraintDeps <> :stmt_location 0 :stmt_len 1006})] Actual row: ...... :constraintDeps <> :stmt_location 56 :stmt_len 406})] ``` After debugging, I found that this test runs ysql upgrade twice: 1. First time it runs ysql upgrade triggers running the following scripts sequentially ``` V44__19211__yb_stream_id_in_pg_replication_slots.sql V45__19128__yb_ash.sql ... V52__22028__yb_reset_analyze_statistics.sql ``` 2. Second time it deletes everything from pg_yb_migration table, and run ysql upgrade again. This caused to run sequentially ``` V9__10150__yb_extension_role.sql V10__9178__yb_is_local_table_function.sql ... V52__22028__yb_reset_analyze_statistics.sql ``` The script that caused the test failure is `V33__14209__yb_terminated_queries.sql`. This is a migration script that predates the 2.20.3.1 initdb snapshot. When the 2.20.3.1 initdb snapshot was built it ran yb_system_views.sql. In yb_system_views.sql there is "CREATE VIEW yb_terminated_queries ..." but in the migration script `V33__14209__yb_terminated_queries.sql` it has "CREATE OR REPLACE VIEW pg_catalog.yb_terminated_queries ...". These two statements differ textually and therefore their parse results will have different character position, statement length, etc. If we ran `V33__14209__yb_terminated_queries.sql` when the the initdb already has the view, the test fails because of parse results changes like :stmt_len 1006 vs :stmt_len 406. To fix test failure like this, I changed this test to only rerun the migration scripts that come after the 2.20.3.1 release by keeping the <baseline> row in pg_yb_migration. To ensure the default test behavior remains the same, when running with 2.0.9.0 (the default snapshot), the test still delete every row from pg_yb_migration table. Jira: DB-12122 Test Plan: 1. Run the default ysql upgrade test ./yb_build.sh release --java-test 'org.yb.pgsql.TestYsqlUpgrade' 2. Download various releases to get their initdb snapshots to validate the test. ./yb_build.sh release --sj --java-test 'org.yb.pgsql.TestYsqlUpgrade' --java-test-args '-Dysql_sys_catalog_snapshot_path=/tmp/yugabyte-2.20.3.1/share/initial_sys_catalog_snapshot' ./yb_build.sh release --sj --java-test 'org.yb.pgsql.TestYsqlUpgrade' --java-test-args '-Dysql_sys_catalog_snapshot_path=/tmp/yugabyte-2.0.9.0/share/initial_sys_catalog_snapshot' ./yb_build.sh release --sj --java-test 'org.yb.pgsql.TestYsqlUpgrade' --java-test-args '-Dysql_sys_catalog_snapshot_path=/tmp/yugabyte-2.11.2.0/share/initial_sys_catalog_snapshot' ./yb_build.sh release --sj --java-test 'org.yb.pgsql.TestYsqlUpgrade' --java-test-args '-Dysql_sys_catalog_snapshot_path=/tmp/yugabyte-2.12.12.0/share/initial_sys_catalog_snapshot' ./yb_build.sh release --sj --java-test 'org.yb.pgsql.TestYsqlUpgrade' --java-test-args '-Dysql_sys_catalog_snapshot_path=/tmp/yugabyte-2.14.13.0/share/initial_sys_catalog_snapshot' ./yb_build.sh release --sj --java-test 'org.yb.pgsql.TestYsqlUpgrade' --java-test-args '-Dysql_sys_catalog_snapshot_path=/tmp/yugabyte-2.16.9.0/share/initial_sys_catalog_snapshot' ./yb_build.sh release --sj --java-test 'org.yb.pgsql.TestYsqlUpgrade' --java-test-args '-Dysql_sys_catalog_snapshot_path=/tmp/yugabyte-2.20.2.0/share/initial_sys_catalog_snapshot' ./yb_build.sh release --sj --java-test 'org.yb.pgsql.TestYsqlUpgrade' --java-test-args '-Dysql_sys_catalog_snapshot_path=/tmp/yugabyte-2.14.17.0/share/initial_sys_catalog_snapshot' ./yb_build.sh release --sj --java-test 'org.yb.pgsql.TestYsqlUpgrade' --java-test-args '-Dysql_sys_catalog_snapshot_path=/tmp/yugabyte-2.18.8.0/share/initial_sys_catalog_snapshot' ./yb_build.sh release --sj --java-test 'org.yb.pgsql.TestYsqlUpgrade' --java-test-args '-Dysql_sys_catalog_snapshot_path=/tmp/yugabyte-2.20.4.1/share/initial_sys_catalog_snapshot' ./yb_build.sh release --sj --java-test 'org.yb.pgsql.TestYsqlUpgrade' --java-test-args '-Dysql_sys_catalog_snapshot_path=/tmp/yugabyte-2.18.3.0/share/initial_sys_catalog_snapshot' ./yb_build.sh release --sj --java-test 'org.yb.pgsql.TestYsqlUpgrade' --java-test-args '-Dysql_sys_catalog_snapshot_path=/tmp/yugabyte-2.21.1.0/share/initial_sys_catalog_snapshot' ./yb_build.sh release --sj --java-test 'org.yb.pgsql.TestYsqlUpgrade' --java-test-args '-Dysql_sys_catalog_snapshot_path=/tmp/yugabyte-2024.1.0.0/share/initial_sys_catalog_snapshot' Reviewers: yguan Reviewed By: yguan Subscribers: yql Differential Revision: https://phorge.dev.yugabyte.com/D36523
jasonyb
pushed a commit
that referenced
this issue
Jul 17, 2024
Summary: cc63aaf [docs] updates to diagnostics report page (#23162) Excluded: 1773ae2 [#22937] docdb: Backward scans: make pggate be aware of fast backward scan capability 39c6228 [PLAT-12732] toggle tls and cert rotation v2 apis d7cf125 [PLAT-14539][xCluster] need_bootstrap API does not work for old universes 65232ff [PLAT-14654]: Volume Size does not increase when ULTRA storage type is selected 4b39933 [PLAT-14606]: Disable options to ensure shrinking the RF is not permitted edit universe scenario 98d3fed [#23182] YSQL: Fix upgrade test failure when using 2.20.3.1 snapshot Excluded: 19ab966 [#22902] YSQL: Add support for dropping a key column 835e30d [#22479] docdb: Pass epoch through DB cloning calls 3273e9b [#21789] docdb: Add tablet splitting support for clone 52f7e79 [#23064] YSQL: pg_partman: disable p_retention_schema parameter 66ed3a5 [#23197] YSQL: pg_partman: Disable Gist index creation 3996f55 [YNP][PLAT-14664] make node register to provider idempotent Excluded: 6ec058d [PLAT-14668] - Move YSQL/YCQL configuration RBAC check to universe actions level and integrate RBAC for PG Compatibility Excluded: 18bb9b8 [#23034] YSQL: Add Support for OIDC IDP URL (jwt_jwks_url) to fetch and refresh JKWS 68cb1d2 [PLAT-14675][YNP] Fix the formatting for preflight checks db445ce [PLAT-14471][PLAT-14576] Added configurable deadline, keepAlive and unavailable retries to Ybc Java client Excluded: 47da28c [#23192] YSQL: Simplify/cleanup code in PgDml/PgSelect/PgSelectIndex etc Excluded: 1bc7a50 fix : gflag group toggle issue Test Plan: Jenkins: rebase: pg15-cherrypicks Reviewers: jason, tfoucher Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D36624
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jira Link: DB-12122
Description
The following upgrade test that uses a 2.20.3.1 initdb snapshot currently fails (commit 3455fb2):
./yb_build.sh release --java-test 'org.yb.pgsql.TestYsqlUpgrade#upgradeIsIdempotent' --java-test-args '-Dysql_sys_catalog_snapshot_path=/tmp/yugabyte-2.20.3.1/share/initial_sys_catalog_snapshot'
The error looks like some parser output related changes that are stored in pg_rewrite table.
Issue Type
kind/bug
Warning: Please confirm that this issue does not contain any sensitive information
The text was updated successfully, but these errors were encountered: