[YSQL][Query Diagnostics] Establish Infrastructure for yb_query_diagnostics #22144
Assignees
Labels
area/ysql
Yugabyte SQL (YSQL)
kind/enhancement
This is an enhancement of an existing feature
priority/medium
Medium priority issue
Comments
IshanChhangani
added
area/ysql
yugabyte-ci
added
kind/enhancement
IshanChhangani
changed the title
11 tasks
IshanChhangani
added a commit
that referenced
this issue
Jul 12, 2024
…nostics Summary: This diff includes the code for setting up Query Diagnostics infrastructure and adds bind variables functionality. **`Basic architecture`**:- We use a shared hash table to store data bundles. - Each entry consists of the key, which is the query ID. - The value contains all related information such as PGSS, ASH, query ID string, bind variables, schema details, and explain plans. (In this diff, we only add bind variables) - Each QueryDiagnostics Bundle is represented by an entry within the hash table. We have installed executorEnd hook to get bind_variables and other data from the `QueryDesc`. Within `QueryDesc`, the params pointer contains the bind variables as a list, allowing us to iterate and extract them as needed. For stopping the bundle, we use a background worker. This background worker iterates over the hash table and checks for the expired entries based on their `diagnostics_interval_sec` parameter. It then flushes all data related to these entries to the filesystem and removes them from the shared hash table. **`Directory structure of query diagnostics`** :- We create a folder named query-diagnostics within the pg_data directory. A corresponding folder is created for each bundle and named after its query_id. Inside each query_id folder is a timestamped folder containing the file bind_vars.csv. This diff does not handle process utility statements. Jira: DB-11072, DB-11073 Test Plan: ./yb_build.sh --java-test TestYsqlUpgrade ./yb_build.sh --java-test TestYbQueryDiagnostics Reviewers: asaha, hbhanawat, amartsinchyk Reviewed By: amartsinchyk Subscribers: ybase, yql Differential Revision: https://phorge.dev.yugabyte.com/D34903
jasonyb
pushed a commit
that referenced
this issue
Jul 17, 2024
Summary: de8c392 [#22923] YSQL: In EXPLAIN command, fix relids of prefix keys when under a subquery distinct index scan. a137121 [#22349] yugabyted: xCluster Integration with yugabyted. 040071d note about Jumpcloud support (#23193) d1572d9 [PLAT-13410] Allow YCQL index table selection Excluded: 47f40bb [#22144, #22145] YSQL, QueryDiagnostics: Infrastructure for QueryDiagnostics c51e6c9 [PLAT-14519] Master failover must not fail when some tservers are down c319a56 [#23100] YSQL: drop column does not complete in docdb when alter schema rpc is missed 26eaeaa [#23071] yugabyted: Add tests for yugabyted cluster creation. 160f24e [PLAT-14162] Support bootstrapping for editing list of databases in DR replicatoin a65795e [doc][yba] Ports and encryption (#23152) 2ca9deb [#23079] DocDB: Small cleanups to stay closer to pg15-upgrade branch cba0254 [PLAT-14655] Fix ephemeral upload directory on container installs 6e99ea0 [PLAT-14546] Ansible task to replace CentOS 7 repo URLs a9f675e [PLAT-14570] Deploy YBC build based on the architecture of destination node d56446f [docs] Updated OS tables (#23156) 019ab00 [#23055] docdb: Add in-progress restore/deletion checks to DeleteSnapshot and RestoreSnapshot. 9aed20a [PLAT-14248][PLAT-14249] - feat : Enable/Disable Enhanced PG Compatibility 79888cf [#20482] YSQL : Allow updating follower reads in a transaction block a946721 [#23122] YSQL: Add pg_partman to build e5aea11 [PLAT-11188] Handle a node instance without node name set properly in health checks 04cf65d [#13549] YSQL: Disable ACCESS EXCLUSIVE LOCK in pg_partman ed1809c [#23123] YSQL: Disable other partition methods except native in pg_partman bc55a46 [#23125] YSQL: Disable pg_partman background worker e1ef208 [#23124] YSQL: Disable advisory locks usage in pg_partman - explain.c: - yb_fix_indexpr_mutator: master de8c392 adds this function. Change varnoold to varnosyn in accordance with upstream PG 9ce77d75c5ab094637cc4a446296dc3be6e3c221. Test Plan: Jenkins: rebase: pg15-cherrypicks, urgent Reviewers: jason, tfoucher Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D36576
IshanChhangani
added a commit
that referenced
this issue
Jul 19, 2024
…nfrastructure for QueryDiagnostics Summary: Original commit: 47f40bb / D34903 The following conflicts were resolved: - ipci.c - CreateSharedMemoryAndSemaphores() - upstream PG 0bd305ee1d427ef29f5fa4fa20567e3b3f5ff792 moves code into CalculateShmemSize. YB 47f40bb adds code there. Move that code to the new location. Note also upstream PG 4f2400cb3f10aa79f99fba680c198237da28dd38 modifying an adjacent line. - pgstat.c - pgstat_get_wait_activity(). - This File has been removed in pg15 a333476b925134f6185037eaff3424c07a9f466f, and the code for `WaitEventActivity` was shifted to wait_event.h - YB 47f40bb adds `YB_WAIT_EVENT_QUERY_DIAGNOSTICS_MAIN` to pgstat.c - Moved wait_state `YB_WAIT_EVENT_QUERY_DIAGNOSTICS_MAIN` to wait_event.c - pgstat.h - ifdef YB_TODO - master had several structs relating to wait_events, which have been shifted to wait_event.h in PG a333476b925134f6185037eaff3424c07a9f466f as mentioned above. - MakeFile (backend/utils/misc) - OBJS . - PG 01368e5d9da77099b38aac527b01b85cc7869b25 : OBJS format has been changed to one entry per line. - YB 47f40bb: Adds yb_query_diagnostics.o to OBJS. Trivial change - yb_query_diagnostics.c - DumpToFile() - PG 15 c24dcd0cfd949bdf245814c4c2b3df828ee7db36 commit adds another parameter to the FileWrite() function. - YB 47f40bb adds the FileWrite() function. - Added offset as 0. Summary of the original diff: ``` This diff includes the code for setting up Query Diagnostics infrastructure and adds bind variables functionality. **`Basic architecture`**:- We use a shared hash table to store data bundles. - Each entry consists of the key, which is the query ID. - The value contains all related information such as PGSS, ASH, query ID string, bind variables, schema details, and explain plans. (In this diff, we only add bind variables) - Each QueryDiagnostics Bundle is represented by an entry within the hash table. We have installed executorEnd hook to get bind_variables and other data from the `QueryDesc`. Within `QueryDesc`, the params pointer contains the bind variables as a list, allowing us to iterate and extract them as needed. For stopping the bundle, we use a background worker. This background worker iterates over the hash table and checks for the expired entries based on their `diagnostics_interval_sec` parameter. It then flushes all data related to these entries to the filesystem and removes them from the shared hash table. **`Directory structure of query diagnostics`** :- We create a folder named query-diagnostics within the pg_data directory. A corresponding folder is created for each bundle and named after its query_id. Inside each query_id folder is a timestamped folder containing the file bind_vars.csv. This diff does not handle process utility statements. ``` Jira: DB-11072, DB-11073 Test Plan: ./yb_build.sh --java-test TestYsqlUpgrade ./yb_build.sh --java-test TestYbQueryDiagnostics Reviewers: jason, tfoucher Reviewed By: jason Subscribers: yql, ybase Differential Revision: https://phorge.dev.yugabyte.com/D36654
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jira Link: DB-11072
Description
Issue Type
kind/enhancement
Warning: Please confirm that this issue does not contain any sensitive information
The text was updated successfully, but these errors were encountered: