[YSQL][Query Diagnostics] Collect ASH data #22148
Assignees
Labels
area/ysql
Yugabyte SQL (YSQL)
kind/enhancement
This is an enhancement of an existing feature
priority/medium
Medium priority issue
Comments
IshanChhangani
added
area/ysql
yugabyte-ci
added
kind/enhancement
11 tasks
IshanChhangani
added a commit
that referenced
this issue
Sep 4, 2024
Summary: - This diff builds over 47f40bb and adds the active_session_history.csv file within the bundle output. - When `yb_ash_enable_infra` and `yb_enable_ash` are enabled, all ASH samples with timestamps within the bundle's duration are dumped to the active_session_history.csv file upon bundle completion - Note: active_session_history.csv contains data relating to all queryids and not just for the queryid that is being bundled. - We perform a binary search on the circular ASH buffer using the bundle's start_time and end_time to find the relevant indices for data dumping. active_session_history.csv looks like:- ``` sample_time root_request_id rpc_request_id wait_event_component wait_event_class wait_event top_level_node_id query_id pid client_node_ip wait_event_aux sample_weight wait_event_type ysql_dbid 2024-08-20 14:44:31.240577+05:30 00000000-0000-0000-0000-000000000000 12910 TServer Consensus Raft_WaitingForReplication 00000000-0000-0000-0000-000000000000 0 34750 e7bdd012d94744b 1.000000 Network 0 2024-08-20 14:44:34.05494+05:30 6f31b822-c206-e7c1-64ff-87c28406c825 0 YSQL TServerWait StorageRead fe0838c0-ba6f-4bad-897b-aa34fcec82bd -3472416937101760393 35329 127.0.0.1:56082 1.000000 Network 13254 2024-08-20 14:44:34.55692+05:30 37176a3a-c408-77ca-45e8-f6fed4524e29 0 YSQL TServerWait StorageRead fe0838c0-ba6f-4bad-897b-aa34fcec82bd -3472416937101760393 35329 127.0.0.1:56082 1.000000 Network 13254 2024-08-20 14:44:35.016742+05:30 bac30675-39ad-7f03-9f10-9a7e109b44b7 0 YSQL YSQLQuery QueryProcessing fe0838c0-ba6f-4bad-897b-aa34fcec82bd -3472416937101760393 35329 127.0.0.1:56082 1.000000 Cpu 13254 2024-08-20 14:44:35.023447+05:30 bac30675-39ad-7f03-9f10-9a7e109b44b7 0 YSQL YSQLQuery QueryProcessing fe0838c0-ba6f-4bad-897b-aa34fcec82bd -3472416937101760393 35329 127.0.0.1:56082 1.000000 Cpu 13254 2024-08-20 14:44:35.501807+05:30 1e937160-16d5-430d-040b-20978a612432 0 YSQL TServerWait StorageRead fe0838c0-ba6f-4bad-897b-aa34fcec82bd -3472416937101760393 35329 127.0.0.1:56082 1.000000 Network 13254 2024-08-20 14:44:35.508461+05:30 1e937160-16d5-430d-040b-20978a612432 0 YSQL TServerWait StorageRead fe0838c0-ba6f-4bad-897b-aa34fcec82bd -3472416937101760393 35329 127.0.0.1:56082 1.000000 Network 13254 2024-08-20 14:44:35.967882+05:30 0ac4f4a6-8409-fccd-d306-db3e3b582406 0 YSQL TServerWait StorageRead fe0838c0-ba6f-4bad-897b-aa34fcec82bd -3472416937101760393 35329 127.0.0.1:56082 1.000000 Network 13254 ``` Note: this diff also fixes `YB_LAST_USED_OID` value from 8069 to 8070. This change was missed in `D36656` Jira: DB-11076 Test Plan: ./yb_build.sh --java-test TestYsqlUpgrade ./yb_build.sh --java-test TestYbQueryDiagnostics#checkSchemaDetails Reviewers: asaha, amartsinchyk, hbhanawat, telgersma Reviewed By: telgersma Subscribers: svc_phabricator, yql Differential Revision: https://phorge.dev.yugabyte.com/D36272
jasonyb
pushed a commit
that referenced
this issue
Sep 5, 2024
Summary: f8049b8 [doc][xcluster][2024.1.2] Semi-automatic transactional xCluster (#23710) f2c8470 [docs][yugabyted][2024.1.2] Add Read Replica and xCluster examples (#23289) ea3157c [doc][yba][2024.1.2] Export database audit logs (#23605) Excluded: 8bae488 [#18822] YSQL: Write out unmodified cols to main table when update optimizations are enabled 5012632 [PLAT-15152] Upgrade YBC client and server version to 2.2.0.0-b5 cccb1e1 Fix YCQL index table workflow in docs (#23774) 9a3f5f7 [PLAT-15028] yba installer simplify postgres initdb a2b5e72 [docs] Release notes for 2024.1.2.0 (#23679) 9a9690d Fix 2024.1.2 build number (#23779) aec361b [doc] Voyager 1.8 (#23764) a2540e0 [DEVOPS-3185, DEVOPS-3114] Bump up frozen pip modules to latest versions compatible with py3.8 88f23dd [docs] [xcluster] Add min version for semi-automatic mode (#23776) 1d61e67 Revise the entire assessment page (#23784) 142c04a [PLAT-15089] HA sync immediately after promotion 315110f [PLAT-15132][PLAT-15153] Allow users to configure logging level for YNP, also minor bug fixes 82ff8d1 [PLAT-6779] Handle relative paths in yb_platform_backup.sh Excluded: 45c9cf8 [#23263] YSQL, ASH: Instrument more wait states Excluded: 9f8acff [#22148] YSQL, QueryDiagnostics: Adding support for Ash data. 39670e8 download links (#23790) 2674a79 [#23780] YSQL: Modify catalog version mismatch test assertions with Connection Manager enabled 3eb31b8 [#23777] yugabyted: update the gflags of pg parity to remove sized based fetching and add bitmap scans. fc5accd [docs] Enclose `allowed_preview_flags_csv` CSV parameters in brackets (#23758) Test Plan: Jenkins: rebase: pg15-cherrypicks Reviewers: jason, tfoucher Differential Revision: https://phorge.dev.yugabyte.com/D37772
IshanChhangani
added a commit
that referenced
this issue
Sep 11, 2024
…pport for Ash data.
Summary:
- pgstat.c
- pgstat_get_wait_activity().
- YB 9f8acff removes redundant checking of `isYugabyteEndabled()`
- This File has been removed in pg15 a333476b925134f6185037eaff3424c07a9f466f, and the code for getting wait_event types was shifted to wait_event.c
- Moved the same changes from pgstat.c (YB) to wait_event.c (pg15)
- yb_query_diagnostics.c
- DumpToFile()
- YB 9f8acff adjacent lines were changed
- YB-PG15 602cb2d adds extra parameter of offset 0 to FileWrite
- Accepted combination of else statement as well as extra parameter to FileWrite
old summary:
- This diff builds over 47f40bb and adds the active_session_history.csv file within the bundle output.
- When `yb_ash_enable_infra` and `yb_enable_ash` are enabled, all ASH samples with timestamps within the bundle's duration are dumped to the active_session_history.csv file upon bundle completion
- Note: active_session_history.csv contains data relating to all queryids and not just for the queryid that is being bundled.
- We perform a binary search on the circular ASH buffer using the bundle's start_time and end_time to find the relevant indices for data dumping.
active_session_history.csv looks like:-
```
sample_time root_request_id rpc_request_id wait_event_component wait_event_class wait_event top_level_node_id query_id pid client_node_ip wait_event_aux sample_weight wait_event_type ysql_dbid
2024-08-20 14:44:31.240577+05:30 00000000-0000-0000-0000-000000000000 12910 TServer Consensus Raft_WaitingForReplication 00000000-0000-0000-0000-000000000000 0 34750 e7bdd012d94744b 1.000000 Network 0
2024-08-20 14:44:34.05494+05:30 6f31b822-c206-e7c1-64ff-87c28406c825 0 YSQL TServerWait StorageRead fe0838c0-ba6f-4bad-897b-aa34fcec82bd -3472416937101760393 35329 127.0.0.1:56082 1.000000 Network 13254
2024-08-20 14:44:34.55692+05:30 37176a3a-c408-77ca-45e8-f6fed4524e29 0 YSQL TServerWait StorageRead fe0838c0-ba6f-4bad-897b-aa34fcec82bd -3472416937101760393 35329 127.0.0.1:56082 1.000000 Network 13254
2024-08-20 14:44:35.016742+05:30 bac30675-39ad-7f03-9f10-9a7e109b44b7 0 YSQL YSQLQuery QueryProcessing fe0838c0-ba6f-4bad-897b-aa34fcec82bd -3472416937101760393 35329 127.0.0.1:56082 1.000000 Cpu 13254
2024-08-20 14:44:35.023447+05:30 bac30675-39ad-7f03-9f10-9a7e109b44b7 0 YSQL YSQLQuery QueryProcessing fe0838c0-ba6f-4bad-897b-aa34fcec82bd -3472416937101760393 35329 127.0.0.1:56082 1.000000 Cpu 13254
2024-08-20 14:44:35.501807+05:30 1e937160-16d5-430d-040b-20978a612432 0 YSQL TServerWait StorageRead fe0838c0-ba6f-4bad-897b-aa34fcec82bd -3472416937101760393 35329 127.0.0.1:56082 1.000000 Network 13254
2024-08-20 14:44:35.508461+05:30 1e937160-16d5-430d-040b-20978a612432 0 YSQL TServerWait StorageRead fe0838c0-ba6f-4bad-897b-aa34fcec82bd -3472416937101760393 35329 127.0.0.1:56082 1.000000 Network 13254
2024-08-20 14:44:35.967882+05:30 0ac4f4a6-8409-fccd-d306-db3e3b582406 0 YSQL TServerWait StorageRead fe0838c0-ba6f-4bad-897b-aa34fcec82bd -3472416937101760393 35329 127.0.0.1:56082 1.000000 Network 13254
```
Note: this diff also fixes `YB_LAST_USED_OID` value from 8069 to 8070. This change was missed in `D36656`
Jira: DB-11076
Original commit: 9f8acff / D36272
Test Plan:
./yb_build.sh --java-test TestYsqlUpgrade
./yb_build.sh --java-test TestYbQueryDiagnostics#checkSchemaDetails
Reviewers: jason, tfoucher
Reviewed By: jason
Subscribers: yql, svc_phabricator
Differential Revision: https://phorge.dev.yugabyte.com/D37879
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jira Link: DB-11076
Description
This issue focuses on adding functionality to extract and integrate data from ASH into the query diagnostics framework. The extracted data will be specific to a particular query ID and the duration during which the query diagnostics bundle was active
Issue Type
kind/enhancement
Warning: Please confirm that this issue does not contain any sensitive information
The text was updated successfully, but these errors were encountered: