Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Query execution tracing and replay tool #9668

Open
xiaoxmeng opened this issue Apr 30, 2024 · 6 comments
Open

Query execution tracing and replay tool #9668

xiaoxmeng opened this issue Apr 30, 2024 · 6 comments
Assignees
@duanmeng
Labels
enhancement New feature or request

Comments

@xiaoxmeng
Copy link
Contributor

xiaoxmeng commented Apr 30, 2024 •
edited by mbasmanova
Loading

Description

Add query execution tracing and replay tool to facilitate query analysis. The tool shall allow us to replay a part of query execution on a local computer instead of replaying the whole query in a production environment or in a real Prestissimo cluster. The tool consists two parts:

(1) trace collection: run a query with trace collection enabled through query configs (and the corresponding session properties in Prestissimo context). The query execution will collect the trace by dumping the input vectors of a particular set of specified operators (data) and the corresponding query plan info (meta data) into a specified storage location;

(2) trace replay: constructs the a sub-query plan using the dumped query plan meta, and then load the dumped input vectors into memory and feed into the constructed sub-query plan for replay. If the input is too large, then we can build a special source operator to read the dumped input vector from storage in batches.

The replay can be done at different level: operator level, pipeline level and task level. We can start with the operator level and extend to pipeline and task level next.

cc @mbasmanova @duanmeng @huamn
@xiaoxmeng xiaoxmeng added the enhancement New feature or request label Apr 30, 2024
@duanmeng duanmeng self-assigned this Apr 30, 2024
@mbasmanova
Copy link
Contributor

CC: @aditi-pandit

@aditi-pandit
Copy link
Collaborator

+1. This would be really useful.

@FelixYBW
Copy link
Contributor

Similar as Gluten's microbenchmark reproduce tool. It will be super useful for debug and performance analysis. Good feature!

@duanmeng duanmeng mentioned this issue Aug 18, 2024
Closed
facebook-github-bot pushed a commit that referenced this issue Aug 21, 2024
@duanmeng @facebook-github-bot
Add Query Trace Writers and Readers (#10774)
55107b6 
Summary:
Add a query tracer to log the input data, and metadata (including query configurations,
connector properties, and query plans). This logged data and metadata can be used to
replay the operations of a specific operator or pipeline.

Part of #9668

Pull Request resolved: #10774

Reviewed By: Yuhta

Differential Revision: D61514971

Pulled By: xiaoxmeng

fbshipit-source-id: 9a0b901ee1475a6c35169fe77eb19e797e31e210
@duanmeng duanmeng mentioned this issue Aug 25, 2024
Closed
facebook-github-bot pushed a commit that referenced this issue Aug 27, 2024
@duanmeng @facebook-github-bot
Trace metadata during task creation (#10815)
763c19c 
Summary:
Create a directory named `$QueryTraceBaseDir/$taskId` when a task is initiated,
if query tracing is enabled. This directory will store metadata related to the task,
including the query plan node tree, query configurations, and connector properties.

Part of #9668

Pull Request resolved: #10815

Reviewed By: Yuhta

Differential Revision: D61808438

Pulled By: xiaoxmeng

fbshipit-source-id: 57eff8f4b70405ba5c60fcd8315b025b22c2317b
This was referenced Aug 31, 2024
Closed
Closed
Joe-Abraham pushed a commit to Joe-Abraham/velox that referenced this issue Sep 2, 2024
@duanmeng @Joe-Abraham
Trace metadata during task creation (facebookincubator#10815)
bdbc64b 
Summary:
Create a directory named `$QueryTraceBaseDir/$taskId` when a task is initiated,
if query tracing is enabled. This directory will store metadata related to the task,
including the query plan node tree, query configurations, and connector properties.

Part of facebookincubator#9668

Pull Request resolved: facebookincubator#10815

Reviewed By: Yuhta

Differential Revision: D61808438

Pulled By: xiaoxmeng

fbshipit-source-id: 57eff8f4b70405ba5c60fcd8315b025b22c2317b
Joe-Abraham pushed a commit to Joe-Abraham/velox that referenced this issue Sep 3, 2024
@duanmeng @Joe-Abraham
Trace metadata during task creation (facebookincubator#10815)
14d7200 
Summary:
Create a directory named `$QueryTraceBaseDir/$taskId` when a task is initiated,
if query tracing is enabled. This directory will store metadata related to the task,
including the query plan node tree, query configurations, and connector properties.

Part of facebookincubator#9668

Pull Request resolved: facebookincubator#10815

Reviewed By: Yuhta

Differential Revision: D61808438

Pulled By: xiaoxmeng

fbshipit-source-id: 57eff8f4b70405ba5c60fcd8315b025b22c2317b
@xiaodouchen
Copy link
Contributor

This tool looks great, but I have two concerns:

  1. Currently, the trace tool saves the full input data. If the data is too large, it may cause the node to crash in test or production environments.
  2. There's no mechanism to delete trace data. If trace data is generated on multiple nodes, it needs to be manually deleted from each node.

I suggest adding a config query_trace_input_count to control the number of saved input data. When set to n (n>0), it means to save n inputs. When n is -1, it means to delete the trace file. Regarding deletion, since each query has a different taskId, it would be necessary to asynchronously delete the query_trace_dir during QueryTrace initialization. Moreover, for the same query_trace_dir, the generation and deletion of trace files should not occur concurrently to avoid potential conflicts or issues.

@duanmeng @xiaoxmeng What are your thoughts on this suggestion? I look forward to hearing your suggestions.

@duanmeng
Copy link
Collaborator

This tool looks great, but I have two concerns:

@xiaodouchen Thanks for your review.

  1. Currently, the trace tool saves the full input data. If the data is too large, it may cause the node to crash in test or production environments.
  • The data can be logged on remote storage.
  • We can extend it later to support only trace from a few nodes or a few tasks.
  1. There's no mechanism to delete trace data. If trace data is generated on multiple nodes, it needs to be manually deleted from each node.

Garbage collection is a different thing.

cc @xiaoxmeng

@duanmeng
Copy link
Collaborator

duanmeng commented Sep 14, 2024 •
edited
Loading

  • Add query trace writer and reader.
  • Trace metadata during task creation.
  • Add query replayer.
  • Query trace source and sink nodes/operators.
  • Support table writer replay.
  • Support query trace splits.
  • Support table scan replay.
  • Support aggregate replay.

@xiaodouchen xiaodouchen mentioned this issue Sep 18, 2024
Open
facebook-github-bot pushed a commit that referenced this issue Sep 24, 2024
@duanmeng @facebook-github-bot
Add query replayer (#10897)
cc46d81 
Summary:
Velox can record the query metadata (query plan and configs)
during task creation and input vectors of the traced operator,
see #10774 and #10815.

This PR adds a query replayer, it can be used to replay a query locally
using the metadata and input vectors from the production environment.
It supports showing the summary of a query at present, and more traced
operators' replaying supports will be added in the future.

Also, this PR adds two query configs `query_trace_max_bytes` and
`query_trace_task_reg_exp` to constraint the record input data size
and trace tasks respectively to ensure the stability of the cluster
in the prod.

Part of #9668

Pull Request resolved: #10897

Reviewed By: tanjialiang

Differential Revision: D62336733

Pulled By: xiaoxmeng

fbshipit-source-id: d196738dfa92c29fe5de67a944f652a328903814
facebook-github-bot pushed a commit that referenced this issue Sep 26, 2024
@duanmeng @facebook-github-bot
Add trace support for TableWriter (#10910)
3fe248f 
Summary:
Create a `QueryDataWriter` in `exec::TableWriter` if the query trace is enabled,
recording the input vectors batch by batch. Each operator writes its data to the
directory `$rootDir/$pipelineId/$driverId/data`. The recorded data will be used
to replay the execution of `exec::TableWriter`, which will be supported in the
follow-up.

Design notes: https://docs.google.com/document/d/1crIIeVz4tWKYQnBoHoxrv2i-4zAML9HSYLps8h5SDrc/edit#heading=h.y6j2ojtr7hm9

Part of #9668

Pull Request resolved: #10910

Reviewed By: pedroerp

Differential Revision: D63444416

Pulled By: xiaoxmeng

fbshipit-source-id: ddd74ff6dd56de7bce31ec536035b32211453364
@duanmeng duanmeng mentioned this issue Oct 3, 2024
Closed
facebook-github-bot pushed a commit that referenced this issue Oct 5, 2024
@duanmeng @facebook-github-bot
Add TableWriterRepalyer (#11100)
a933331 
Summary:
Adds `TableWriterReplayer` to facilitate the replaying of `TableWriter` operator.
Uses the given plan node ID to find the traced `TableWriteNode` from the traced plan.
It helps create a new `TableWriterNode` and rebuild a query plan with a `QueryTraceScanNode`,
then apply the traced configurations, and rerun.

`QueryTraceScanNode` holds the traced data type and dir for a given plan node ID.
These information can be utilized to build the `QueryTraceScan` operator. It creates
a `QueryDataReader` using the traced data type and input data file. To find the right
input data file for replaying, we need to use both the pipeline ID and driver ID, which
are only known during operator creation, so we need to figure out the input traced
data file and the output type dynamically.

Part of #9668

Pull Request resolved: #11100

Reviewed By: tanjialiang

Differential Revision: D63774083

Pulled By: xiaoxmeng

fbshipit-source-id: 912bef3cb20d9b1a1685af625ba2f319e2dc7509
This was referenced Oct 7, 2024
Open
Closed
Closed
facebook-github-bot pushed a commit that referenced this issue Oct 8, 2024
@duanmeng @facebook-github-bot
Add HashAggregation Replayer (#11176)
74c2133 
Summary:
Records input in `HashAggregation` and AggregationReplayer to support the replaying.

part of #9668

Pull Request resolved: #11176

Reviewed By: tanjialiang

Differential Revision: D64017836

Pulled By: xiaoxmeng

fbshipit-source-id: 4392e511fb889dfc232eaf64c7228655c50d623f
facebook-github-bot pushed a commit that referenced this issue Oct 8, 2024
@duanmeng @facebook-github-bot
Add HiveConnectorSplit Serde (#11184)
2a42006 
Summary:
Add HiveConnectorSplit Serde, a prerequisite for supporting `TableScan` tracing.

part of #9668

Pull Request resolved: #11184

Reviewed By: tanjialiang

Differential Revision: D64019689

Pulled By: xiaoxmeng

fbshipit-source-id: 22b89d8415d218f9bd3d4f7e589a0be276406fec
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@duanmeng duanmeng

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@duanmeng @mbasmanova @FelixYBW @aditi-pandit @xiaoxmeng @xiaodouchen