Add anonymized query plan in json format to QueryCompletedEvent #12968

gaurav8297 · 2022-06-24T00:57:27Z

Description

Add anonymized query plan in json format to QueryCompletedEvent
Implement EXPLAIN (TYPE DISTRIBUTED, FORMAT JSON)

Is this change a fix, improvement, new feature, refactoring, or other?

New feature

Is this a change to the core query engine, a connector, client library, or the SPI interfaces? (be specific)

Event listener SPI, EXPLAIN

How would you describe this change to a non-technical end user or system administrator?

Provides anonymised query plan in json format to event listener to enable offline analysis without leaking sensitive info.
Implements EXPLAIN (TYPE DISTRIBUTED, FORMAT JSON)

Documentation

( ) No documentation is needed.
(x) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.

Release notes

( ) No release notes entries required.
(x) Release notes entries required with the following suggested text:

# General
* Implement EXPLAIN (TYPE DISTRIBUTED, FORMAT JSON). ({issue}`12968`)
* Add anonymized query plan in json format to QueryCompletedEvent. ({issue}`12968`)

raunaqmorarka

Please add examples of anonymised plans to the PR

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/PlanPrinter.java

core/trino-main/src/main/java/io/trino/sql/planner/SystemPartitioningHandle.java

core/trino-main/src/main/java/io/trino/event/QueryMonitor.java

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/anonymize/Anonymizer.java

core/trino-main/src/main/java/io/trino/execution/scheduler/SqlQueryScheduler.java

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/TableInfoSupplier.java

core/trino-main/src/main/java/io/trino/execution/scheduler/SqlQueryScheduler.java

core/trino-main/src/main/java/io/trino/execution/TableInfo.java

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/PlanPrinter.java

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/anonymize/Anonymizer.java

.../trino-main/src/main/java/io/trino/sql/planner/planprinter/anonymize/HashCodeAnonymizer.java

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/anonymize/Anonymizer.java

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/PlanPrinter.java

raunaqmorarka · 2022-07-04T06:32:16Z

Please add examples of anonymised plans to the PR

ping

raunaqmorarka · 2022-07-04T06:32:32Z

Test failures are related, PTAL

core/trino-main/src/main/java/io/trino/sql/analyzer/QueryExplainer.java

gaurav8297 · 2022-07-05T06:16:41Z

Please add examples of anonymised plans to the PR

Query: SELECT quantity FROM lineitem LIMIT 10

{
  "0" : {
    "id" : "6",
    "name" : "Output",
    "descriptor" : {
      "columnNames" : "[column_717400177]"
    },
    "outputs" : [ {
      "symbol" : "symbol_717400177",
      "type" : "double"
    } ],
    "details" : [ ],
    "estimates" : [ ],
    "children" : [ {
      "id" : "98",
      "name" : "Limit",
      "descriptor" : {
        "count" : "10",
        "withTies" : "",
        "inputPreSortedBy" : ""
      },
      "outputs" : [ {
        "symbol" : "symbol_717400177",
        "type" : "double"
      } ],
      "details" : [ ],
      "estimates" : [ ],
      "children" : [ {
        "id" : "171",
        "name" : "LocalExchange",
        "descriptor" : {
          "connectorHandleType" : "SystemPartitioningHandle",
          "partitioning" : "SINGLE",
          "function" : "SINGLE",
          "isReplicateNullsAndAny" : "",
          "hashColumn" : "",
          "arguments" : ""
        },
        "outputs" : [ {
          "symbol" : "symbol_717400177",
          "type" : "double"
        } ],
        "details" : [ ],
        "estimates" : [ ],
        "children" : [ {
          "id" : "138",
          "name" : "RemoteSource",
          "descriptor" : {
            "sourceFragmentIds" : "[1]"
          },
          "outputs" : [ {
            "symbol" : "symbol_717400177",
            "type" : "double"
          } ],
          "details" : [ ],
          "estimates" : [ ],
          "children" : [ ]
        } ]
      } ]
    } ]
  },
  "1" : {
    "id" : "137",
    "name" : "LimitPartial",
    "descriptor" : {
      "count" : "10",
      "withTies" : "",
      "inputPreSortedBy" : ""
    },
    "outputs" : [ {
      "symbol" : "symbol_717400177",
      "type" : "double"
    } ],
    "details" : [ ],
    "estimates" : [ ],
    "children" : [ {
      "id" : "0",
      "name" : "TableScan",
      "descriptor" : {
        "connector" : "tpch",
        "table" : "catalog_225088199.schema_225081830.table_3191690477"
      },
      "outputs" : [ {
        "symbol" : "symbol_717400177",
        "type" : "double"
      } ],
      "details" : [ "symbol_717400177 := column_2086283230" ],
      "estimates" : [ ],
      "children" : [ ]
    } ]
  }
}

core/trino-main/src/test/java/io/trino/sql/planner/planprinter/TestJsonRepresentation.java

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/PlanPrinter.java

.../trino-main/src/main/java/io/trino/sql/planner/planprinter/anonymize/HashCodeAnonymizer.java

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/PlanPrinter.java

...ino-main/src/test/java/io/trino/sql/planner/planprinter/TestAnonymizeJsonRepresentation.java

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/anonymize/NoOpAnonymizer.java

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/PlanPrinter.java

core/trino-main/src/main/java/io/trino/event/QueryMonitor.java

sopel39 · 2022-07-06T14:31:49Z

core/trino-main/src/main/java/io/trino/execution/scheduler/SqlQueryScheduler.java

+            Optional<String> connectorName = metadata.listCatalogs(session).stream()
+                    .filter(catalogInfo -> catalogInfo.getCatalogName().equals(tableSchema.getCatalogName()))
+                    .map(CatalogInfo::getConnectorName)
+                    .findFirst();


this should always exists, right?

Ideally yes, but I don't know if we should assume that. In future if catalogs can be removed on the fly, I'm not sure if that assumption would continue to hold.

In future if catalogs can be removed on the fly, I'm not sure if that assumption would continue to hold.

I don't think they can be removed for active query

this should always exists, right?

I tried making it not optional, but there were many tests which started failing. IIRC these errors were coming in the case of an information schema.

Ok. Could you tell why for information schema connector name is missing?

core/trino-main/src/main/java/io/trino/execution/TableInfo.java

martint · 2022-06-29T16:49:56Z

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/anonymize/Anonymizer.java

+import static io.trino.sql.planner.plan.StatisticsWriterNode.WriteStatisticsTarget;
+import static io.trino.sql.planner.plan.TableWriterNode.WriterTarget;
+
+public interface Anonymizer


Why is this an interface? There should only be one implementation of the anonymizer, so an interface is overkill.

One implementation performs anonymisation, the other doesn't. Could you take a look at its usage in PlanPrinter and suggest a different way if this can be improved ?

martint · 2022-07-06T16:44:05Z

core/trino-spi/src/main/java/io/trino/spi/eventlistener/QueryMetadata.java

@@ -56,6 +57,7 @@ public QueryMetadata(
            List<RoutineInfo> routines,
            URI uri,
            Optional<String> plan,
+            Optional<String> anonymizedJsonPlan,


What's the use case for including both the regular plan and the anonymized plan? Most event listeners will just forward and store the event as is, which defeats the purpose of having an anonymized plan.

It is possible to have different use cases for regular plan and anonymised plan. E.g. Regular plan can be used to display query plan for past queries to a user in a UI. This would be user specific data that might not be made easily accessible to everyone, it's also formatted in a way that is not ideal for offline analysis.
Anonymized plan could be sent to a different downstream system which is more widely accessible and easier to work with for offline analysis.
Currently we send same event to all event listeners, so we also can't use different event listeners to get different versions of plan.

sopel39 · 2022-07-06T14:36:34Z

core/trino-main/src/main/java/io/trino/sql/analyzer/QueryExplainer.java

@@ -48,6 +48,7 @@
 import static io.trino.sql.analyzer.QueryType.EXPLAIN;


nit: should it be separate PR?

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/anonymize/Anonymizer.java

gaurav8297 · 2022-07-08T12:20:58Z

@raunaqmorarka @sopel39 PTAL

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/HashCodeAnonymizer.java

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/PlanPrinter.java

raunaqmorarka

lgtm
will wait to see if @martint or @sopel39 have further comments

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/HashCodeAnonymizer.java

sopel39 · 2022-07-11T11:50:13Z

core/trino-main/src/main/java/io/trino/execution/scheduler/SqlQueryScheduler.java

+            Optional<String> connectorName = metadata.listCatalogs(session).stream()
+                    .filter(catalogInfo -> catalogInfo.getCatalogName().equals(tableSchema.getCatalogName()))
+                    .map(CatalogInfo::getConnectorName)
+                    .findFirst();


Ok. Could you tell why for information schema connector name is missing?

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/Anonymizer.java

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/HashCodeAnonymizer.java

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/HashingAnonymizer.java

gaurav8297 · 2022-07-12T09:26:08Z

Could you tell why for information schema connector name is missing?

For some reason, We don't register the system or information schema's catalog name to CatalogManager, which is used during listCatalogs. That's why internal catalog names are missing from listCatalogs results.

https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/connector/ConnectorManager.java#L255

PS: Github is not letting me add a comment in the open thread 😞

cc @sopel39

sopel39

lgtm % comments

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/JsonRenderer.java

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/HashingAnonymizer.java

...ino-main/src/test/java/io/trino/sql/planner/planprinter/TestAnonymizeJsonRepresentation.java

sopel39 · 2022-07-15T10:13:44Z

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/PlanPrinter.java

        if (replicateNullsAndAny) {
            builder.append(format("Output partitioning: %s (replicate nulls and any) [%s]%s\n",
                    partitioningScheme.getPartitioning().getHandle(),
                    Joiner.on(", ").join(arguments),
-                    formatHash(partitioningScheme.getHashColumn())));


why not keep using formatHash method here?

We've made formatHash to be a non-static method

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/PlanPrinter.java

sopel39 · 2022-07-15T10:22:52Z

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/PlanPrinter.java

    }

-    public static String formatAggregation(Aggregation aggregation)
+    public static String formatAggregation(Anonymizer anonymizer, Aggregation aggregation)


nit: consider making it non static so that you don't have to pass anonymizer

It'll be redundant to make formatAggregation static because it is being used in GraphVizPlanPrinter too.

Anyways in PlanPrinter, formatAggregation is only used at 2-3 places.

sopel39 · 2022-07-15T10:27:44Z

Is failure related?

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/PlanPrinter.java

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/HashingAnonymizer.java

core/trino-main/src/test/java/io/trino/sql/planner/planprinter/TestExpressionAnonymization.java

core/trino-main/src/test/java/io/trino/sql/planner/planprinter/TestJsonRepresentation.java

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/HashingAnonymizer.java

gaurav8297 · 2022-08-05T00:34:41Z

@martint @raunaqmorarka @sopel39 Implemented counter-based anonymization

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/JsonRenderer.java

core/trino-main/src/main/java/io/trino/eventlistener/EventListenerManager.java

core/trino-main/src/main/java/io/trino/testing/TestingEventListenerManager.java

core/trino-spi/src/main/java/io/trino/spi/eventlistener/EventListener.java

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/CounterBasedAnonymizer.java

martint · 2022-08-16T18:56:27Z

core/trino-main/src/test/java/io/trino/sql/planner/planprinter/TestCounterBasedAnonymizer.java

+        };
+    }
+
+    private static class TestLiteral


What's the purpose of this?

The AST is meant to be a closed hierarchy and not to be extended by third-parties. There's a lot of infrastructure that depends on knowing exactly what classes are part of the AST. Eventually, we'll update it to use Java 17's sealed types and this won't be possible to do at all and enforced at compile time.

The idea here is to test the scenario that a new sub-class of Literal is added but CounterBasedAnonymizer#anonymizeExpression isn't updated to handle that new class. In this scenario, we should still anonymise the literal.
It's not strictly necessary though, let me know if you want this removed or handled differently.

Yes, let’s remove it. It’s not a correct usage of the AST classes and will break in the future.

As long as the anonymizer fails if a new class is added (very unlikely), we’ll be able to catch that very quickly and add the relevant code, so we don’t even have to handle anonymization for the general case.

I've removed this now and modified CounterBasedAnonymizer#anonymizeExpression to throw UnsupportedOperationException in case of un-handled Literal sub-class.

martint · 2022-08-19T18:23:44Z

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/CounterBasedAnonymizer.java

+        return anonymizeExpression(expression).toString();
+    }
+
+    private Expression anonymizeExpression(Expression expression)


I still have concerns about this, which I mentioned before in a related context. Anonymized expressions are not valid SQL, so we should not be trying to construct an AST out of them (effectively what this method does)

The API of Anonymizer is String anonymize(Expression expression), so we don't really want an Anonymized expression, an anonymised string representation of the Expression will do.
This method is creating an Expression because it was convenient to use an ExpressionRewriter to anonymise literals and then use Expression#toString on the result.
I think an alternative could be that we write something similar to ExpressionFormatter#Formatter that delegates to existing formatter for all methods except the ones we want to anonymise (visitXXXLiteral).
Would that be better or is there another way that you would recommend instead ?

That would be better. Alternatively, we should consider and explore making the expression formatter itself anonymizer-aware.

I've updated the expression formatter such that we could use that directly instead of creating anonymized AST. PTAL @martint

This will be used in PlanPrinter to print connector name as part of table scan node in case of anonymization.

@sopel39

The general approach looks good now. I'll leave it up to @sopel39 to do the final review

sopel39

lgtm % comments

core/trino-parser/src/main/java/io/trino/sql/ExpressionFormatter.java

sopel39 · 2022-08-25T14:50:42Z

testing/trino-tests/src/test/java/io/trino/execution/TestEventListenerBasic.java

+                                ImmutableList.of("symbol_1 := column_2"),
+                                ImmutableList.of(),
+                                ImmutableList.of()))));
+        assertThat(event.getMetadata().getJsonPlan())


We should anonymize both jsonPlan and plan. Could you create an issue + add TODO?

We are anonymizing both plan and jsonPlan. It's just there's no test for plan in TestEventListenerBasic. In general, there's no good testing of the text plan.

core/trino-spi/src/main/java/io/trino/spi/eventlistener/EventListener.java

This can be used to collect anonymised plans through query event listeners and to print anonymized plans from EXPLAIN.

gaurav8297 · 2022-08-26T09:21:36Z

@sopel39 I've addressed comments

sopel39 · 2022-08-26T10:35:36Z

core/trino-spi/src/main/java/io/trino/spi/eventlistener/EventListener.java

@@ -26,4 +26,12 @@ default void queryCompleted(QueryCompletedEvent queryCompletedEvent)
    default void splitCompleted(SplitCompletedEvent splitCompletedEvent)
    {
    }
+
+    /**
+     * Specify whether the plan included in QueryCompletedEvent should be anonymized or not


nit: this should mention that both plan and jsonPlan are anonymized

cla-bot bot added the cla-signed label Jun 24, 2022

github-actions bot added docs ui Web UI labels Jun 24, 2022

gaurav8297 requested a review from raunaqmorarka June 27, 2022 08:24

gaurav8297 removed docs ui Web UI labels Jun 27, 2022

gaurav8297 changed the title ~~[WIP] Anonymize Plan~~ Expose anonymized json plan in QueryCompletedEvent Jun 29, 2022

gaurav8297 marked this pull request as ready for review June 29, 2022 09:25

raunaqmorarka reviewed Jun 29, 2022

View reviewed changes

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/anonymize/Anonymizer.java Outdated Show resolved Hide resolved

raunaqmorarka reviewed Jul 4, 2022

View reviewed changes

core/trino-main/src/main/java/io/trino/sql/analyzer/QueryExplainer.java Outdated Show resolved Hide resolved

raunaqmorarka reviewed Jul 5, 2022

View reviewed changes

core/trino-main/src/test/java/io/trino/sql/planner/planprinter/TestJsonRepresentation.java Outdated Show resolved Hide resolved

github-actions bot added the docs label Jul 5, 2022

raunaqmorarka changed the title ~~Expose anonymized json plan in QueryCompletedEvent~~ Add anonymized query plan in json format to QueryCompletedEvent Jul 6, 2022

raunaqmorarka reviewed Jul 6, 2022

View reviewed changes

sopel39 reviewed Jul 6, 2022

View reviewed changes

martint reviewed Jul 6, 2022

View reviewed changes

sopel39 reviewed Jul 6, 2022

View reviewed changes

raunaqmorarka reviewed Jul 10, 2022

View reviewed changes

raunaqmorarka reviewed Jul 11, 2022

View reviewed changes

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/HashCodeAnonymizer.java Outdated Show resolved Hide resolved

sopel39 reviewed Jul 11, 2022

View reviewed changes

raunaqmorarka reviewed Jul 12, 2022

View reviewed changes

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/HashingAnonymizer.java Outdated Show resolved Hide resolved

raunaqmorarka reviewed Jul 12, 2022

View reviewed changes

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/HashingAnonymizer.java Outdated Show resolved Hide resolved

sopel39 approved these changes Jul 15, 2022

View reviewed changes

sopel39 reviewed Jul 18, 2022

View reviewed changes

core/trino-main/src/main/java/io/trino/sql/planner/planprinter/PlanPrinter.java Outdated Show resolved Hide resolved

gaurav8297 mentioned this pull request Jul 18, 2022

Implement EXPLAIN ANONYMIZED for all types and formats #13215

Open

martint previously requested changes Jul 18, 2022

View reviewed changes

gaurav8297 added the performance label Aug 2, 2022

raunaqmorarka approved these changes Aug 5, 2022

View reviewed changes

sopel39 requested a review from martint August 10, 2022 19:06

martint reviewed Aug 16, 2022

View reviewed changes

raunaqmorarka requested a review from martint August 17, 2022 11:55

martint reviewed Aug 19, 2022

View reviewed changes

gaurav8297 requested a review from martint August 23, 2022 20:26

gaurav8297 added 3 commits August 24, 2022 20:21

Add connector name to TableInfo

4b2aa28

This will be used in PlanPrinter to print connector name as part of table scan node in case of anonymization.

Extract formatOrderingScheme in PlanPrinter

9eaf47e

Use json string comparison in TestJsonRepresentation

641edca

sopel39 approved these changes Aug 25, 2022

View reviewed changes

gaurav8297 added 4 commits August 25, 2022 23:10

Add ability to anonymise output of PlanPrinter

dffc654

This can be used to collect anonymised plans through query event listeners and to print anonymized plans from EXPLAIN.

Add anonymized query plan in json format to QueryCompletedEvent

ee284cf

Implement EXPLAIN (TYPE DISTRIBUTED, FORMAT JSON)

9f118d0

Add docs for explain distributed json plan

8f56796

sopel39 approved these changes Aug 26, 2022

View reviewed changes

sopel39 merged commit d0563dc into trinodb:master Aug 26, 2022

sopel39 mentioned this pull request Aug 26, 2022

Release notes for 394 #13723

Closed

sopel39 reviewed Aug 26, 2022

View reviewed changes

github-actions bot added this to the 394 milestone Aug 26, 2022

gaurav8297 deleted the gaurav8297/anonymized_plan_new branch August 26, 2022 11:19

colebow mentioned this pull request Aug 29, 2022

Add Trino 394 release notes #13813

Merged

findepi mentioned this pull request Apr 9, 2024

Lazy computation of some fields in QueryMetadata and QueryStatistics #21445

Merged

		@@ -48,6 +48,7 @@
		import static io.trino.sql.analyzer.QueryType.EXPLAIN;

Add anonymized query plan in json format to QueryCompletedEvent #12968

Add anonymized query plan in json format to QueryCompletedEvent #12968

Conversation

gaurav8297 commented Jun 24, 2022 • edited by raunaqmorarka Loading

Description

Documentation

Release notes

raunaqmorarka left a comment

Choose a reason for hiding this comment

raunaqmorarka commented Jul 4, 2022

raunaqmorarka commented Jul 4, 2022

gaurav8297 commented Jul 5, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gaurav8297 commented Jul 8, 2022

raunaqmorarka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gaurav8297 commented Jul 12, 2022

sopel39 left a comment

Choose a reason for hiding this comment

sopel39 Jul 15, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sopel39 commented Jul 15, 2022

gaurav8297 commented Aug 5, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raunaqmorarka Aug 19, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sopel39 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gaurav8297 commented Aug 26, 2022

Choose a reason for hiding this comment

gaurav8297 commented Jun 24, 2022 •

edited by raunaqmorarka

Loading

sopel39 Jul 15, 2022 •

edited

Loading

raunaqmorarka Aug 19, 2022 •

edited

Loading