Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Query anonymization should work in fallback log #2804

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Query anonymization should work in fallback log #2804

wants to merge 3 commits into from
+47 −1

Conversation

LantaoJin
Copy link
Member

@LantaoJin LantaoJin commented Jul 4, 2024 •
edited
Loading

Description

Query anonymization is not working in the fallback log. JOIN query and other queries which trigger fallback to old engine are not anonymous in logs.

POST /_plugins/_sql
{
  "query" : """
  SELECT s.FlightNum, s.OriginCityName, d.DestCityName
    FROM opensearch_dashboards_sample_data_flights s
      JOIN opensearch_dashboards_sample_data_flights d
        ON s.DestCityName = d.OriginCityName
  """
}

Request SQLQueryRequest(jsonContent={"query":"\n SELECT s.FlightNum, s.OriginCityName, d.DestCityName FROM opensearch_dashboards_sample_data_flights s JOIN opensearch_dashboards_sample_data_flights d ON s.DestCityName = d.OriginCityName\n "}, query=
SELECT s.FlightNum, s.OriginCityName, d.DestCityName FROM opensearch_dashboards_sample_data_flights s JOIN opensearch_dashboards_sample_data_flights d ON s.DestCityName = d.OriginCityName
, path=/_plugins/_sql, format=jdbc, params={pretty=true}, sanitize=true, cursor=Optional.empty) is not supported and falling back to old SQL engine

This PR resolves it by adding a new toString() method with a lambda function parameter anonymizer in SQLQueryRequest since QueryDataAnonymizer is in org.opensearch.sql.legacy package.

With this fixing, the new log show like

Request SQLQueryRequest(query=( SELECT identifier, identifier, identifier FROM table s JOIN table d ON identifier = identifier ), path=/_plugins/_sql, format=jdbc, params={pretty=true}, sanitize=true, cursor=Optional.empty) is not supported and falling back to old SQL engine

Issues Resolved

Resolves #2803

Check List

  • New functionality includes testing.
    • All tests pass, including unit test, integration test and doctest
  • [ ] New functionality has been documented.
    • [ ] New functionality has javadoc added
    • [ ] New functionality has user manual doc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@LantaoJin
Copy link
Member Author

The build failures are not related to this PR. It looks like GLIBC lib is missing in current Linux docker image. All CI tasks passed in Windows docker.

@LantaoJin
Query anonymization should work in fallback log
fd1129d 
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
fix typo
16d681a 
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
address coverage
4572e25 
Signed-off-by: Lantao Jin <ltjin@amazon.com>
dai-chen
dai-chen reviewed Jul 15, 2024
View reviewed changes
sql/src/main/java/org/opensearch/sql/sql/domain/SQLQueryRequest.java
@@ -148,4 +149,22 @@ private boolean shouldSanitize(Map<String, String> params) {
}
return true;
}

/** A new toString() with anonymizer parameter to anonymize its query statement. */
public String toString(Function<String, String> anonymizer) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wonder do you see future requirement for different anonymizer? Is toString(boolean anonymized) or copy to a new request clear enough?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dai-chen dai-chen dai-chen left review comments

@ps48 ps48 Awaiting requested review from ps48 ps48 is a code owner

@kavithacm kavithacm Awaiting requested review from kavithacm kavithacm is a code owner

@derek-ho derek-ho Awaiting requested review from derek-ho derek-ho is a code owner

@joshuali925 joshuali925 Awaiting requested review from joshuali925 joshuali925 is a code owner

@YANG-DB YANG-DB Awaiting requested review from YANG-DB YANG-DB is a code owner

@rupal-bq rupal-bq Awaiting requested review from rupal-bq rupal-bq is a code owner

@mengweieric mengweieric Awaiting requested review from mengweieric mengweieric is a code owner

@vamsi-amazon vamsi-amazon Awaiting requested review from vamsi-amazon vamsi-amazon is a code owner

@Swiddis Swiddis Awaiting requested review from Swiddis Swiddis is a code owner

@penghuo penghuo Awaiting requested review from penghuo penghuo is a code owner

@seankao-az seankao-az Awaiting requested review from seankao-az seankao-az is a code owner

@MaxKsyunz MaxKsyunz Awaiting requested review from MaxKsyunz MaxKsyunz is a code owner

@Yury-Fridlyand Yury-Fridlyand Awaiting requested review from Yury-Fridlyand Yury-Fridlyand is a code owner

@anirudha anirudha Awaiting requested review from anirudha anirudha is a code owner

@forestmvey forestmvey Awaiting requested review from forestmvey forestmvey is a code owner

@acarbonetto acarbonetto Awaiting requested review from acarbonetto acarbonetto is a code owner

@GumpacG GumpacG Awaiting requested review from GumpacG GumpacG is a code owner

@ykmr1224 ykmr1224 Awaiting requested review from ykmr1224 ykmr1224 is a code owner

At least 2 approving reviews are required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[BUG] Query anonymization doesn't work in fallback log
2 participants
@LantaoJin @dai-chen