ADD: support sql block rule #6192

stalary · 2021-07-09T08:09:49Z

Proposed changes

Support configuring sql_block_rule by user level to deny the specified SQL

Types of changes

What types of changes does your code introduce to Doris?
Put an x in the boxes that apply

Bugfix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation Update (if none of the other choices apply)
Code refactor (Modify the code structure, format the code, etc...)

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

I have created an issue on (Fix [Feature] Support configuring blocklist by user level to deny the specified SQL #6103) and described the bug/feature there in detail
Compiling and unit tests pass locally with my changes
I have added tests that prove my fix is effective or that my feature works
If these changes need document changes, I have updated the document
Any dependent changes have been merged

fe/fe-core/src/main/cup/sql_parser.cup

fe/fe-core/src/main/java/org/apache/doris/analysis/AlterSqlBlockRuleStmt.java

fe/fe-core/src/main/java/org/apache/doris/analysis/CreateSqlBlockRuleStmt.java

fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java

fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java

fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java

fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java

fe/fe-core/src/test/java/org/apache/doris/qe/StmtExecutorTest.java

xinghuayu007 · 2021-07-12T06:30:41Z

I think this feature is used to block some abnormal sql, for example , sql caused core. Is it need to limit the number of rules, which will occupied a lot of memory. Is is better to store the hash value of a sql, not the sql self, when a sql is very long?

stalary · 2021-07-12T06:35:47Z

I think this feature is used to block some abnormal sql, for example , sql caused core. Is it need to limit the number of rules, which will occupied a lot of memory. Is is better to store the hash value of a sql, not the sql self, when a sql is very long?

At present, in my scenario, some special user are not allowed to join, and some large tables are not allowed to query the full table, really do not need too many rules

morningman · 2021-07-13T05:55:45Z

I think this feature is used to block some abnormal sql, for example , sql caused core. Is it need to limit the number of rules, which will occupied a lot of memory. Is is better to store the hash value of a sql, not the sql self, when a sql is very long?

Using hash to block problematic SQL may also be a practical requirement. In some cases, we need to quickly block some SQL that affects the cluster through the block rules. There are pros and cons to using regular expressions or hash values:

It is more flexible for regular expressions, and can match a series of sql through one rule. But the disadvantage is that the threshold for using regular expressions is high, and wrong expressions may result in matching wrong SQL.
The disadvantage of Hash value is that it can only perform complete matching, but it takes up less space and matches more quickly. But any change in SQL will cause a change in its hash value, such as spaces, line breaks, and so on. Therefore, I suggest that we can print the hash value corresponding to sql in fe.audit.log, so that users can conveniently use the hash value in the log to quickly add block rule.

In summary, we can implement two block rules, one based on hash and the other based on regular expressions.

stalary · 2021-07-13T13:42:29Z

I think this feature is used to block some abnormal sql, for example , sql caused core. Is it need to limit the number of rules, which will occupied a lot of memory. Is is better to store the hash value of a sql, not the sql self, when a sql is very long?

Using hash to block problematic SQL may also be a practical requirement. In some cases, we need to quickly block some SQL that affects the cluster through the block rules. There are pros and cons to using regular expressions or hash values:

It is more flexible for regular expressions, and can match a series of sql through one rule. But the disadvantage is that the threshold for using regular expressions is high, and wrong expressions may result in matching wrong SQL.

The disadvantage of Hash value is that it can only perform complete matching, but it takes up less space and matches more quickly. But any change in SQL will cause a change in its hash value, such as spaces, line breaks, and so on. Therefore, I suggest that we can print the hash value corresponding to sql in fe.audit.log, so that users can conveniently use the hash value in the log to quickly add block rule.

In summary, we can implement two block rules, one based on hash and the other based on regular expressions.

Okay, I'm going to support both matches

docs/zh-CN/administrator-guide/block-rule/sql-block.md

fe/fe-core/src/main/cup/sql_parser.cup

fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java

fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRuleMgr.java

docs/zh-CN/administrator-guide/block-rule/sql-block.md

fe/fe-core/src/main/java/org/apache/doris/analysis/AlterSqlBlockRuleStmt.java

fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRule.java

fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRuleMgr.java

fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java

fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRuleMgr.java

fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java

fe/fe-core/src/main/java/org/apache/doris/blockrule/SqlBlockRuleMgr.java

fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/UserProperty.java

fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java

fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/UserProperty.java

morningman

LGTM

github-actions · 2021-08-09T06:31:36Z

PR approved by at least one committer and no changes requested.

github-actions · 2021-08-09T06:31:38Z

PR approved by anyone and no changes requested.

morningman

LGTM

github-actions · 2021-08-12T14:42:02Z

PR approved by at least one committer and no changes requested.

Support grammar: - SHOW SQL_BLOCK_RULE [FOR NAME] - CREATE SQL_BLOCK_RULE test_rule PROPERTIES ("user"="default", "sql"="select .* from .* join .*", "enable": "true"); - ALTER SQL_BLOCK_RULE test_rule PROPERTIES ("user"="test_user", "enable": "false"); - DROP SQL_BLOCK_RULE test_rule1,test_rule2;

morningman added area/sql-block-list kind/feature Categorizes issue or PR as related to a new feature. labels Jul 9, 2021

morningman reviewed Jul 9, 2021

View reviewed changes

morningman reviewed Jul 16, 2021

View reviewed changes

morningman reviewed Jul 25, 2021

View reviewed changes

stalary force-pushed the support_sql_block_rule branch 2 times, most recently from 2a51703 to b0c8bcc Compare July 27, 2021 06:44

stalary force-pushed the support_sql_block_rule branch from b1aa4ef to 80ce658 Compare August 8, 2021 07:48

morningman reviewed Aug 8, 2021

View reviewed changes

stalary force-pushed the support_sql_block_rule branch from 71423dd to 2e0b697 Compare August 8, 2021 15:32

morningman reviewed Aug 9, 2021

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java Outdated Show resolved Hide resolved

fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/UserProperty.java Outdated Show resolved Hide resolved

morningman previously approved these changes Aug 9, 2021

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 9, 2021

github-actions bot added the reviewed label Aug 9, 2021

stalary dismissed morningman’s stale review via f8b2d20 August 9, 2021 06:51

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Aug 9, 2021

stalary added 8 commits August 12, 2021 22:40

ADD: support sql block rule

2b68c0e

ADD: mod some code, add test, support sql hash match

56e76d6

MOD: move user to UserProperty

c8ffc85

MOD: mod some code

1c2afcc

FIX: fix some bug

c68bddf

MOD: sidebar

d58e18c

MOD: sidebar

919cf7a

MOD: mod some code

961e5e1

stalary added 6 commits August 12, 2021 22:40

MOD: sort code

9caf01a

MOD: move sqlBlockRulesSplit to CommonUserProperties

72363ae

FIX: npe

4ffc55f

ADD: ut

fa6e47e

FIX: fix ut

e2bd907

MOD: clean code

a4b6d49

stalary force-pushed the support_sql_block_rule branch from 5d5bb52 to a4b6d49 Compare August 12, 2021 14:40

morningman approved these changes Aug 12, 2021

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 12, 2021

morningman merged commit 5e6f1b8 into apache:master Aug 13, 2021

ccoffline mentioned this pull request Sep 8, 2021

Custom config handler #6508 #6577

Merged

13 tasks

morningman mentioned this pull request Oct 14, 2021

Release note 0.15.0 #6806

Closed

stalary deleted the support_sql_block_rule branch November 11, 2022 00:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ADD: support sql block rule #6192

ADD: support sql block rule #6192

stalary commented Jul 9, 2021 •

edited

Loading

xinghuayu007 commented Jul 12, 2021

stalary commented Jul 12, 2021

morningman commented Jul 13, 2021

stalary commented Jul 13, 2021

morningman left a comment

github-actions bot commented Aug 9, 2021

github-actions bot commented Aug 9, 2021

morningman left a comment

github-actions bot commented Aug 12, 2021

ADD: support sql block rule #6192

ADD: support sql block rule #6192

Conversation

stalary commented Jul 9, 2021 • edited Loading

Proposed changes

Types of changes

Checklist

xinghuayu007 commented Jul 12, 2021

stalary commented Jul 12, 2021

morningman commented Jul 13, 2021

stalary commented Jul 13, 2021

morningman left a comment

Choose a reason for hiding this comment

github-actions bot commented Aug 9, 2021

github-actions bot commented Aug 9, 2021

morningman left a comment

Choose a reason for hiding this comment

github-actions bot commented Aug 12, 2021

stalary commented Jul 9, 2021 •

edited

Loading