Add toggle for specifying dynamic filtering refresh interval #4388
Conversation
There was a problem hiding this comment.
Not sure what is the config parameters deprecation policy. E.g If plan to remove it soon anyway should we maybe already mark it as @deprecated?
Also I was not able to understand motivation for the change based on commit message :)
sopel39
force-pushed
the
ks/dyn_filtering_toggle
branch
from
a1a2fe2 to
4fcca0c
Compare
I think
I've improved it slightly |
| @@ -786,6 +790,21 @@ public FeaturesConfig setDynamicFilteringMaxPerDriverSize(DataSize dynamicFilter | |||
| return this; | |||
| } | |||
|
|
|||
| @MinDuration("1ms") | |||
| @MaxDuration("10s") | |||
| @@ -131,6 +134,7 @@ | |||
| private boolean enableDynamicFiltering = true; | |||
| private int dynamicFilteringMaxPerDriverRowCount = 100; | |||
| private DataSize dynamicFilteringMaxPerDriverSize = DataSize.of(10, KILOBYTE); | |||
| private Duration dynamicFilteringRefreshInterval = new Duration(100, MILLISECONDS); | |||
There was a problem hiding this comment.
What was the default previously? Arent we changing rate too aggressively?
There was a problem hiding this comment.
The earlier default was 1 sec. It might be worth checking the time taken by collectDynamicFilters when there are large no. of queries active with multiple stages containing dynamic filters.
There was a problem hiding this comment.
I've changed it to 200ms. I don't think it's too aggressive. Anyway, it's limited to single thread.
sopel39
force-pushed
the
ks/dyn_filtering_toggle
branch
from
4fcca0c to
002f1a5
Compare
Previously, dynamic filtering refresh interval was set to "task.status-refresh-max-wait". However, task status will be refreshed more frequently since "task.status-refresh-max-wait" specifies maximum waiting time. Therefore a dedicated refresh toggle is added that makes dynamic filtering collection more frequent. This toggle is marked as experimental as it will eventually be replaced by delta-based task status update mechanism.
sopel39
force-pushed
the
ks/dyn_filtering_toggle
branch
from
002f1a5 to
44d2fef
Compare
Previously, dynamic filtering refresh interval was set to
"task.status-refresh-max-wait". However, task status will be
refreshed more frequently since "task.status-refresh-max-wait"
specifies max wait. Therefore a dedicated refresh toggle is added.
This toggle is marked as experimental as it will eventually be replaced by
delta-based task status update mechanism.