You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In case the constraint is large sending the constraint (TupleDomain) per split is wasteful (for the case the constraint is the same per split).
In our connector we have relatively large number of small splits and we've stumbled upon the use case of querying for 10K string keywords in a single query we hacked/fixed it by adding the constraint to TaskSource and then infest it to all its splits.
Does it makes sense? Is this approach acceptable? Would it be more clean to add another kind of connectorHandle per taskSource?
The text was updated successfully, but these errors were encountered:
The information is contained in the TableHandle associated with the TableScanNode. What we might need to do is change the PageSourceProvider API to take the TableHandle in addition to the Split. That way, anything that's common for the whole query doesn't need to be included in every split (only what's relevant to a split).
In case the constraint is large sending the constraint (TupleDomain) per split is wasteful (for the case the constraint is the same per split).
In our connector we have relatively large number of small splits and we've stumbled upon the use case of querying for 10K string keywords in a single query we hacked/fixed it by adding the constraint to TaskSource and then infest it to all its splits.
Does it makes sense? Is this approach acceptable? Would it be more clean to add another kind of connectorHandle per taskSource?
The text was updated successfully, but these errors were encountered: