Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable querying policy-enabled table in MSQ, and use RestrictedDataSource as a base in DataSourceAnalysis. #17666

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

cecemei
Copy link
Contributor

@cecemei cecemei commented Jan 25, 2025

Description

This PR enables querying policy-enabled table in MSQ.


Key changed/added classes in this PR
  • DataSourceAnalysis, getBaseTableDataSource can now return the base of RestrictedDataSource. This is a more robust solution than using the underlying table as base.
  • DruidQuery, can also be created by withPolicy, which just applies policy restriction to the original query.
  • MSQTaskQueryMaker would apply restrictions on DruidQuery, instead of throw permission error.
  • DataSourcePlan can handle RestrictedDataSource.
  • a new class RestrictedInputNumberDataSource, which basically wraps a NumberDataSource with a policy, and its SegmentMapFn can be used to create a RestrictedSegment.
  • RunWorkOrder, try to make a few refactors to make the code clear, no behavior change. ShufflePipelineBuilder.build(), it was not clear before that the channel future should only be returned when the resultFuture is ready. Also, the sanity check is moved to OutputChannels.
  • BaseLeafFrameProcessorFactory.makeProcessors(). Previously, makeSegmentMapFnProcessor can return null. After this PR, it cannot. The difference is that now the SegmentMapFn from query.dataSource is handled in ChainedProcessorManager for all query (was just join query before).

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@github-actions github-actions bot added Area - Batch Ingestion Area - Querying Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 labels Jan 25, 2025
@cecemei cecemei changed the title Enable RestrictedDataSource as a base in DataSourceAnalysis, and enab… Enable querying policy-enabled table in MSQ, and use RestrictedDataSource as a base in DataSourceAnalysis. Jan 25, 2025
@cecemei cecemei marked this pull request as ready for review January 28, 2025 04:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area - Batch Ingestion Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 Area - Querying
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant