feat(ingest/snowflake): Add feature flag for view defintions #10914
Conversation
WalkthroughThe updates to the Snowflake ingestion codebase primarily focus on adding control over whether view definitions are included in the result sets. This is achieved through the introduction of the Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant SnowflakeSchemaGen
participant SnowflakeSchema
User->>SnowflakeSchemaGen: Request to retrieve views
SnowflakeSchemaGen->>SnowflakeSchema: Call get_views_for_database(include_view_definitions)
SnowflakeSchema-->>SnowflakeSchemaGen: Return views (with or without definitions)
SnowflakeSchemaGen->>User: Return processed views
Poem
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configration File (
|
github-actions
bot
added
the
ingestion
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (3)
- metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_config.py (1 hunks)
- metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_schema.py (2 hunks)
- metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_schema_gen.py (1 hunks)
Additional comments not posted (4)
metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_config.py (1)
184-187: LGTM! The new fieldinclude_view_definitionsis correctly added.The addition of the
include_view_definitionsfield with a default value ofTrueand a description is consistent with the existing configuration fields.metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_schema.py (2)
323-325: LGTM! The methodget_views_for_databaseis correctly updated.The addition of the
include_view_definitionsparameter to theget_views_for_databasemethod is consistent with the intended functionality.
358-360: LGTM! The logic to include view definitions is correctly implemented.The conditional inclusion of view definitions based on the
include_view_definitionsparameter is correctly implemented.metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_schema_gen.py (1)
1020-1022: LGTM! The methodget_views_for_schemais correctly updated.The addition of the
include_view_definitionsparameter to theget_views_for_databasemethod call is consistent with the intended functionality.
metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_config.py
Show resolved
Hide resolved
There was a problem hiding this comment.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (2)
- metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_schema.py (1 hunks)
- metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_schema_gen.py (3 hunks)
Files skipped from review as they are similar to previous changes (1)
- metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_schema_gen.py
Additional comments not posted (1)
metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_schema.py (1)
135-137: Refactor: Simplified initialization of_table_tags.The change to initialize
_table_tagsusing a more concise lambda expression is a good practice as it improves readability and maintainability of the code. However, ensure that this refactoring does not alter the behavior, especially in edge cases where the nested dictionary structure is significant.
There was a problem hiding this comment.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (2)
- metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_schema_gen.py (1 hunks)
- metadata-models/src/main/pegasus/com/linkedin/dataset/ViewProperties.pdl (1 hunks)
Files skipped from review as they are similar to previous changes (1)
- metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_schema_gen.py
Additional comments not posted (1)
metadata-models/src/main/pegasus/com/linkedin/dataset/ViewProperties.pdl (1)
26-26: Addition of theviewLogicfield is well-integrated.The addition of the
viewLogicfield as an optional string is consistent with the feature flag implementation for view definitions. This allows flexibility in whether or not view logic is included in the dataset metadata, aligning with the PR's objectives.Ensure that the field is properly documented in the code to maintain clarity and consistency with other fields in the record.
There was a problem hiding this comment.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (1)
- metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_schema_gen.py (1 hunks)
Files skipped from review as they are similar to previous changes (1)
- metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_schema_gen.py
metadata-ingestion/src/datahub/ingestion/source/snowflake/snowflake_config.py
Show resolved
Hide resolved
hsheth2
changed the title
Co-authored-by: Ethan Cartwright <ethan.cartwright@acryl.io>
Checklist
Summary by CodeRabbit
New Features
Improvements