Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

glue: glue connections to be assigned to glue tables #24741

Closed
1 of 2 tasks
Rizxcviii opened this issue Mar 22, 2023 · 2 comments · Fixed by #24753
Closed
1 of 2 tasks

glue: glue connections to be assigned to glue tables #24741

Rizxcviii opened this issue Mar 22, 2023 · 2 comments · Fixed by #24753
Labels
@aws-cdk/aws-glue Related to AWS Glue effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p2

Comments

@Rizxcviii
Copy link
Contributor

Rizxcviii commented Mar 22, 2023 •
edited
Loading

Describe the feature

When creating a table, you are able to add an additional parameter to the Table properties called connectionName which will allow developers to assign a Glue Data Connection to the Connection property.
There should be a field to allow you to directly manipulate the connectionName property. In addition, there could potentially be a need to allow direct access of the location property, as crawlers tend to change the location property, if crawling a third party database.

Use Case

Databases that contain structured data, that may never change (or change very rarely) would not need the use of a crawler to update the schema. This could be good in reducing cost of the end user.
In addition, when running glue jobs, developers can directly use the table, as this will also contain the connection needed.

Proposed Solution

  • Add a connection property that could potentially take an existing glue connection, or it's associating connection name.
  • Allow dynamic changing of the location name, if the glue data source does not reside in S3. This could either be a separate feature request, or could be included here.

Other Information

No response

Acknowledgements

  • I may be able to implement this feature request
  • This feature might incur a breaking change

CDK version used

2.68.0

Environment details (OS name and version, etc.)

Amazon Linux 2
@Rizxcviii Rizxcviii added feature-request A feature should be added or improved. needs-triage This issue or PR still needs to be triaged. labels Mar 22, 2023
@github-actions github-actions bot added the @aws-cdk/aws-glue Related to AWS Glue label Mar 22, 2023
@pahud
Copy link
Contributor

pahud commented Mar 22, 2023

Thanks for the use case and this seems to be a useful feature. Feel free to submit your PR when it's ready. Thanks!

@pahud pahud added p2 effort/medium Medium work item – several days of effort and removed needs-triage This issue or PR still needs to be triaged. labels Mar 22, 2023
@Rizxcviii Rizxcviii mentioned this issue Mar 22, 2023
Merged
@mergify mergify bot closed this as completed in #24753 Sep 11, 2023
mergify bot pushed a commit that referenced this issue Sep 11, 2023
@Rizxcviii
feat(glue): add ExternalTable for use with connections (#24753)
1c03cb3 
Changing the table structure to include an initial `TableBase` abstract class, allowing different tables of different data sources to be created from. Initially there are two, `S3Table` and `ExternalTable`.

- `S3Table`: The current table structure that has been used throughout the previous versions of the CDK
- `ExternalTable`: The new glue table that will be used to store metadata about external data sources. This subclass will contain an `externalDataLocation` property to explicitly specify the `Location` property of the underlying `CfnTable` L1 construct
- `Table`: This is now `@deprecated` to shift the usage towards `S3Table`

Closes #24741.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@github-actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

mikewrighton pushed a commit that referenced this issue Sep 14, 2023
@Rizxcviii
feat(glue): add ExternalTable for use with connections (#24753)
926530a 
Changing the table structure to include an initial `TableBase` abstract class, allowing different tables of different data sources to be created from. Initially there are two, `S3Table` and `ExternalTable`.

- `S3Table`: The current table structure that has been used throughout the previous versions of the CDK
- `ExternalTable`: The new glue table that will be used to store metadata about external data sources. This subclass will contain an `externalDataLocation` property to explicitly specify the `Location` property of the underlying `CfnTable` L1 construct
- `Table`: This is now `@deprecated` to shift the usage towards `S3Table`

Closes #24741.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
@aws-cdk/aws-glue Related to AWS Glue effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p2
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(glue): add ExternalTable for use with connections
2 participants
@pahud @Rizxcviii