Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(glue): Missing parameter *parameters* from class Table (construct) #14159

Closed
1 task done
braidenjudd opened this issue Apr 14, 2021 · 7 comments · Fixed by #27643
Closed
1 task done

(glue): Missing parameter *parameters* from class Table (construct) #14159

braidenjudd opened this issue Apr 14, 2021 · 7 comments · Fixed by #27643
Labels
@aws-cdk/aws-glue Related to AWS Glue effort/small Small work item – less than a day of effort feature/coverage-gap Gaps in CloudFormation coverage by L2 constructs feature-request A feature should be added or improved. good first issue Related to contributions. See CONTRIBUTING.md p1

Comments

@braidenjudd
Copy link

Add cfn-glue-table-tableinput-parameters to the Glue Table Construct

Use Case

I want to be able to set parameters at create time of the table to make the table compatible with a crawler I am using.

Proposed Solution

Optional props parameters to TableProps and append them here Table

  • 👋 I may be able to implement this feature request

This is a 🚀 Feature Request

@braidenjudd braidenjudd added feature-request A feature should be added or improved. needs-triage This issue or PR still needs to be triaged. labels Apr 14, 2021
@braidenjudd braidenjudd changed the title (module name): short issue description Glue: Missing parameter *parameters* from class Table (construct) Apr 14, 2021
@github-actions github-actions bot added the @aws-cdk/aws-glue Related to AWS Glue label Apr 14, 2021
@iliapolo iliapolo changed the title Glue: Missing parameter *parameters* from class Table (construct) (glue): Missing parameter *parameters* from class Table (construct) Apr 15, 2021
@iliapolo iliapolo added effort/small Small work item – less than a day of effort p2 and removed needs-triage This issue or PR still needs to be triaged. labels Apr 15, 2021
@iliapolo
Copy link
Contributor

We are unassigning and marking this issue as p2, which means that we are unable to work on this immediately.
We use +1s to help prioritize our work, and are happy to revaluate this issue based on community feedback. You can reach out to the cdk.dev community on Slack to solicit support for reprioritization.

If you'd like to contribute this feature, that would also be great :)

@fs111
Copy link

fs111 commented Nov 24, 2021

I ran into this limitation yesterday and it took me a while to figure out a work-around. Could you please reconsider adding this?

If you come here and wonder how to solve this, you can "hack" around it by using the CDK escape hatch into Cfn types like so (example is in java):

var glueTable = Table.Builder.create(this, "glueTable")[...].build();
CfnTable cfnTable = (CfnTable) glueTable.getNode().getDefaultChild();
cfnTable.addPropertyOverride("TableInput.StorageDescriptor.SerdeInfo.Parameters.serialization\\.format", 1);

@mattiamatrix
Copy link
Contributor

This is a possible solution in TS

(<CfnTable>table.node.defaultChild).addPropertyOverride(
    'TableInput.StorageDescriptor.SerdeInfo.Parameters.serialization\\.format',
    1
);

@kaizencc kaizencc added the good first issue Related to contributions. See CONTRIBUTING.md label Mar 21, 2022
@lihonosov
Copy link

lihonosov commented Jun 30, 2023

@mattiamatrix @fs111 thanks for the workaround, it was useful! I needed to add useGlueParquetWriter=true to the Parameters section:

(<CfnTable>table.node.defaultChild).addPropertyOverride(
      "TableInput.Parameters.useGlueParquetWriter",
      "true"
);

@github-actions github-actions bot added p1 and removed p2 labels Jul 2, 2023
@github-actions
Copy link

github-actions bot commented Jul 2, 2023

This issue has received a significant amount of attention so we are automatically upgrading its priority. A member of the community will see the re-prioritization and provide an update on the issue.

@lawofcycles
Copy link
Contributor

lawofcycles commented Oct 20, 2023

I may be able to implement this feature.

sumupitchayan added a commit that referenced this issue Dec 27, 2023
…table construct (#27643)

Add
[cfn-glue-table-tableinput-parameters](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-glue-table-tableinput.html#cfn-glue-table-tableinput-parameters)
to Glue Table construct as optional props

User can specify additional table parameter when creating Glue Table. 
Any key/value can be set depending on each user's requirement like
table's additional metadata or statistics. Some parameter can be used
when AWS services / 3rd party tools read table like
`skip.header.line.count`.

Closes #14159.

---
All Submissions:
- [x] Have you followed the guidelines in our [Contributing
guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md)
Adding new Unconventional Dependencies:
- [ ] This PR adds new unconventional dependencies following the process
described
[here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies)
New Features
- [x] Have you added the new feature to an [integration
test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)?
- [x] Did you use yarn integ to deploy the infrastructure and generate
the snapshot (i.e. yarn integ without --dry-run)?
---
By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache-2.0 license

---------

Co-authored-by: Vinayak Kukreja <78971045+vinayak-kukreja@users.noreply.github.com>
Co-authored-by: Sumu Pitchayan <35242245+sumupitchayan@users.noreply.github.com>
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

paulhcsun pushed a commit to paulhcsun/aws-cdk that referenced this issue Jan 5, 2024
…table construct (aws#27643)

Add
[cfn-glue-table-tableinput-parameters](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-glue-table-tableinput.html#cfn-glue-table-tableinput-parameters)
to Glue Table construct as optional props

User can specify additional table parameter when creating Glue Table. 
Any key/value can be set depending on each user's requirement like
table's additional metadata or statistics. Some parameter can be used
when AWS services / 3rd party tools read table like
`skip.header.line.count`.

Closes aws#14159.

---
All Submissions:
- [x] Have you followed the guidelines in our [Contributing
guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md)
Adding new Unconventional Dependencies:
- [ ] This PR adds new unconventional dependencies following the process
described
[here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies)
New Features
- [x] Have you added the new feature to an [integration
test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)?
- [x] Did you use yarn integ to deploy the infrastructure and generate
the snapshot (i.e. yarn integ without --dry-run)?
---
By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache-2.0 license

---------

Co-authored-by: Vinayak Kukreja <78971045+vinayak-kukreja@users.noreply.github.com>
Co-authored-by: Sumu Pitchayan <35242245+sumupitchayan@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-glue Related to AWS Glue effort/small Small work item – less than a day of effort feature/coverage-gap Gaps in CloudFormation coverage by L2 constructs feature-request A feature should be added or improved. good first issue Related to contributions. See CONTRIBUTING.md p1
Projects
None yet
8 participants