Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Parameters removed when creating Iceberg Glue Tables #33370

Closed
gdowmont-gss opened this issue Sep 8, 2023 · 4 comments · Fixed by #33374
Closed

[Bug]: Parameters removed when creating Iceberg Glue Tables #33370

gdowmont-gss opened this issue Sep 8, 2023 · 4 comments · Fixed by #33374
Labels
bug Addresses a defect in current functionality. prioritized Part of the maintainer teams immediate focus. To be addressed within the current quarter. regression Pertains to a degraded workflow resulting from an upstream patch or internal enhancement. service/glue Issues and PRs that pertain to the glue service.
Milestone

Comments

@gdowmont-gss
Copy link

gdowmont-gss commented Sep 8, 2023

Terraform Core Version

1.5.0

AWS Provider Version

5.16.0

Affected Resource(s)

aws_glue_catalog_table

Expected Behavior

When modifying existing Glue Table with Iceberg configuration enabled (open_table_format_input) provider should modify configuration of the table while persisting Parameters values for:

"Parameters": {
            "metadata_location": "s3://some-path/test2/metadata/00000-fb573dc6-8b72-46eb-8093-6757a808c57e.metadata.json",
            "table_type": "ICEBERG"
        }

Actual Behavior

Parameters metadata_location and table_type are removed and the table is no longer usable as an Iceberg table.

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

resource "aws_glue_catalog_table" "aws_glue_catalog_table_test" {
  name          = "test"
  database_name = "db_name"
  table_type    = "EXTERNAL_TABLE"

  open_table_format_input {
    iceberg_input {
      metadata_operation = "CREATE"
      version            = 2
    }
  }

  storage_descriptor {
    location = "s3://bucket_name/"
    columns {
      name = "field1"
      type = "string"
    }
  }
}

Steps to Reproduce

  1. Create new glue table while specifying:
open_table_format_input {
    iceberg_input {
      metadata_operation = "CREATE"
      version            = 2
    }
}

and setting location in storage_descriptor
2. Check that table incudes Parameters: metadata_location and table_type
3. Add/rename column in storage descriptor
4. Apply terraform
5. After successful apply table loses Parameters: metadata_location and table_type

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

#33157

Would you like to implement a fix?

None

@gdowmont-gss gdowmont-gss added the bug Addresses a defect in current functionality. label Sep 8, 2023
@github-actions
Copy link

github-actions bot commented Sep 8, 2023

Community Note

Voting for Prioritization

  • Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
  • Please see our prioritization guide for information on how we prioritize.
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

  • If you are interested in working on this issue, please leave a comment.
  • If this would be your first contribution, please review the contribution guide.

@github-actions github-actions bot added the service/glue Issues and PRs that pertain to the glue service. label Sep 8, 2023
@terraform-aws-provider terraform-aws-provider bot added the needs-triage Waiting for first response or review from a maintainer. label Sep 8, 2023
@ewbankkit ewbankkit added regression Pertains to a degraded workflow resulting from an upstream patch or internal enhancement. and removed needs-triage Waiting for first response or review from a maintainer. labels Sep 8, 2023
@terraform-aws-provider terraform-aws-provider bot added the prioritized Part of the maintainer teams immediate focus. To be addressed within the current quarter. label Sep 8, 2023
@ewbankkit
Copy link
Contributor

func flattenNonManagedParameters(table *glue.TableData) map[string]string {
allParameters := table.Parameters
if aws.StringValue(allParameters["table_type"]) == "ICEBERG" {
delete(allParameters, "table_type")
delete(allParameters, "metadata_location")
}
return aws.StringValueMap(allParameters)
}

@github-actions github-actions bot added this to the v5.17.0 milestone Sep 8, 2023
@ewbankkit ewbankkit modified the milestones: v5.17.0, v5.16.1 Sep 8, 2023
@github-actions github-actions bot removed the bug Addresses a defect in current functionality. label Sep 11, 2023
@github-actions
Copy link

This functionality has been released in v5.16.1 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

@github-actions
Copy link

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 12, 2023
@justinretzolk justinretzolk added the bug Addresses a defect in current functionality. label Feb 10, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Addresses a defect in current functionality. prioritized Part of the maintainer teams immediate focus. To be addressed within the current quarter. regression Pertains to a degraded workflow resulting from an upstream patch or internal enhancement. service/glue Issues and PRs that pertain to the glue service.
Projects
None yet
3 participants