Database Migration Service provider 'resource_aws_dms_endpoint.go' breaks/drops extra-connection-attributes due to conditional logic error

[jonassteinberg1]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.13.7
hashicorp/aws v2.70.0

Affected Resource(s)

resource "aws_dms_endpoint" "source" {
  count = local.enabled ? 1 : 0

  endpoint_id   = format("%s-source", module.source_endpoint_label.id)
  endpoint_type = "source"

  server_name = local.aurora_postgres_hostname
  username    = local.aurora_postgres_username
  password    = local.aurora_postgres_password

  engine_name                 = var.source_engine_name
  database_name               = format("%s-%s-%s-%s", module.source_endpoint_label.namespace, module.source_endpoint_label.environment, module.source_endpoint_label.stage, var.database_name)
  port                        = var.source_port
  ssl_mode                    = var.source_ssl_mode
  extra_connection_attributes = var.source_extra_connection_attributes
  tags                        = module.source_endpoint_label.tags

  depends_on = [
    shell_script.stop_replication_task[0]
  ]
}

Expected Behavior

extra-connection-attributes settings are persisted after an endpoint change is made.

Actual Behavior

extra-connection-attributes settings are dropped and AWS defaults are used.

Important Factoids

If you examine the following the following line in resource_aws_dms_endpoint.go you will notice that the logic is as follows:

1. a request is built
2. it is defaulted to 'hasChanges = false'
3. if any of the other attributes change, the endpoint is updated, but in the case of extra-connection-attributes AWS sdk returns the default settings. So in the case where someone is setting the dataFormat=parquet parameter: it reverts to AWS's default of 'csv'. Currently: AWS says not to use extra-connection-attributes. But: for setting the dataFormat=parquet you have to. The consequence is that if you use this provider all of your storage, e.g. s3, will start being written to in csv.

[jonassteinberg1]

additionally: the setting I need to specify, which is not persisted after endpoint settings are updated, is not handled for in this or else I would use that and the problem would be solved (at least for my use case; there are a ton of other settings that this provider isn't handling for (and I can see why))

[anGie44]

Hi @jonassteinberg1 , thank you for raising this issue. Do you mind sharing what you are setting extra-connection-attributes and/or s3_settings to? There are some open issues related to extra_connection_attributes/s3_settings that could be responsible for the behavior you're seeing but will need to confirm.

[FredStober]

I'm having the same problem. I'm trying to set the extra connection attribute: "addColumnName=true" for the s3 engine.
I can change the value to true / false and it won't even show up in the plan.

[jonassteinberg1]

@anGie44 Apologies for the delayed response. The problem setting is dataFormat=parquet, but as I've specified above there seems to be a bug in the logic of line such that

1. a request is built
2. it is defaulted to 'hasChanges = false'
3. if any of the other attributes change, the endpoint is updated, but in the case of extra-connection-attributes AWS sdk returns the default settings. So in the case where someone is setting the dataFormat=parquet parameter: it reverts to AWS's default of 'csv'. Currently: AWS says not to use extra-connection-attributes. But: for setting the dataFormat=parquet you have to. The consequence is that if you use this provider all of your storage, e.g. s3, will start being written to in csv.

I've listed my entire extra-connection-settings file below as you've requested:

cdcInsertsOnly=false;compressionType=GZIP;csvDelimiter=\,;csvRowDelimiter=\n;dataFormat=parquet;DatePartitionDelimiter=NONE;datePartitionEnabled=true;DatePartitionSequence=YYYYMMDD;includeOpForFullLoad=true;parquetTimestampInMillisecond=true;timestampColumnName=test-timestamp;

[github-actions]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.