[Bug]: DMS Task needs to be paused before applying DMS changes

[kamkarm]

Terraform Core Version

1.2.6

AWS Provider Version

4.38.0

Affected Resource(s)

• aws_dms_endpoint

Expected Behavior

Following this pr, I expected DMS tasks to be automatically stopped & resumed after a Terraform DMS change when start_replication_task is set to true in the aws_dms_replication_task resource.

Actual Behavior

This is the case when modifications are made to the aws_dms_replication_task resource, i.e updating the lob_max_size in the replication_task_settings parameter. However, when I made changes to the aws_dms_endpoint resource (which is tied to the task), I received the error below stating that the task needs to be paused before the changes can be applied.

It seems like the change to add start_replication_task from this pr only addressed automating stoping/resuming DMS tasks when changes are made to resource aws_dms_replication_task.

Relevant Error/Panic Output Snippet

Terraform will perform the following actions:

  # aws_dms_endpoint.dms-endpoint-target will be updated in-place
  ~ resource "aws_dms_endpoint" "dms-endpoint-target" {
        id                          = "sandbox-dms-endpoint-target"
        tags                        = {}
        # (7 unchanged attributes hidden)

      ~ s3_settings {
          ~ max_file_size                               = 1048575 -> 1048576
            # (31 unchanged attributes hidden)
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

...


Error: updating DMS Endpoint (sandbox-dms-endpoint-target): InvalidResourceStateFault: Endpoint cannot be modified with running or starting tasks.

Terraform Configuration Files

IAM policy

# Create Iam policy for DMS
data "aws_iam_policy_document" "dms-role-policy" {
  statement {
    actions = [
      "sts:AssumeRole"
    ]

    principals {
      type = "Service"
      identifiers = ["dms.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "dms-role" {
  name = "some_name"
  path = "/"
  assume_role_policy = data.aws_iam_policy_document.dms-role-policy.json
}

Changing max_file_size below

locals {
  add_column_name          = true
  bucket_folder            = "dms_target"
  cdc_max_batch_interval   = 0
  cdc_min_file_size        = 0
  compression_type         = "NONE"
  csv_delimiter            = ","
  csv_row_delimiter        = "\\n"
  csv_null_value           = ""
  data_page_size           = 0
  date_partition_delimiter = "NONE"
  date_partition_enabled   = false
  dict_page_size_limit     = 0
  encoding_type            = "plain"
  max_file_size            = 1048575
  timestamp_column_name    = "date_cdc"
  rfc_4180                 = false
  row_group_length         = 0
}

resource "aws_dms_endpoint" "dms-endpoint-target" {
  // endpoint_id can only contain alphanumeric characters or hyphens
  endpoint_id   = "some_id"
  endpoint_type = "target"
  engine_name   = "s3"

  s3_settings {
    add_column_name          = local.add_column_name
    bucket_name              = 'some_bucket_name'
    bucket_folder            = local.bucket_folder
    cdc_max_batch_interval   = local.cdc_max_batch_interval
    cdc_min_file_size        = local.cdc_min_file_size
    compression_type         = local.compression_type
    csv_delimiter            = local.csv_delimiter
    csv_null_value           = local.csv_null_value
    csv_row_delimiter        = local.csv_row_delimiter
    data_page_size           = local.data_page_size
    date_partition_delimiter = local.date_partition_delimiter
    date_partition_enabled   = local.date_partition_enabled
    dict_page_size_limit     = local.dict_page_size_limit
    encoding_type            = local.encoding_type
    max_file_size            = local.max_file_size
    service_access_role_arn  = aws_iam_role.dms-role.arn
    timestamp_column_name    = local.timestamp_column_name
    rfc_4180                 = local.rfc_4180
    row_group_length         = local.row_group_length
  }

  lifecycle {
    create_before_destroy = true
  }

}

To This

locals {
  add_column_name          = true
  bucket_folder            = "dms_target"
  cdc_max_batch_interval   = 0
  cdc_min_file_size        = 0
  compression_type         = "NONE"
  csv_delimiter            = ","
  csv_row_delimiter        = "\\n"
  csv_null_value           = ""
  data_page_size           = 0
  date_partition_delimiter = "NONE"
  date_partition_enabled   = false
  dict_page_size_limit     = 0
  encoding_type            = "plain"
  max_file_size            = 1048576
  timestamp_column_name    = "date_cdc"
  rfc_4180                 = false
  row_group_length         = 0
}

resource "aws_dms_endpoint" "dms-endpoint-target" {
  // endpoint_id can only contain alphanumeric characters or hyphens
  endpoint_id   = "${var.env}-dms-endpoint-target"
  endpoint_type = "target"
  engine_name   = "s3"

  s3_settings {
    add_column_name          = local.add_column_name
    bucket_name              = aws_s3_bucket.dms.bucket
    bucket_folder            = local.bucket_folder
    cdc_max_batch_interval   = local.cdc_max_batch_interval
    cdc_min_file_size        = local.cdc_min_file_size
    compression_type         = local.compression_type
    csv_delimiter            = local.csv_delimiter
    csv_null_value           = local.csv_null_value
    csv_row_delimiter        = local.csv_row_delimiter
    data_page_size           = local.data_page_size
    date_partition_delimiter = local.date_partition_delimiter
    date_partition_enabled   = local.date_partition_enabled
    dict_page_size_limit     = local.dict_page_size_limit
    encoding_type            = local.encoding_type
    max_file_size            = local.max_file_size
    service_access_role_arn  = aws_iam_role.dms-role.arn
    timestamp_column_name    = local.timestamp_column_name
    rfc_4180                 = local.rfc_4180
    row_group_length         = local.row_group_length
  }

  lifecycle {
    create_before_destroy = true
  }

}

Steps to Reproduce

1. Update a parmater in the aws_dms_endpoint endpoint
2. Terraform apply will fail

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

No response

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[bennylu2]

I took a quick look at this and I'm not sure it should be labeled good first issue

Whoever picks this up would need to identify all replication tasks that are connected to the endpoint and loop through them to check the status of the task, stop the task, modify the endpoint, and resume/restart all tasks afterwards

This can be used as a reference from replication_task.go

The caveat is all the replication task info is available in that context but not in endpoint.go. I think it would require implementing another function in find.go similar to FindReplicationTaskByID that can query by endpoint arn returning a list of tasks

[YakDriver]

I agree that this would be a great issue for someone to pickup but maybe not a first issue. I like the plan you outline @bennylu2. If you're feeling up to it, I think it'll work well.

[github-actions]

This functionality has been released in v5.25.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.