[Bug]: Unable to pass kerberos keytab and config as Base64 string in aws_datasync_location_hdfs resource

[lukaszkarwot]

Terraform Core Version

1.5.5

AWS Provider Version

5.13.1

Affected Resource(s)

aws_datasync_location_hdfs

Expected Behavior

As described in AWS CLI documentation, create-location-hdfs api call parameters:

• kerberos-keytab
• kerberos-krb5-conf
  can be provided either as a file or a base64-encoded text.

Actual Behavior

This feature works when calling AWS API directly, but fails when parameters are passed with aws_datasync_location_hdfs resource:

Values for "keytab"and "config" stored in secret are base64 encoded text.

After Terraform apply, resource is created without any errors, but datasync task fails to authenticate with Kerberos.
When I update the created location with AWC CLI update-location-hdfs api call providing exactly the same values from secret it is able to authenticate on process data synchronization task.

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

locals {
  datasync_agent_arn = "<DATASYNC_AGENT>"
  base64_keytab      = "<BASE_64_KEYTAB>"
  base64_krb5_conf   = "<BASE_64_KRB5_CONF>"
  principal          = "<KERBEROS_PRINCIPAL>"
  name_node          = "<HOST>:<IP>"
}

resource "aws_datasync_location_hdfs" "source" {
  count               = length(local.data_mappings)
  subdirectory        = local.data_mappings[count.index].source.path
  agent_arns          = [local.datasync_agent_arn]
  authentication_type = "KERBEROS"
  kerberos_keytab     = data.base64_keytab
  kerberos_principal  = data.principal
  kerberos_krb5_conf  = data.base64_krb5_conf
  
  name_node {
    hostname = split(":",local.name_node)[0]
    port     = split(":",local.name_node)[1]
  }

  qop_configuration {
    data_transfer_protection = "DISABLED"
    rpc_protection           = "AUTHENTICATION" 
  }
}

Steps to Reproduce

• create aws_datasync_location_hdfs resource with authenticationType = KERBEROS and provide kerberos keytab and krb5 config as base64 encoded text
• create a datasync task with created hdfs location and execute it

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

https://discuss.hashicorp.com/t/how-to-load-kerberos-keytab-in-terraform-script/47394

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[ewbankkit]

AWS SDK for Go v1

https://github.com/aws/aws-sdk-go/blob/fdce8a5fc70227970e0adffd97f3e8edee308925/service/datasync/api.go#L7294-L7301

        // The Kerberos key table (keytab) that contains mappings between the defined
        // Kerberos principal and the encrypted keys. You can load the keytab from a
        // file by providing the file's address. If you're using the CLI, it performs
        // base64 encoding for you. Otherwise, provide the base64-encoded text.
        //
        // If KERBEROS is specified for AuthenticationType, this parameter is required.
        // KerberosKeytab is automatically base64 encoded/decoded by the SDK.
        KerberosKeytab []byte `type:"blob"`

The datasync service uses the JSON RPC protocol, and []byte is base64 encoded/decoded by the SDK:

                case []byte:
                        if !value.IsNil() {
                                buf.WriteByte('"')
                                if len(converted) < 1024 {
                                        // for small buffers, using Encode directly is much faster.
                                        dst := make([]byte, base64.StdEncoding.EncodedLen(len(converted)))
                                        base64.StdEncoding.Encode(dst, converted)
                                        buf.Write(dst)
                                } else {
                                        // for large buffers, avoid unnecessary extra temporary
                                        // buffer space.
                                        enc := base64.NewEncoder(base64.StdEncoding, buf)
                                        enc.Write(converted)
                                        enc.Close()
                                }
                                buf.WriteByte('"')
                        }

AWS SDK for Go v2

https://github.com/aws/aws-sdk-go-v2/blob/10c44870fbdabbbac25ffe1a28115efe4fd560d9/service/datasync/api_op_CreateLocationHdfs.go#L59-L64

        // The Kerberos key table (keytab) that contains mappings between the defined
        // Kerberos principal and the encrypted keys. You can load the keytab from a file
        // by providing the file's address. If you're using the CLI, it performs base64
        // encoding for you. Otherwise, provide the base64-encoded text. If KERBEROS is
        // specified for AuthenticationType , this parameter is required.
        KerberosKeytab []byte

We are currently using AWS SDK for Go v1 for the datasync service so as the Terraform kerberos_keytab attribute is a string, we need to remove the base64 encoding.

Maybe add a kerberos_keytab_base64 attribute?
#4197 (comment).

[github-actions]

This functionality has been released in v5.42.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.