Support for EC2 Transit Gateway VPN Attachment

[kdegraaf]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Based on the existence of the ec2_transit_gateway_vpc_attachment data source and resource type, I was expecting to find a corresponding data source and resource type called ec2_transit_gateway_vpn_attachment. They do not appear to exist.

This caused three issues:

1. I spent time finding a workaround: creating an aws_vpn_connection resource with the transit_gateway_id attribute pointing to my transit gateway.
2. I was unable to manage the tags on the resulting tgw-attach-xxxxxxxx resource, such as the Name tag and other tags required by my organization.
3. I was unable to supply the tgw-attach-xxxxxxxx ID to the aws_ec2_transit_gateway_route_table_association and aws_ec2_transit_gateway_route_table_propagation resources by reference. Instead, I copy-pasted these IDs into the code after standing up the VPN attachment.

If I've missed something, please feel free to point me in the right direction. Thanks!

[bflad]

I don't think you're missing anything from what I remember from the initial development of the EC2 Transit Gateway functionality.

The current EC2 API has explicit API methods for managing Transit Gateway VPC attachments, but not VPN attachments. If I had to guess about the intention, its that VPN attachments are implicitly created/managed by VPN connections and it would introduce an awkward workflow to create/manage them outside of that. W certainly have a gap in the Terraform AWS provider though since there is no way to find/reference the VPN attachment IDs. I think there are few approaches that may be viable here:

Creating a aws_ec2_transit_gateway_vpn_attachment data source

Using DescribeTransitGatewayAttachments with filtering on resource-type vpn and resource-id presumably being the VPN connection ID or something similar (sorry I forget off the exact value required off top of my head).

An example configuration may look like:

# Design sketch, this configuration is not implemented and may change during discussion/development
resource "aws_vpn_connection" "example" {
  customer_gateway_id = "${aws_customer_gateway.example.id}"
  transit_gateway_id  = "${aws_ec2_transit_gateway.example.id}"
  type                = "${aws_customer_gateway.example.type}"
}

data "aws_ec2_transit_gateway_vpn_attachment" "example" {
  vpn_connection_id = "${aws_vpn_connection.example.id}"
}

resource "aws_ec2_transit_gateway_route_table_association" "example" {
  transit_gateway_attachment_id  = "${data.aws_ec2_transit_gateway_vpn_attachment.example.id}"
  transit_gateway_route_table_id = "${aws_ec2_transit_gateway_route_table.example.id}"
}

Baking the Transit Gateway Attachment ID Lookup into aws_vpn_connection

From a workflow perspective this option seems like it would be more preferable, as you get a configuration like:

# Design sketch, this configuration is not implemented and may change during discussion/development
resource "aws_vpn_connection" "example" {
  customer_gateway_id = "${aws_customer_gateway.example.id}"
  transit_gateway_id  = "${aws_ec2_transit_gateway.example.id}"
  type                = "${aws_customer_gateway.example.type}"
}

resource "aws_ec2_transit_gateway_route_table_association" "example" {
  transit_gateway_attachment_id  = "${aws_vpn_connection.example.transit_gateway_attachment_id}"
  transit_gateway_route_table_id = "${aws_ec2_transit_gateway_route_table.example.id}"
}

Although I think there are a few potential gotchas worth considering:

• Attempting to read Transit Gateway information might be fairly unexpected within the context of the VPN connection resource for operators (e.g. setting up IAM permissions)
• I believe VPN connections can be setup to cross-account Transit Gateways, so attempting to read the Transit Gateway information from the wrong account may only yield access/resource not found errors in this scenario. I could be very wrong though! 😄

---

Tagging is more problematic, but we've discussed in the past potentially supporting something like an aws_ec2_tag resource for exactly these strange situations, e.g. #5898 (comment)

# Design sketch, this configuration is not implemented and may change during discussion/development
resource "aws_ec2_tag" "example" {
  resource_id = "" # (Required, ForceNew) ami-12345678, i-12345678, etc
  key = "" # (Required, ForceNew)
  value = "" # (Required)
}

Since in this case it seems the tagging ask is separate from the attachment ID referencing ask (due to the API), I would suggest commenting/upvoting/subscribing to #5898 for that. 👍

---

Hope this brain dump and context helps. 😅

[juanbecerra]

Same issue here, hoping for an update soon...

[juanbecerra]

any updates?

[juanbecerra]

the current API allows to get the list of attachments and filter them by type. gateway ID and VPN ID... I think terrraform implements the datasource from "DescribeTransitGatewayVpcAttachments" but it could also add "DescribeTransitGatewayAttachments" it will be something like:

DescribeTransitGatewayAttachments (resource-id=>VPN_ID, resourceType=>vpn, transitGatewayId=>transitGatewayId)

https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeTransitGatewayAttachments.html

[nyrider]

Same issue here. Great solution ideas above.

[piersf]

Are there any plans to support Transit Gateway VPN Attachments in the near future?
Currently, this can be done only manually through the AWS Console.

[piersf]

You can implicitly create the VPN Attachment by creating the aws_vpn_connection resource and passing a transit_gateway_id instead of a vpn_gateway_id.
If you do that, the VPN attachment gets created. But you have no way of getting the Attachment's ID so you can do further stuff with it, like association and propagation.

And the fact that neither the aws_vpn_connection or the aws_customer_gateway resources expose/output the Attachment ID, doesn't make it easy. So we really need either a datasource to get the Attachment ID or a resource that does the VPN attachment.

[piersf]

So, as this became kind of critical for my team, I implemented a workaround that works almost perfectly.
As I wrote in the previous comment above, you can implicitly create the VPN Attachment(s) by creating the aws_vpn_connection resource and passing a transit_gateway_id instead of a vpn_gateway_id for an attribute.
After doing this, however, we still need that ID's of these VPN Attachments that were created implicitly so we can associate with and propagate to a routing table. Unfortunately, there's no Terraform data source that could give us this information nor does the aws_vpn_connection output the VPN Attachment ID(if it has been passed a transit_gateway_id attribute as stated above).

At this point, Bash came to the rescue. I was able to overcome this blocker by using an external data source that executes a Bash script. This bash script, in turn, executes the AWS CLI aws ec2 describe-transit-gateway-attachments command to get the ID's of the VPN Attachment given a TGW ID and few other parameters. Here's how one can achive this:

./modules/tgw-vpn-attachment/main.tf:

locals {
  # TEMP - until Terraform supports VPN Attachment resource - the below 'split' method converts the returned comma separated string into a list
  transit_gateway_vpn_attachment_ids = ["${split(",", data.external.get_transit_gateway_vpn_attachment_id.result.transit_gateway_ids)}"]
}

data "external" "get_transit_gateway_vpn_attachment_id" {
  program = ["bash", "${path.module}/get-transit-gateway-vpn-attachment-id.sh"]

  query = {
    # arbitrary map from strings to strings, passed to the external program(script being executed) as the data query - the script must return a valid JSON object.
    aws_profile = "${var.application_key}"
    aws_region = "${var.aws_region}"
    shared_credentials_file = "${var.shared_credentials_file}"
    resource_type = "${var.awscli_filter_resource_type}"
    transit_gateway_id = "${var.transit_gateway_id}"
  }
}

resource "aws_vpn_connection" "vpn_connection-tgw-vpn-attachment" {
  # Creates implicitly TGW-VPN Attachments because we pass a 'transit_gateway_id'
  count 				= "${length(var.customer_gateway_ids) == 0 ? 1 : length(var.customer_gateway_ids)}"
  transit_gateway_id 	= "${var.transit_gateway_id}"
  customer_gateway_id 	= "${var.customer_gateway_ids[count.index]}"
  type 					= "${var.vpn_conection_type_for_implicit_tgw_vpn_attachment}"
  static_routes_only 	= "${var.vpn_conection_static_routes_for_implicit_tgw_vpn_attachment}"
  tags 					= "${var.vpn_conection_tags_for_implicit_tgw_vpn_attachment}"
}


resource "aws_ec2_transit_gateway_route_table_association" "transit_gateway_route_table_association" {
  count 							= "${length(local.transit_gateway_vpn_attachment_ids)}"
  transit_gateway_attachment_id 	= "${local.transit_gateway_vpn_attachment_ids[count.index]}"
  transit_gateway_route_table_id 	= "${module.transit_gateway_route_table_vpn.id}"
  depends_on 						= ["aws_vpn_connection.vpn_connection-tgw-vpn-attachment", "module.transit_gateway_route_table_vpn"]
}


resource "aws_ec2_transit_gateway_route_table_propagation" "transit_gateway_route_table_propagation" {
  count 							= "${length(local.transit_gateway_vpn_attachment_ids)}"
  transit_gateway_attachment_id 	= "${local.transit_gateway_vpn_attachment_ids[count.index]}"
  transit_gateway_route_table_id 	= "${module.transit_gateway_route_table_vpn.id}"
  depends_on 						= ["aws_vpn_connection.vpn_connection-tgw-vpn-attachment", "module.transit_gateway_route_table_vpn"]
}

./my-env/main.tf - note that some values below come from other modules that are not shown here:

module "transit-gateway-vpn-attachment-setup" {
  source = "../modules/tgw-vpn-attachment"
  
  customer_gateway_ids = "${module.vpn.customer_gateway_ids}"
  transit_gateway_id = "${module.transit-gateway.transit_gateway_id}"
  vpn_conection_type_for_implicit_tgw_vpn_attachment = "ipsec.1"
  vpn_conection_static_routes_for_implicit_tgw_vpn_attachment = "false"
  vpn_conection_tags_for_implicit_tgw_vpn_attachment = "${map("Name", "VPN-Transit-Gateway-Attachment")}"
  application_key = "${var.application_key}"
  aws_region = "${var.aws_region}"
  shared_credentials_file = "$HOME/${var.shared_credentials_file}"
  awscli_filter_resource_type = "vpn"
  transit_gateway_route_table_vpn_tags = "${map("Name", "Transit-Gateway-Routing-Table_TO_VPN", "Created", "24-01-2019")}"
}

The Bash ./modules/tgw-vpn-attachment/get-transit-gateway-vpn-attachment-id.sh script which returns the VPN Attachment IDs:

#!/bin/bash

#set -eu -o pipefail

function quit() {
	echo "$1" 1>&2
	exit 1
}

function verify_dependecies() {
	test -f $(which aws) || quit "AWS CLI command not detected in path, please install it. Version 1.16.90 or above required"
	test -f $(which jq) || quit "jq command not detected in path, please install it: wget -O jq https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 && chmod +x ./jq && cp jq /usr/bin"
}

function process_vars_from_terraform() {
	# Verify variables required by AWS CLI
	eval "$(jq -r '@sh "export AWS_PROFILE=\(.aws_profile) AWS_REGION=\(.aws_region) AWS_SHARED_CREDENTIALS_FILE=\(.shared_credentials_file) AWS_CONFIG_FILE=\(.shared_credentials_file) RESOURCE_TYPE=\(.resource_type) TRANSIT_GATEWAY_ID=\(.transit_gateway_id)"')"
	if [[ -z "${RESOURCE_TYPE}" ]]; then export RESOURCE_TYPE="NULL"; fi
	if [[ -z "${TRANSIT_GATEWAY_ID}" ]]; then export TRANSIT_GATEWAY_ID="NULL"; fi
	if [[ -z "${AWS_PROFILE}" ]]; then export AWS_PROFILE="NULL"; fi
	if [[ -z "${AWS_REGION}" ]]; then export AWS_REGION="NULL"; fi
	if [[ -z "${AWS_SHARED_CREDENTIALS_FILE}" ]]; then export AWS_SHARED_CREDENTIALS_FILE="NULL"; fi
	if [[ -z "${AWS_CONFIG_FILE}" ]]; then export AWS_CONFIG_FILE="NULL"; fi
}

function output_for_terraform_datasource() {
	# without the use of --query: aws ec2 describe-transit-gateway-attachments --profile $AWS_PROFILE --region $AWS_REGION --filters "Name=transit-gateway-id,Values=${TRANSIT_GATEWAY_ID},Name=resource-type,Values=${RESOURCE_TYPE},Name=state,Values=available" --output json | jq -r '.TransitGatewayAttachments[].TransitGatewayAttachmentId'
	
	# For some reason when adding to the filter 'Name=state,Values=available' it also returns 'VPC' type of Attachments. Therefore, we use also '--query' for further filtering.
	# But the jq -r flag changes a bit when including '--query'. See above how the jq -r flag is without use of '--query'
	TRANSIT_GATEWAY_VPN_ATTACHMENT_IDS=$(echo $(aws ec2 describe-transit-gateway-attachments --profile $AWS_PROFILE --region $AWS_REGION \
  											   		--filters "Name=transit-gateway-id,Values=${TRANSIT_GATEWAY_ID},Name=resource-type,Values=${RESOURCE_TYPE},Name=state,Values=available" \
  											   		--query 'TransitGatewayAttachments[?starts_with(ResourceId, `vpn-`) == `true` && ResourceType != `vpc`]' \
  											   		--output json | jq -r '.[].TransitGatewayAttachmentId') | tr -d '\n' | sed 's/ /,/g')
  
  	jq -n --arg transit_gateway_ids "$TRANSIT_GATEWAY_VPN_ATTACHMENT_IDS" '{transit_gateway_ids:$transit_gateway_ids}'
}

function catch_errors() {
	errcode=$?
	read line file <<<$(caller)
    printf "Failed to get VPN Attachment IDs! Error (code: $errcode) executing command [> $BASH_COMMAND <] in line ${BASH_LINENO[0]} of file $file" >&2 && exit $errcode
}

trap catch_errors ERR

verify_dependecies
process_vars_from_terraform
output_for_terraform_datasource

Brief Explanation:
What happanes here is that we are using the external data source to execute a Bash script and we pass several arguments that the script requires. Then it uses those arguments to execute the AWS CLI ec2 describe-transit-gateway-attachments command given an TGW ID and other parameters. Then the script uses the jq command to build a JSON object to return back. This is important because the external data source expects to receive back a valid JSON object result. Read more here: https://www.terraform.io/docs/providers/external/data_source.html

Once the data source has executed the script succesffuly and received back the JSON result, then you can access that result from other resources just as with any other data source.

One thing that i wasn't able to do was to tag the VPN Attachment(s) created implicitly. The AWS CLI does not support this for an existing VPN Attachment(s). Only if you create a new one you can tag it. If somebody knows of a way, then please comment here on how to do it.

Hope this helps somebody.

[piersf]

Please note that in the above ./modules/tgw-vpn-attachment/main.tf file, i have changed the following line inside the locals block
From

transit_gateway_vpn_attachment_ids = ["${split(",", data.external.get_transit_gateway_vpn_attachment_id.result.transit_gateway_ids)}"]

To

transit_gateway_vpn_attachment_ids = ["${split(",", data.external.get_transit_gateway_vpn_attachment_id.result["transit_gateway_ids"])}"]

This is because if the aws_vpn_connection connection resource(s) have not been created beforehand and we use the syntax result.transit_gateway_ids, the variable inside the locals block will return an empty string which will be converted to an empty list. And that will result in all sort of errors in the association and propagation resources. But if we use the syntax result["transit_gateway_ids"], this somehow works correctly(discovered the solution for this in #16762 ).

Also another thing i discovered while moving things around, is that if you make the external datasource to depend_on the aws_vpn_connection resource, it will trigger a recreation of the aws_ec2_transit_gateway_route_table_association and aws_ec2_transit_gateway_route_table_propagation resources with every plan/apply action.

[piersf]

if you add
vpn_attachment_ids = "${aws_vpn_connection.vpn_connection_tgw_vpn_attachment.*.id[0]}" to the query block of the external data source, you'll be able to solve the recreation issue i describe above.

This is just to force the data source to wait on the attachment creation. The bash script does not really use the vpn_attachment_ids variable that we are passing to it.

[dbektas]

is anyone working on this? Any updates?

[dbektas]

be possible to use data source provider (https://www.terraform.io/docs/providers/aws/d/ec2_transit_gateway_vpc_attachment.html)

to actually filtrate using vpn-id (https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeTransitGatewayAttachments.html)

as suggested in previous comments by @juanbecerra.

[AzCii]

I think as @bflad suggested, the aws_ec2_transit_gateway_vpn_attachment data resource needs to be created.

Are anyone working on solving this issue?

[wrsuarez]

This is sorely needed and is a blocker for implementing terraform for TGW for anyone using VPN's. We have a urgent need for this

[piersf]

@wrsuarez above you can find a workaround we ended implementing since this was urgent for us as well until TGW VPN Attachment with Terraform is supported. The workaround works pretty well so far.

[juanbecerra]

anyone looking at this issue? the work around is only a temporary solution and this should be addressed in the core code

[m4h3]

@piersf your workaround works but somehow apply may need to be executed twice as the transit_gateway_attachment_id are not immediately available and the script returns an empty result on the first initial run.

In order to modify this behavior:
--filters "Name=transit-gateway-id,Values=${TRANSIT_GATEWAY_ID},Name=resource-type,Values=${RESOURCE_TYPE},Name=state,Values=available"
modified to:
--filters "Name=transit-gateway-id,Values=${TRANSIT_GATEWAY_ID},Name=resource-type,Values=${RESOURCE_TYPE},Name=state,Values=available,pending"

[muillome]

Hello,
Still no luck on this issue ?
Will try the workaround in a few days but i'd rather get the "real" code.

[bluemalkin]

Agreed I'm stuck with this, the solution above is way too hacky. So I've just had to hard-code the attachment-IDs created via aws_vpn_connection.

[mrsaraiva]

@bluemalkin for me the bash script worked much better than having to create the attachments manually and hard-coding the IDs. I do agree that official support is needed, though.

[bluemalkin]

@mrsaraiva Terraform can do the attachment automatically with aws_vpn_connection
I provide the attachment ID as a variable for the associations later. That's what I do whilst I wait for the new resource

[isaacblum]

+1 for default support

[mrsaraiva]

@bluemalkin i'm doing the attachments the same way, but i need the script to be able to get the VPN attachment id, which i use to add routes to the tgw route table.

[ghost]

+1 for doing this.

Although the workaround looks very clever, it's a terrible hack.
Without this creating shared VPNs is a nightmare.

[bflad]

Pull request submitted for adding a transit_gateway_attachment_id attribute to the aws_vpn_connection resource: #8070

We'll likely also implement a new aws_ec2_transit_gateway_vpn_attachment data source as well.

[soumitmishra]

+1

[bflad]

The addition of the transit_gateway_attachment_id attribute in the aws_vpn_connection resource has been merged and will release with version 2.4.0 of the Terraform AWS Provider, likely later this week.

To compliment other environments where a data source would be more convenient, a pull request for a new aws_ec2_transit_gateway_vpn_attachment data source has been submitted here: #8071

[bflad]

The new data source has been merged as well and will release with version 2.4.0 of the Terraform AWS Provider later this week. 👍

For additional feature requests or bug reports please open new GitHub issues. Thanks!

[NicMaze]

TGW has been the reason I'm giving terraform a shot, CloudFormation has similar issues plus a couple more like being unable to attach the TGW to the VPC route table. Anyway, the quick fix on this issue is great news and I'm officially sold on Terraform. Thank you for this.

[bflad]

The new data source and new attribute have been released in version 2.4.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For any additional feature requests or bug reports, please open a new GitHub issue. 👍

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!