Integrate data account S3 events to OrcaBus event source SQS

[victorskl]

Context:

Find a solution to send S3 events from data account to beta(dev)|gamma(stg)|prod.

Couple of ways possible with EventBridge to EventBridge cross accounts integration.

See
https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-cross-account.html

Actions:

• Integrate it with current data account bucket setup
• Route data account S3 events to OrcaBus event-source SQS target

[victorskl]

This integration task in the critical path for OrcaBus release Phase 1.

[victorskl]

Amazon EventBridge - Routing events to buses in other AWS accounts
https://www.youtube.com/watch?v=pX_xIW_EuCE

[victorskl]

Simplifying cross-account access with Amazon EventBridge resource policies
https://aws.amazon.com/blogs/compute/simplifying-cross-account-access-with-amazon-eventbridge-resource-policies/

[reisingerf]

Started working on this on the feature/data-s3-event-forward branch

[reisingerf]

This will need a resource policy counterpart on the target event bus(es) to allow PutEvents from the source (data) account
=> OrcaBus CDK update

[reisingerf]

Changes to OrcaBus event bus are on the update/x-account-events branch of the orcabus repo.

[victorskl]

Let us pair coding and complete this together tomorrow, Flo.

[victorskl]

Caveat 1:

• When we integrate our BYOB bucket to ICAv2 project as underlay storage configuration, ICAv2 subscribes couple of bucket notifications to our buckets.
• These bucket notification subscription are not in our control.
• When we also want to integrate our bucket notification to EventBridge, we need to turn it on the boolean flag.
• The relevant terraform syntax are as follows.

infrastructure/terraform/stacks/data_archive/byob_ica_v2.tf

Lines 511 to 515 in 4ac93e0

	# NOTE: don't control notification settings from TF, as some is controlled by ICA
	# resource "aws_s3_bucket_notification" "development_data" {
	# bucket = aws_s3_bucket.development_data.id
	# eventbridge = true
	# }

• Note that we have commented out the TF code. The reason being as follows.

NOTE:
S3 Buckets only support a single notification configuration. Declaring multiple aws_s3_bucket_notification resources to the same S3 Bucket will cause a perpetual difference in configuration. See the example "Trigger multiple Lambda functions" for an option.
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_notification

• Basically we do not wish to undo nor clash with bucket notification with ICAv2 BYOB integration.
• For now, we simply handle turning on EventBridge bucket notification manually.

[victorskl]

Caveat 2:

• Refer to AWS doc, we are following EventBridge to EventBridge cross account integration.
• There are source account and target account.
• At source account (UoM data account) we configure terraform to forward S3 events to target. These code block.
• At target account EventBridge, we still have to allow events from the source. This has to configure permission policy at designated target bus. In this case, we are targeting to "default" bus at destination. As follows.

infrastructure/terraform/stacks/data_archive/byob_ica_v2.tf

Line 17 in 4ac93e0

target_event_bus_arn_dev = "arn:aws:events:ap-southeast-2:${local.account_id_dev}:event-bus/default"

• Since destination bus is AWS EventBridge "default" bus, we configure the permission, manually. As follows.

AWS Console > EventBridge > Event buses > default > Permissions

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowDataAccountToPutEvents",
    "Effect": "Allow",
    "Principal": {
      "AWS": "arn:aws:iam::<DATA_ACCOUNT_ID>:root"
    },
    "Action": "events:PutEvents",
    "Resource": "arn:aws:events:ap-southeast-2:<DEV_ACCOUNT_ID>:event-bus/default"
  }]
}