Support to reference DynamoDB table latest stream ARN in Lambda EventSourceMapping

[ghorak-happening]

Problem

DynamoDB Latest Stream ARN is autogenerated by AWS and cannot be constructed upfront. That makes it very difficult to create AWS Lambda EventSourceMapping with DynamoDB Stream as a source.

The EventSourceARN supports several AWS resources, as per documentation (https://github.com/aws-controllers-k8s/lambda-controller/blob/6a1f71e1b68480530e9a9ac1389b8a0f7bc30499/apis/v1alpha1/event_source_mapping.go#L69):

// The Amazon Resource Name (ARN) of the event source.
//
//   - Amazon Kinesis – The ARN of the data stream or a stream consumer.
//
//   - Amazon DynamoDB Streams – The ARN of the stream.
//
//   - Amazon Simple Queue Service – The ARN of the queue.
//
//   - Amazon Managed Streaming for Apache Kafka – The ARN of the cluster.
//
//   - Amazon MQ – The ARN of the broker.
EventSourceARN *string                                  `json:"eventSourceARN,omitempty"`

Unfortunately, the EventSourceRef is implemented to support only managed kafka cluster ARNs:

https://github.com/aws-controllers-k8s/lambda-controller/blob/6a1f71e1b68480530e9a9ac1389b8a0f7bc30499/apis/v1alpha1/generator.yaml#L137

I'm happy to contribute this feature, but would need direction towards best solution.

Possible solutions

1. Re-use existing EventSourceRef field.

   To ignore defined field reference kind in the generator.yaml and implement custom logic to resolve proper ARN.
   This would need to be done as custom code and would clash with code generators. Not sure if this is even possible,

2. Create xxxEventSourceRef fields for each reference kind.

   ARN resolving would need to be done with hooks in this case.

3. Extend the core code-generator to support multiple resource kinds as field references

   Something like this:

   EventSourceARN:
     references:
       oneOf:
       - resource: Cluster
         path: Status.ACKResourceMetadata.ARN
         service_name: kafka
       - resource: Table
         path: Status.latestStreamARN
         service_name: dynamodb

   ACK Custom resource names are unique within namespace, regardless of the resource kind, so the implementation should be straightforward.

[a-hilaly]

Hi @ghorak-happening - thanks for bringing up this issue! We're currently tracking dynamic resource references (option3 ) in #1576 - it should be possible to implement by leveraging k8s dynamic clients and relying on imports such as https://github.com/aws-controllers-k8s/lambda-controller/blob/main/go.mod#L6. Feel free to pick up the issue if this is something you'd like to contribute to!
Also community meeting is a good place to bring up conversations/questions https://github.com/aws-controllers-k8s/lambda-controller/blob/main/go.mod#L6

[ghorak-happening]

Hi @a-hilaly, I like the idea. Let me dig a bit deeper into the #1576 to see if I'd be able to pick it up.

[ack-bot]

Issues go stale after 180d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 60d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/aws-controllers-k8s/community.
/lifecycle stale

[a-hilaly]

Closing in favor of #1576
/duplicate
/close

[ack-prow]

@a-hilaly: Closing this issue.

In response to this:

Closing in favor of #1576
/duplicate
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.