Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use sts to construct stream arn #463

Merged
merged 3 commits into from
Dec 23, 2022
Merged

Conversation

zengyu714
Copy link
Contributor

@zengyu714 zengyu714 commented Dec 19, 2022

Description of changes:

  1. In order for users to benefit from account endpoints, StreamARN is included into both ListShards and PutRecords requests
  2. Updated few dependencies to the most recent versions: aws-sdk-cpp. curl, and openssl, zlib
  3. Added the STS dependency for building the StreamARN
    • Include STS endpoint and STS port in the configuration, and protobuf is updated accordingly
    • When the STS call fails, the KPL would exit. Therefore we can assure that KPL is running with StreamARNs
  4. Fixed issue

Testing

  • Compiled in OSX, and tested with amazon-kinesis-producer-sample
[kpl-daemon-0003] INFO com.amazonaws.services.kinesis.producer.LogInputStreamReader - [2022-12-19 10:55:35.409865] [0x0000fe11][0x0000700009aea000] [info] [AWS Log: WARN](STSAssumeRoleWithWebIdentityCredentialsProvider)Token file must be specified to use STS AssumeRole web identity creds provider.
[kpl-daemon-0003] INFO com.amazonaws.services.kinesis.producer.LogInputStreamReader - [2022-12-19 10:55:36.023747] [0x0000fe11][0x0000700009aea000] [info] [pipeline.h:218] StreamARN "arn:aws:kinesis:us-east-1:448057645220:stream/kpltest" has been successfully configured, and will be used in requests including ListShards and PutRecords
[kpl-daemon-0003] INFO com.amazonaws.services.kinesis.producer.LogInputStreamReader - [2022-12-19 10:55:36.025112] [0x0000fe11][0x0000700009aea000] [info] [shard_map.cc:89] Updating shard map for stream "kpltest"
[pool-1-thread-1] INFO com.amazonaws.services.kinesis.producer.sample.SampleProducer - Put 2000 of 200000 so far (1.00 %), 0 have completed (0.00 %)
[pool-1-thread-1] INFO com.amazonaws.services.kinesis.producer.sample.SampleProducer - Oldest future as of now in millis is 994
[kpl-daemon-0003] INFO com.amazonaws.services.kinesis.producer.LogInputStreamReader - [2022-12-19 10:55:36.626632] [0x0000fe11][0x000070000a191000] [info] [shard_map.cc:152] Successfully updated shard map for stream "kpltest" (arn: "arn:aws:kinesis:us-east-1:448057645220:stream/kpltest"). Found 4 shards
[pool-1-thread-1] INFO com.amazonaws.services.kinesis.producer.sample.SampleProducer - Put 4000 of 200000 so far (2.00 %), 0 have completed (0.00 %)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Comment on lines +214 to +215
arn << "arn:" << sts_arn.GetPartition() << ":kinesis:" << region << ":" << result.GetAccount()
<< ":stream/" << stream_name;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this stream arn format same for all regions? I thought we had some issues with arn format in some regions?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good call out, we once had a problem with pod1, but I think the format is good for all public commercial regions

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, we should still test to make sure this works in all partitions.

Copy link
Contributor

@shanmsac shanmsac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Comment on lines +214 to +215
arn << "arn:" << sts_arn.GetPartition() << ":kinesis:" << region << ":" << result.GetAccount()
<< ":stream/" << stream_name;

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, we should still test to make sure this works in all partitions.

return arn_str;
}

LOG(warning) << "Failed to get StreamARN using STS GetCallerIdentity with exception: "

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should throw if this fails.

that will avoid operating in dual-mode, and have all applications start using ARN from this version onwards.

@shanmsac shanmsac self-requested a review December 23, 2022 06:37
Copy link
Contributor

@shanmsac shanmsac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@zengyu714 zengyu714 merged commit 934e697 into awslabs:master Dec 23, 2022
@zengyu714 zengyu714 deleted the streamarn-dev branch December 23, 2022 23:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants