Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(integ-tests-alpha): awsApiCall failing for Kinesis putRecord - cannot properly pass Data #26798

Closed
postsa opened this issue Aug 18, 2023 · 4 comments · Fixed by #27009
Closed
Assignees
Labels
@aws-cdk/aws-kinesis Related to Amazon Kinesis bug This issue is a bug. effort/medium Medium work item – several days of effort node18-upgrade Any work (bug, feature) related to Node 18 upgrade p1 sdk-v3-upgrade Tag issues that are associated to SDK V3 upgrade. Not limited to CR usage of SDK only.

Comments

@postsa
Copy link
Contributor

postsa commented Aug 18, 2023

Describe the bug

When I pass an expected Uint8Array to the params of Kinesis.putRecord I receive the failure message

Received response status [FAILED] from custom resource. Message returned: Cannot read properties of undefined (reading 'byteLength')

Expected Behavior

Put record should succeed with the input expected by the PutRecordCommandInput interface

Current Behavior

Here is the relevant log from the assertion lambda:

2023-08-18T00:01:41.346Z	5ad0ff87-aaa8-4784-9458-8f1de4e21e51	INFO	SDK request to Kinesis.putRecord with parameters 
{
    "StreamName": "KinesisEventProcessingIntegrationStack",
    "PartitionKey": "testKey",
    "Data": {
        "0": 123,
        "1": 34,
        "2": 110,
        "3": 97,
        "4": 109,
        "5": 101,
        "6": 34,
        "7": 58,
        "8": 34,
        "9": 119,
        "10": 111,
        "11": 114,
        "12": 108,
        "13": 100,
        "14": 34,
        "15": 125
    }
}

I think this is because the parameters are encoded with JSON.stringify which turns a Uint8Array into a JSON

> data
Uint8Array(16) [
  123,  34, 110,  97, 109,
  101,  34,  58,  34, 119,
  111, 114, 108, 100,  34,
  125
]
> JSON.stringify(data)
'{"0":123,"1":34,"2":110,"3":97,"4":109,"5":101,"6":34,"7":58,"8":34,"9":119,"10":111,"11":114,"12":108,"13":100,"14":34,"15":125}'

Reproduction Steps

Create a test stack with a kinesis stream and put a record on it with assertions.awsApiCall

 const kinesisStream: IStream = new Stream(stack, "TestKinesisStream");

 const buf = Buffer.from(JSON.stringify({ name: "world" }));
 const data = new Uint8Array(
   buf.buffer,
   buf.byteOffset,
   buf.byteLength / Uint8Array.BYTES_PER_ELEMENT
 );

 testCase.assertions.awsApiCall("Kinesis", "putRecord", {
   StreamName: kinesisStream.streamName,
   Data: data,
   PartitionKey: "testKey",
 });

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.93.0

Framework Version

No response

Node.js Version

18.12.0

OS

MacOS 13.5 (22G74)

Language

Typescript

Language Version

No response

Other information

No response

@postsa postsa added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Aug 18, 2023
@github-actions github-actions bot added the @aws-cdk/aws-kinesis Related to Amazon Kinesis label Aug 18, 2023
@postsa postsa changed the title (integ-runner-alpha): (awsApiCall failing for Kinesis putRecord - cannot properly pass Data) (integ-tests-alpha): (awsApiCall failing for Kinesis putRecord - cannot properly pass Data) Aug 18, 2023
@peterwoodworth
Copy link
Contributor

Looks to be same as the issue this PR fixed #26797

@peterwoodworth peterwoodworth added p1 effort/medium Medium work item – several days of effort and removed needs-triage This issue or PR still needs to be triaged. labels Aug 18, 2023
@postsa postsa changed the title (integ-tests-alpha): (awsApiCall failing for Kinesis putRecord - cannot properly pass Data) (integ-tests-alpha): awsApiCall failing for Kinesis putRecord - cannot properly pass Data Aug 24, 2023
@postsa
Copy link
Contributor Author

postsa commented Aug 24, 2023

Looks to be same as the issue this PR fixed #26797

Excellent, I will upgrade, verify, and close the issue if so. Thanks.

@postsa
Copy link
Contributor Author

postsa commented Aug 28, 2023

Upgraded to 2.93.0, but the issue persists

@mrgrain mrgrain added node18-upgrade Any work (bug, feature) related to Node 18 upgrade sdk-v3-upgrade Tag issues that are associated to SDK V3 upgrade. Not limited to CR usage of SDK only. labels Aug 31, 2023
@otaviomacedo otaviomacedo self-assigned this Sep 4, 2023
@mergify mergify bot closed this as completed in #27009 Sep 6, 2023
mergify bot pushed a commit that referenced this issue Sep 6, 2023
AWS SDK v3 has changed some data types to Uint8Array. If a value of this type was provided by the user, it would not be decoded properly back to a Uint8Array before making the call.

Recognize when there is an encoded Uint8Array and decode it back properly.

Closes #26798.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@github-actions
Copy link

github-actions bot commented Sep 6, 2023

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

mikewrighton pushed a commit that referenced this issue Sep 14, 2023
AWS SDK v3 has changed some data types to Uint8Array. If a value of this type was provided by the user, it would not be decoded properly back to a Uint8Array before making the call.

Recognize when there is an encoded Uint8Array and decode it back properly.

Closes #26798.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-kinesis Related to Amazon Kinesis bug This issue is a bug. effort/medium Medium work item – several days of effort node18-upgrade Any work (bug, feature) related to Node 18 upgrade p1 sdk-v3-upgrade Tag issues that are associated to SDK V3 upgrade. Not limited to CR usage of SDK only.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants