Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Paginators return error when next token is not present #405

Closed
imp opened this issue Jan 18, 2022 · 7 comments
Closed

Paginators return error when next token is not present #405

imp opened this issue Jan 18, 2022 · 7 comments
Assignees
@rcoh
Labels
bug This issue is a bug.

Comments

@imp
Copy link
Contributor

imp commented Jan 18, 2022 •
edited
Loading

What is the problem?

While migrating my code that does various ec2::describe_XXX calls to use the new paginator available in 0.4.x I am hitting the

Error: failed to construct request: next token did not change, aborting paginator. This indicates an SDK or AWS service bug.

It feels like it happens when there are only a few items to fetch from the service, but I have no clear proof for it. Looping manually passing tokens from the previous call to the next one works as expected.

Version

├── aws-config v0.4.1
│   ├── aws-http v0.4.1
│   │   ├── aws-smithy-http v0.34.1
│   │   │   ├── aws-smithy-types v0.34.1
│   │   ├── aws-smithy-types v0.34.1 (*)
│   │   ├── aws-types v0.4.1
│   │   │   ├── aws-smithy-async v0.34.1
│   │   │   ├── aws-smithy-types v0.34.1 (*)
│   ├── aws-sdk-sts v0.4.1
│   │   ├── aws-endpoint v0.4.1
│   │   │   ├── aws-smithy-http v0.34.1 (*)
│   │   │   ├── aws-types v0.4.1 (*)
│   │   ├── aws-http v0.4.1 (*)
│   │   ├── aws-sig-auth v0.4.1
│   │   │   ├── aws-sigv4 v0.4.1
│   │   │   │   ├── aws-smithy-http v0.34.1 (*)
│   │   │   ├── aws-smithy-http v0.34.1 (*)
│   │   │   ├── aws-types v0.4.1 (*)
│   │   ├── aws-smithy-async v0.34.1 (*)
│   │   ├── aws-smithy-client v0.34.1
│   │   │   ├── aws-smithy-async v0.34.1 (*)
│   │   │   ├── aws-smithy-http v0.34.1 (*)
│   │   │   ├── aws-smithy-http-tower v0.34.1
│   │   │   │   ├── aws-smithy-http v0.34.1 (*)
│   │   │   ├── aws-smithy-types v0.34.1 (*)
│   │   ├── aws-smithy-http v0.34.1 (*)
│   │   ├── aws-smithy-http-tower v0.34.1 (*)
│   │   ├── aws-smithy-query v0.34.1
│   │   │   ├── aws-smithy-types v0.34.1 (*)
│   │   ├── aws-smithy-types v0.34.1 (*)
│   │   ├── aws-smithy-xml v0.34.1
│   │   ├── aws-types v0.4.1 (*)
│   ├── aws-smithy-async v0.34.1 (*)
│   ├── aws-smithy-client v0.34.1 (*)
│   ├── aws-smithy-http v0.34.1 (*)
│   ├── aws-smithy-http-tower v0.34.1 (*)
│   ├── aws-smithy-json v0.34.1
│   │   └── aws-smithy-types v0.34.1 (*)
│   ├── aws-smithy-types v0.34.1 (*)
│   ├── aws-types v0.4.1 (*)
├── aws-sdk-cloudformation v0.4.1
│   ├── aws-endpoint v0.4.1 (*)
│   ├── aws-http v0.4.1 (*)
│   ├── aws-sig-auth v0.4.1 (*)
│   ├── aws-smithy-async v0.34.1 (*)
│   ├── aws-smithy-client v0.34.1 (*)
│   ├── aws-smithy-http v0.34.1 (*)
│   ├── aws-smithy-http-tower v0.34.1 (*)
│   ├── aws-smithy-query v0.34.1 (*)
│   ├── aws-smithy-types v0.34.1 (*)
│   ├── aws-smithy-xml v0.34.1 (*)
│   ├── aws-types v0.4.1 (*)
├── aws-sdk-ec2 v0.4.1
│   ├── aws-endpoint v0.4.1 (*)
│   ├── aws-http v0.4.1 (*)
│   ├── aws-sig-auth v0.4.1 (*)
│   ├── aws-smithy-async v0.34.1 (*)
│   ├── aws-smithy-client v0.34.1 (*)
│   ├── aws-smithy-http v0.34.1 (*)
│   ├── aws-smithy-http-tower v0.34.1 (*)
│   ├── aws-smithy-query v0.34.1 (*)
│   ├── aws-smithy-types v0.34.1 (*)
│   ├── aws-smithy-xml v0.34.1 (*)
│   ├── aws-types v0.4.1 (*)

Platform

Darwin hellride 21.2.0 Darwin Kernel Version 21.2.0: Sun Nov 28 20:28:54 PST 2021; root:xnu-8019.61.5~1/RELEASE_X86_64 x86_64

AWS Services

I have tried this with ec2 and cloudformation

Description

Here is the code snippet that produces the error:

        self.vpcs = self
            .client
            .describe_vpcs()
            .into_paginator()
            .items()
            .send()
            .collect::<Result<_, _>>()
            .await?;

Logs

Response from the AWS service ``` 2022-01-18T15:01:56.863335Z TRACE aws_smithy_http::middleware: http_response=Response { status: 200, version: HTTP/1.1, headers: {"x-amzn-requestid": "ef502f55-3764-4362-a0d0-be5090febfce", "cache-control": "no-cache, no-store", "strict-transport-security": "max-age=31536000; includeSubDomains", "vary": "accept-encoding", "content-type": "text/xml;charset=UTF-8", "content-length": "2427", "date": "Tue, 18 Jan 2022 15:01:56 GMT", "server": "AmazonEC2"}, body: b"\n\n ef502f55-374-4332-a0c0-bc6090febfce\n \n \n vpc-xxxxxxxxxxxxxxxxx\n xxxxxxxxxxxx\n available\n 10.16.1.192/28\n \n \n 10.16.1.192/28\n vpc-cidr-assoc-xxxxxxxxxxxxxxxxx\n \n associated\n \n \n \n 10.112.1.192/28\n vpc-cidr-assoc-xxxxxxxxxxxxxxxxx\n \n associated\n \n \n \n 10.16.1.208/28\n vpc-cidr-assoc-xxxxxxxxxxxxxxxxx\n \n associated\n \n \n \n 10.112.1.208/28\n vpc-cidr-assoc-xxxxxxxxxxxxxxxxx\n \n associated\n \n \n \n dopt-xxxxxxxx\n \n \n sigs.k8s.io/cluster-api-provider-aws/cluster/mgmt2\n owned\n \n \n sigs.k8s.io/cluster-api-provider-aws/role\n common\n \n \n Name\n mgmt2-vpc\n \n \n default\n false\n \n \n" } ```

It seems there is no next token at all (implying there is no next page and all the requested data is present?)
@imp imp added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jan 18, 2022
@rcoh
Copy link
Contributor

rcoh commented Jan 18, 2022

thanks for the report! I see the problem and will get a fix out soon

@rcoh rcoh removed the needs-triage This issue or PR still needs to be triaged. label Jan 18, 2022
@rcoh rcoh self-assigned this Jan 18, 2022
@rcoh rcoh changed the title Short description of bug Paginators return error when next token is not present Jan 18, 2022
@rcoh rcoh mentioned this issue Jan 18, 2022
Merged
4 tasks
@Velfi Velfi added the pending-release This issue will be fixed by an approved PR that hasn't been released yet. label Jan 18, 2022
@rcoh
Copy link
Contributor

rcoh commented Jan 19, 2022

The fix has landed, you can test it out prior to the the next release with:

# you'll also need the matching version of aws-config
aws-config = { git = "https://github.com/awslabs/aws-sdk-rust", branch = "next" }

aws-sdk-ec2 = { git = "https://github.com/awslabs/aws-sdk-rust", branch = "next" }

@imp
Copy link
Contributor Author

imp commented Jan 19, 2022

I can confirm it works with next branch. Thanks for addressing the issue quickly.
If it is possible to publish a fixed drop to crates.io it would be fantastic!

@imp imp closed this as completed Jan 19, 2022
@github-actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

@rcoh rcoh reopened this Jan 19, 2022
@rcoh
Copy link
Contributor

rcoh commented Jan 19, 2022

Keeping the issue open until it's in a release in case other folks run into it.

A new release should go out soon

@rcoh
Copy link
Contributor

rcoh commented Jan 21, 2022

landed in 0.5.2

@rcoh rcoh closed this as completed Jan 21, 2022
@github-actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

@Velfi Velfi removed the pending-release This issue will be fixed by an approved PR that hasn't been released yet. label Jan 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rcoh rcoh

Labels
bug This issue is a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@imp @rcoh @Velfi