Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Do not retry on create/update/delete failures #17

Closed
jaypipes opened this issue Jun 25, 2024 · 0 comments
Closed

Do not retry on create/update/delete failures #17

jaypipes opened this issue Jun 25, 2024 · 0 comments
Assignees
@jaypipes
Labels
bug Something isn't working

Comments

@jaypipes
Copy link
Member

The recent centralization of retries into the gdt-dev/gdt Scenario.Run method was great for taking away from plugins the need to handle retries, however some actions like create/update/delete/apply for Kubernetes should not be retried if an error is returned, otherwise you get predicatable problems:

=== RUN   TestKubernetesClusterHappyPath
[gdt] [fixtures.undercloud.start] undercloud cluster is up
=== RUN   TestKubernetesClusterHappyPath/happy-path
[gdt] [happy-path] using retry (exponential: true) [plugin default]
[gdt] [happy-path] using timeout of 5s [plugin default]
[gdt] [happy-path/0:create-cluster] kube.create: kubernetesclusters (ns: nc-system)
[gdt] [happy-path/0:create-cluster] run: attempt 1 after 600ns ok: false
[gdt] [happy-path/0:create-cluster] run: attempt 1 after 600ns failure: assertion failed: unexpected error: KubernetesCluster.aks.afo-nc.microsoft.com "" is invalid: metadata.name: Required value: name or generateName is required
[gdt] [happy-path/0:create-cluster] kube.create: kubernetesclusters (ns: nc-system)
[gdt] [happy-path/0:create-cluster] run: attempt 2 after 379.63448ms ok: false
[gdt] [happy-path/0:create-cluster] run: attempt 2 after 379.63448ms failure: assertion failed: unexpected error: KubernetesCluster.aks.afo-nc.microsoft.com "" is invalid: metadata.name: Required value: name or generateName is required
[gdt] [happy-path/0:create-cluster] kube.create: kubernetesclusters (ns: nc-system)
[gdt] [happy-path/0:create-cluster] run: attempt 3 after 1.500700752s ok: false
[gdt] [happy-path/0:create-cluster] run: attempt 3 after 1.500700752s failure: assertion failed: unexpected error: KubernetesCluster.aks.afo-nc.microsoft.com "" is invalid: metadata.name: Required value: name or generateName is required
[gdt] [happy-path/0:create-cluster] kube.create: kubernetesclusters (ns: nc-system)
[gdt] [happy-path/0:create-cluster] run: attempt 4 after 2.265334563s ok: false
[gdt] [happy-path/0:create-cluster] run: attempt 4 after 2.265334563s failure: assertion failed: unexpected error: KubernetesCluster.aks.afo-nc.microsoft.com "" is invalid: metadata.name: Required value: name or generateName is required
[gdt] [happy-path/0:create-cluster] kube.create: kubernetesclusters (ns: nc-system)
[gdt] [happy-path/0:create-cluster] run: attempt 5 after 4.005659175s ok: false
[gdt] [happy-path/0:create-cluster] run: attempt 5 after 4.005659175s failure: assertion failed: unexpected error: KubernetesCluster.aks.afo-nc.microsoft.com "" is invalid: metadata.name: Required value: name or generateName is required
    run.go:116: assertion failed: unexpected error: KubernetesCluster.aks.afo-nc.microsoft.com "" is invalid: metadata.name: Required value: name or generateName is required
[gdt] [happy-path] wait: 5m before
^Csignal: interrupt
@jaypipes jaypipes added the bug Something isn't working label Jun 25, 2024
@jaypipes jaypipes self-assigned this Jun 25, 2024
jaypipes added a commit to jaypipes/gdt-core that referenced this issue Jun 25, 2024
@jaypipes
allow an Evaluable to override retry behaviour
db376cc 
We need to allow a plugin's Evaluable to override the plugin's default
retry behaviour. In the case of gdt-kube, we want to *not* retry in the
case of `kubectl.create`, `kubectl.apply` and `kubectl.delete`, however
we *do* want to retry in the case of `kubectl.get`...

Issue gdt-dev/kube#17

Signed-off-by: Jay Pipes <jaypipes@gmail.com>
@jaypipes jaypipes mentioned this issue Jun 25, 2024
Merged
jaypipes added a commit to jaypipes/gdt-kube that referenced this issue Jun 25, 2024
@jaypipes
ensure no retry for create/apply/delete
464030a 
We only want to retry kube.get actions, not create, apply or delete. In
bringing in gdt@v1.9.0, we now have override ability for an individual
Evaluable's Retry and Timeout.

When adding this functionality, it became evident that KinD clusters
don't immediately add the default service account, which is required for
simple tests like the creation of an nginx Pod.

So, added a wait loop inside the KindFixture's Start method that waits
for up to 15 seconds until it sees the existence of the default service
account.

Issue gdt-dev#17

Signed-off-by: Jay Pipes <jaypipes@gmail.com>
@jaypipes jaypipes mentioned this issue Jun 25, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jaypipes jaypipes

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jaypipes