Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pulumi crashes with: panic fatal: An assertion has failed #693

Closed
geekflyer opened this issue Aug 5, 2019 · 7 comments · Fixed by #696
Closed

pulumi crashes with: panic fatal: An assertion has failed #693

geekflyer opened this issue Aug 5, 2019 · 7 comments · Fixed by #696
Assignees
@lblackstone
Labels
kind/bug Some behavior is incorrect or out of spec
Milestone
0.26

Comments

@geekflyer
Copy link

Since upgrading some of our stacks from CLI 0.17.25 to v0.17.27 I'm getting in some stacks this rather unhelpful error:

 panic: fatal: An assertion has failed
    goroutine 16 [running]:
    github.com/pulumi/pulumi-kubernetes/vendor/github.com/pulumi/pulumi/pkg/util/contract.failfast(...)
        /home/travis/gopath/src/github.com/pulumi/pulumi-kubernetes/vendor/github.com/pulumi/pulumi/pkg/util/contract/failfast.go:23
    github.com/pulumi/pulumi-kubernetes/vendor/github.com/pulumi/pulumi/pkg/util/contract.Assert(...)
        /home/travis/gopath/src/github.com/pulumi/pulumi-kubernetes/vendor/github.com/pulumi/pulumi/pkg/util/contract/assert.go:26
    github.com/pulumi/pulumi-kubernetes/pkg/provider.(*patchConverter).addPatchValueToDiff(0xc000644fe8, 0xc001344080, 0x6, 0x8, 0x0, 0x0, 0x0, 0x0, 0x2038b20, 0x332dcc0, ...)
        /home/travis/gopath/src/github.com/pulumi/pulumi-kubernetes/pkg/provider/provider.go:1502 +0xbae
    github.com/pulumi/pulumi-kubernetes/pkg/provider.(*patchConverter).addPatchMapToDiff(0xc000644fe8, 0xc001344080, 0x5, 0x8, 0xc000028a20, 0xc001242ab0, 0xc001257500, 0xc001242ab0, 0x1, 0x20b14c0, ...)
        /home/travis/gopath/src/github.com/pulumi/pulumi-kubernetes/pkg/provider/provider.go:1602 +0x305
    github.com/pulumi/pulumi-kubernetes/pkg/provider.(*patchConverter).addPatchValueToDiff(0xc000644fe8, 0xc001344080, 0x5, 0x8, 0x20b8a80, 0xc000028a20, 0x20b8a80, 0xc001242ab0, 0x20b8a80, 0xc001257500, ...)
        /home/travis/gopath/src/github.com/pulumi/pulumi-kubernetes/pkg/provider/provider.go:1523 +0x14b
    github.com/pulumi/pulumi-kubernetes/pkg/provider.(*patchConverter).addPatchArrayToDiff(0xc000644fe8, 0xc000ebe140, 0x4, 0x4, 0xc000eba0f0, 0x1, 0x1, 0xc00123bd50, 0x1, 0x1, ...)
        /home/travis/gopath/src/github.com/pulumi/pulumi-kubernetes/pkg/provider/provider.go:1633 +0x1b0
    github.com/pulumi/pulumi-kubernetes/pkg/provider.(*patchConverter).addPatchValueToDiff(0xc000644fe8, 0xc000ebe140, 0x4, 0x4, 0x2038b20, 0xc000ec2040, 0x2038b20, 0xc001248900, 0x2038b20, 0xc001260980, ...)
        /home/travis/gopath/src/github.com/pulumi/pulumi-kubernetes/pkg/provider/provider.go:1530 +0x92c
    github.com/pulumi/pulumi-kubernetes/pkg/provider.(*patchConverter).addPatchMapToDiff(0xc000644fe8, 0xc000ebe140, 0x3, 0x4, 0xc0000289f0, 0xc001242930, 0xc0012573b0, 0xc001242930, 0x0, 0x20b14c0, ...)
        /home/travis/gopath/src/github.com/pulumi/pulumi-kubernetes/pkg/provider/provider.go:1602 +0x305
    github.com/pulumi/pulumi-kubernetes/pkg/provider.(*patchConverter).addPatchValueToDiff(0xc000644fe8, 0xc000ebe140, 0x3, 0x4, 0x20b8a80, 0xc0000289f0, 0x20b8a80, 0xc001242930, 0x20b8a80, 0xc0012573b0, ...)
        /home/travis/gopath/src/github.com/pulumi/pulumi-kubernetes/pkg/provider/provider.go:1523 +0x14b
    github.com/pulumi/pulumi-kubernetes/pkg/provider.(*patchConverter).addPatchMapToDiff(0xc000644fe8, 0xc000ec2980, 0x2, 0x2, 0xc0000289c0, 0xc0012428a0, 0xc001257320, 0xc0012428a0, 0x0, 0x20b14c0, ...)
        /home/travis/gopath/src/github.com/pulumi/pulumi-kubernetes/pkg/provider/provider.go:1602 +0x305
    github.com/pulumi/pulumi-kubernetes/pkg/provider.(*patchConverter).addPatchValueToDiff(0xc000644fe8, 0xc000ec2980, 0x2, 0x2, 0x20b8a80, 0xc0000289c0, 0x20b8a80, 0xc0012428a0, 0x20b8a80, 0xc001257320, ...)
        /home/travis/gopath/src/github.com/pulumi/pulumi-kubernetes/pkg/provider/provider.go:1523 +0x14b
    github.com/pulumi/pulumi-kubernetes/pkg/provider.(*patchConverter).addPatchMapToDiff(0xc000644fe8, 0xc000ebaa60, 0x1, 0x1, 0xc000028990, 0xc001242810, 0xc001257290, 0xc001242810, 0x0, 0x20b14c0, ...)
        /home/travis/gopath/src/github.com/pulumi/pulumi-kubernetes/pkg/provider/provider.go:1602 +0x305
    github.com/pulumi/pulumi-kubernetes/pkg/provider.(*patchConverter).addPatchValueToDiff(0xc000644fe8, 0xc000ebaa60, 0x1, 0x1, 0x20b8a80, 0xc000028990, 0x20b8a80, 0xc001242810, 0x20b8a80, 0xc001257290, ...)
        /home/travis/gopath/src/github.com/pulumi/pulumi-kubernetes/pkg/provider/provider.go:1523 +0x14b
    github.com/pulumi/pulumi-kubernetes/pkg/provider.(*patchConverter).addPatchMapToDiff(0xc000644fe8, 0x0, 0x0, 0x0, 0xc000028900, 0xc001242750, 0xc0012571d0, 0xc001242750, 0xc0004a0000, 0xa, ...)
        /home/travis/gopath/src/github.com/pulumi/pulumi-kubernetes/pkg/provider/provider.go:1602 +0x305
    github.com/pulumi/pulumi-kubernetes/pkg/provider.convertPatchToDiff(0xc000028900, 0xc001242750, 0xc0012571d0, 0xc001242750, 0xc0008c35b8, 0x4, 0xc0008c35bd, 0x2, 0xc0008c35c0, 0xa, ...)
        /home/travis/gopath/src/github.com/pulumi/pulumi-kubernetes/pkg/provider/provider.go:1439 +0x289
    github.com/pulumi/pulumi-kubernetes/pkg/provider.(*kubeProvider).Diff(0xc000184000, 0x2501d00, 0xc001219e60, 0xc0011ea0a0, 0xc000184000, 0x20e0c01, 0xc0011d0cc0)
        /home/travis/gopath/src/github.com/pulumi/pulumi-kubernetes/pkg/provider/provider.go:659 +0xdbc
    github.com/pulumi/pulumi-kubernetes/vendor/github.com/pulumi/pulumi/sdk/proto/go._ResourceProvider_Diff_Handler.func1(0x2501d00, 0xc001219e60, 0x22099a0, 0xc0011ea0a0, 0x221d180, 0x332c6f8, 0x2501d00, 0xc001219e60)
        /home/travis/gopath/src/github.com/pulumi/pulumi-kubernetes/vendor/github.com/pulumi/pulumi/sdk/proto/go/provider.pb.go:1522 +0x86
    github.com/pulumi/pulumi-kubernetes/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1(0x2501d00, 0xc0011e62d0, 0x22099a0, 0xc0011ea0a0, 0xc0011c5860, 0xc0011c5880, 0x0, 0x0, 0x24c5cc0, 0xc000167430)
        /home/travis/gopath/src/github.com/pulumi/pulumi-kubernetes/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:61 +0x360
    github.com/pulumi/pulumi-kubernetes/vendor/github.com/pulumi/pulumi/sdk/proto/go._ResourceProvider_Diff_Handler(0x222bd80, 0xc000184000, 0x2501d00, 0xc0011e62d0, 0xc0011ea050, 0xc0000be140, 0x2501d00, 0xc0011e62d0, 0xc0011fa000, 0x1d86)
        /home/travis/gopath/src/github.com/pulumi/pulumi-kubernetes/vendor/github.com/pulumi/pulumi/sdk/proto/go/provider.pb.go:1524 +0x158
    github.com/pulumi/pulumi-kubernetes/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc00047c900, 0x251c920, 0xc0002d4180, 0xc0000fef00, 0xc000465560, 0x32fe5d8, 0x0, 0x0, 0x0)
        /home/travis/gopath/src/github.com/pulumi/pulumi-kubernetes/vendor/google.golang.org/grpc/server.go:972 +0x470
    github.com/pulumi/pulumi-kubernetes/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00047c900, 0x251c920, 0xc0002d4180, 0xc0000fef00, 0x0)
        /home/travis/gopath/src/github.com/pulumi/pulumi-kubernetes/vendor/google.golang.org/grpc/server.go:1252 +0xda6
    github.com/pulumi/pulumi-kubernetes/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0005b2010, 0xc00047c900, 0x251c920, 0xc0002d4180, 0xc0000fef00)
        /home/travis/gopath/src/github.com/pulumi/pulumi-kubernetes/vendor/google.golang.org/grpc/server.go:691 +0x9f
    created by github.com/pulumi/pulumi-kubernetes/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
        /home/travis/gopath/src/github.com/pulumi/pulumi-kubernetes/vendor/google.golang.org/grpc/server.go:689 +0xa1
 
  kubernetes:apps:Deployment (api-events):
    error: transport is closing

This seems to be happening when it's evaluating the preview of a kubernetes Deployment.

The pulumi kubernetes version is 0.25.5 currently, even though I can reproduce this with multiple pulumi kubernetes versions.

Let me know what other questions you have to get to the bottom of this.
@ellismg ellismg transferred this issue from pulumi/pulumi Aug 5, 2019
@ellismg
Copy link
Contributor

ellismg commented Aug 5, 2019

Very surprising that this has started to happen after changing the CLI version. The crash itself is definitely coming from the kubernetes provider, so I've moved the issue back here for the time being.

/cc @pgavlin, @lblackstone It looks like this is a failing assert when trying to calculate the detailed diff to return from the provider.

@ellismg ellismg added feature/q3 kind/bug Some behavior is incorrect or out of spec labels Aug 5, 2019
@geekflyer
Copy link
Author

to be honest I'm not 100% sure whether this really just occurred because of the CLI update, however we recently upgraded pulumi CLIs locally and in CI to get in #645 and I didn't get this issue before.

@lblackstone
Copy link
Member

The assertion failing here is

pulumi-kubernetes/pkg/provider/provider.go

Line 1502 in 9375329

contract.Assert(v != nil || old != nil)

@pgavlin would have the most context on troubleshooting this.

@pgavlin
Copy link
Member

pgavlin commented Aug 5, 2019

Yep, looking into it.

@joeduffy joeduffy added this to the 0.26 milestone Aug 6, 2019
@martianatwork
Copy link

+1 same issue on v0.17.28

@pgavlin
Copy link
Member

pgavlin commented Aug 7, 2019

I have been able to repro this locally.

pgavlin added a commit to pulumi/pulumi that referenced this issue Aug 7, 2019
@pgavlin
Fix conversion of empty array properties.
4eb7d55 
Empty `[]interface{}` values were being converted to array property
values with a `nil` element, and empty array property values were being
coverted to `nil` `[]interface{}` values. These changes fix the
converters to return empty but non-nil values in both cases.

This is part of the fix for
pulumi/pulumi-kubernetes#693.
@pgavlin pgavlin mentioned this issue Aug 7, 2019
Merged
@pgavlin
Copy link
Member

pgavlin commented Aug 7, 2019

The repro is pretty simple. First deploy the following:

const deployment = new k8s.apps.v1.Deployment("nginx", {
    spec: {
        selector: { matchLabels: appLabels },
        replicas: 1,
        template: {
            metadata: { labels: appLabels },
            spec: {
                containers: [{
                    name: "nginx",
                    image: "nginx",
                    resources: {
                        limits: {
                            // This value will be normalized by the API server to "2Gi" in the returned state. Without
                            // dry-run support, Pulumi will detect a spurious diff due to the normalization.
                            memory: "2048Mi",
                        },
                    },
                }],
            }
        }
    }
});

Then deploy the following, which changes image and adds empty volumes and volumeMounts properties:

const deployment = new k8s.apps.v1.Deployment("nginx", {
    spec: {
        selector: { matchLabels: appLabels },
        replicas: 1,
        template: {
            metadata: { labels: appLabels },
            spec: {
                containers: [{
                    name: "nginx",
                    image: "nginx",
                    resources: {
                        limits: {
                            // This value will be normalized by the API server to "2Gi" in the returned state. Without
                            // dry-run support, Pulumi will detect a spurious diff due to the normalization.
                            memory: "2048Mi",
                        },
                    },
                    volumeMounts: [],
                }],
                volumes: [],
            }
        }
    }
});

The empty arrays will end up confusing the diff logic, causing this crash.

pgavlin added a commit to pulumi/pulumi that referenced this issue Aug 7, 2019
@pgavlin
Fix conversion of empty array properties. (#3047)
6983e26 
Empty `[]interface{}` values were being converted to array property
values with a `nil` element, and empty array property values were being
coverted to `nil` `[]interface{}` values. These changes fix the
converters to return empty but non-nil values in both cases.

This is part of the fix for
pulumi/pulumi-kubernetes#693.
pgavlin added a commit that referenced this issue Aug 7, 2019
@pgavlin
Update pulumi/pulumi.
d567dbf 
Pick up a recent fix to empty array property conversion.

Fixes #693.
@pgavlin pgavlin mentioned this issue Aug 7, 2019
Merged
pgavlin added a commit that referenced this issue Aug 7, 2019
@pgavlin
Update pulumi/pulumi.
ce85cd2 
Pick up a recent fix to empty array property conversion.

Fixes #693.
pgavlin added a commit that referenced this issue Aug 7, 2019
@pgavlin
Update pulumi/pulumi. (#696)
aea8453 
Pick up a recent fix to empty array property conversion.

Fixes #693.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lblackstone lblackstone

Labels
kind/bug Some behavior is incorrect or out of spec
Projects
None yet
Milestone
0.26
Development

Successfully merging a pull request may close this issue.

Update pulumi/pulumi. pulumi/pulumi-kubernetes
6 participants
@geekflyer @pgavlin @joeduffy @lblackstone @ellismg @martianatwork