Intermediate nodes that fails to start

[liorfranko]

What happened:
Every couple of days, we have a node that isn't able to boot and is stuck in NotReady.
Here are the logs:

Aug 15 11:49:04 ip-10-208-73-71 pull-sandbox-image.sh: Attempt 1 of 5
Aug 15 11:49:04 ip-10-208-73-71 systemd: Created slice User Slice of root.
Aug 15 11:49:04 ip-10-208-73-71 systemd: Started Session c11 of user root.
Aug 15 11:49:04 ip-10-208-73-71 pull-sandbox-image.sh: Password: panic: provided file is not a console
Aug 15 11:49:04 ip-10-208-73-71 pull-sandbox-image.sh: goroutine 1 [running]:
Aug 15 11:49:04 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/containerd/console.Current(...)
Aug 15 11:49:04 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/containerd/console/console.go:70
Aug 15 11:49:04 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/cmd/ctr/commands.passwordPrompt(0x0, 0x0, 0x0, 0x0)
Aug 15 11:49:04 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/cmd/ctr/commands/resolver.go:40 +0x3fc
Aug 15 11:49:04 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/cmd/ctr/commands.GetResolver(0x55b1beb1d588, 0xc000116690, 0xc00019d340, 0x55b1beb1d588, 0x55b1beb21730, 0xc000110050, 0xc0001200e0)
Aug 15 11:49:04 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/cmd/ctr/commands/resolver.go:70 +0x3f6
Aug 15 11:49:04 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/cmd/ctr/commands/content.NewFetchConfig(0x55b1beb1d588, 0xc000116690, 0xc00019d340, 0x0, 0x0, 0x0)
Aug 15 11:49:04 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/cmd/ctr/commands/content/fetch.go:115 +0x5c
Aug 15 11:49:04 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/cmd/ctr/commands/images.glob..func7(0xc00019d340, 0x0, 0x0)
Aug 15 11:49:04 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/cmd/ctr/commands/images/pull.go:81 +0x1e5
Aug 15 11:49:04 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.HandleAction(0x55b1be9bd3e0, 0x55b1beaf02c8, 0xc00019d340, 0xc00019d340, 0x0)
Aug 15 11:49:04 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:523 +0x107
Aug 15 11:49:04 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.Command.Run(0x55b1be63e7cc, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x55b1be655d28, 0x1b, 0x0, ...)
Aug 15 11:49:04 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/command.go:174 +0x579
Aug 15 11:49:04 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.(*App).RunAsSubcommand(0xc0003de540, 0xc00019d080, 0x0, 0x0)
Aug 15 11:49:04 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:404 +0x8f4
Aug 15 11:49:04 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.Command.startApp(0x55b1be6404d5, 0x6, 0x0, 0x0, 0x55b1bf002340, 0x2, 0x2, 0x55b1be6480e3, 0xd, 0x0, ...)
Aug 15 11:49:04 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/command.go:329 +0x7ff
Aug 15 11:49:04 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.Command.Run(0x55b1be6404d5, 0x6, 0x0, 0x0, 0x55b1bf002340, 0x2, 0x2, 0x55b1be6480e3, 0xd, 0x0, ...)
Aug 15 11:49:04 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/command.go:102 +0xa14
Aug 15 11:49:04 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.(*App).Run(0xc0003de380, 0xc000032080, 0x8, 0x8, 0x0, 0x0)
Aug 15 11:49:04 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:276 +0x7e8
Aug 15 11:49:04 ip-10-208-73-71 pull-sandbox-image.sh: main.main()
Aug 15 11:49:04 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/cmd/ctr/main.go:37 +0x125
Aug 15 11:49:04 ip-10-208-73-71 systemd: Removed slice User Slice of root.
Aug 15 11:49:05 ip-10-208-73-71 chronyd[5295]: Source 73.61.36.59 replaced with 168.235.86.33 (2.amazon.pool.ntp.org)
Aug 15 11:49:31 ip-10-208-73-71 pull-sandbox-image.sh: Attempt 2 of 5
Aug 15 11:49:31 ip-10-208-73-71 systemd: Created slice User Slice of root.
Aug 15 11:49:31 ip-10-208-73-71 systemd: Started Session c12 of user root.
Aug 15 11:49:31 ip-10-208-73-71 pull-sandbox-image.sh: Password: panic: provided file is not a console
Aug 15 11:49:31 ip-10-208-73-71 pull-sandbox-image.sh: goroutine 1 [running]:
Aug 15 11:49:31 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/containerd/console.Current(...)
Aug 15 11:49:31 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/containerd/console/console.go:70
Aug 15 11:49:31 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/cmd/ctr/commands.passwordPrompt(0x0, 0x0, 0x0, 0x0)
Aug 15 11:49:31 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/cmd/ctr/commands/resolver.go:40 +0x3fc
Aug 15 11:49:31 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/cmd/ctr/commands.GetResolver(0x563e2ce68588, 0xc0004226f0, 0xc00019d080, 0x563e2ce68588, 0x563e2ce6c730, 0xc00042a050, 0xc000420100)
Aug 15 11:49:31 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/cmd/ctr/commands/resolver.go:70 +0x3f6
Aug 15 11:49:31 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/cmd/ctr/commands/content.NewFetchConfig(0x563e2ce68588, 0xc0004226f0, 0xc00019d080, 0x0, 0x0, 0x0)
Aug 15 11:49:31 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/cmd/ctr/commands/content/fetch.go:115 +0x5c
Aug 15 11:49:31 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/cmd/ctr/commands/images.glob..func7(0xc00019d080, 0x0, 0x0)
Aug 15 11:49:31 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/cmd/ctr/commands/images/pull.go:81 +0x1e5
Aug 15 11:49:31 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.HandleAction(0x563e2cd083e0, 0x563e2ce3b2c8, 0xc00019d080, 0xc00019d080, 0x0)
Aug 15 11:49:31 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:523 +0x107
Aug 15 11:49:31 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.Command.Run(0x563e2c9897cc, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x563e2c9a0d28, 0x1b, 0x0, ...)
Aug 15 11:49:31 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/command.go:174 +0x579
Aug 15 11:49:31 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.(*App).RunAsSubcommand(0xc0003dc540, 0xc00019cdc0, 0x0, 0x0)
Aug 15 11:49:31 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:404 +0x8f4
Aug 15 11:49:31 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.Command.startApp(0x563e2c98b4d5, 0x6, 0x0, 0x0, 0x563e2d34d340, 0x2, 0x2, 0x563e2c9930e3, 0xd, 0x0, ...)
Aug 15 11:49:31 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/command.go:329 +0x7ff
Aug 15 11:49:31 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.Command.Run(0x563e2c98b4d5, 0x6, 0x0, 0x0, 0x563e2d34d340, 0x2, 0x2, 0x563e2c9930e3, 0xd, 0x0, ...)
Aug 15 11:49:31 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/command.go:102 +0xa14
Aug 15 11:49:31 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.(*App).Run(0xc0003dc380, 0xc000032080, 0x8, 0x8, 0x0, 0x0)
Aug 15 11:49:31 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:276 +0x7e8
Aug 15 11:49:31 ip-10-208-73-71 pull-sandbox-image.sh: main.main()
Aug 15 11:49:31 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/cmd/ctr/main.go:37 +0x125
Aug 15 11:49:31 ip-10-208-73-71 systemd: Removed slice User Slice of root.
Aug 15 11:49:32 ip-10-208-73-71 dhclient[5585]: XMT: Solicit on eth0, interval 64640ms.
Aug 15 11:50:14 ip-10-208-73-71 pull-sandbox-image.sh: Attempt 3 of 5
Aug 15 11:50:14 ip-10-208-73-71 systemd: Created slice User Slice of root.
Aug 15 11:50:14 ip-10-208-73-71 systemd: Started Session c13 of user root.
Aug 15 11:50:14 ip-10-208-73-71 pull-sandbox-image.sh: Password: panic: provided file is not a console
Aug 15 11:50:14 ip-10-208-73-71 pull-sandbox-image.sh: goroutine 1 [running]:
Aug 15 11:50:14 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/containerd/console.Current(...)
Aug 15 11:50:14 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/containerd/console/console.go:70
Aug 15 11:50:14 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/cmd/ctr/commands.passwordPrompt(0x0, 0x0, 0x0, 0x0)
Aug 15 11:50:14 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/cmd/ctr/commands/resolver.go:40 +0x3fc
Aug 15 11:50:14 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/cmd/ctr/commands.GetResolver(0x560463bd6588, 0xc0000dc390, 0xc00016d760, 0x560463bd6588, 0x560463bda730, 0xc000402050, 0xc000036150)
Aug 15 11:50:14 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/cmd/ctr/commands/resolver.go:70 +0x3f6
Aug 15 11:50:14 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/cmd/ctr/commands/content.NewFetchConfig(0x560463bd6588, 0xc0000dc390, 0xc00016d760, 0x0, 0x0, 0x0)
Aug 15 11:50:14 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/cmd/ctr/commands/content/fetch.go:115 +0x5c
Aug 15 11:50:14 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/cmd/ctr/commands/images.glob..func7(0xc00016d760, 0x0, 0x0)
Aug 15 11:50:14 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/cmd/ctr/commands/images/pull.go:81 +0x1e5
Aug 15 11:50:14 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.HandleAction(0x560463a763e0, 0x560463ba92c8, 0xc00016d760, 0xc00016d760, 0x0)
Aug 15 11:50:14 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:523 +0x107
Aug 15 11:50:14 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.Command.Run(0x5604636f77cc, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x56046370ed28, 0x1b, 0x0, ...)
Aug 15 11:50:14 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/command.go:174 +0x579
Aug 15 11:50:14 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.(*App).RunAsSubcommand(0xc0003b2540, 0xc00016d4a0, 0x0, 0x0)
Aug 15 11:50:14 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:404 +0x8f4
Aug 15 11:50:14 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.Command.startApp(0x5604636f94d5, 0x6, 0x0, 0x0, 0x5604640bb340, 0x2, 0x2, 0x5604637010e3, 0xd, 0x0, ...)
Aug 15 11:50:14 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/command.go:329 +0x7ff
Aug 15 11:50:14 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.Command.Run(0x5604636f94d5, 0x6, 0x0, 0x0, 0x5604640bb340, 0x2, 0x2, 0x5604637010e3, 0xd, 0x0, ...)
Aug 15 11:50:14 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/command.go:102 +0xa14
Aug 15 11:50:14 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.(*App).Run(0xc0003b2380, 0xc000136000, 0x8, 0x8, 0x0, 0x0)
Aug 15 11:50:14 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:276 +0x7e8
Aug 15 11:50:14 ip-10-208-73-71 pull-sandbox-image.sh: main.main()
Aug 15 11:50:14 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/cmd/ctr/main.go:37 +0x125
Aug 15 11:50:14 ip-10-208-73-71 systemd: Removed slice User Slice of root.
Aug 15 11:50:37 ip-10-208-73-71 dhclient[5585]: XMT: Solicit on eth0, interval 115130ms.
Aug 15 11:51:43 ip-10-208-73-71 pull-sandbox-image.sh: Attempt 4 of 5
Aug 15 11:51:43 ip-10-208-73-71 systemd: Created slice User Slice of root.
Aug 15 11:51:43 ip-10-208-73-71 systemd: Started Session c14 of user root.
Aug 15 11:51:44 ip-10-208-73-71 pull-sandbox-image.sh: Password: panic: provided file is not a console
Aug 15 11:51:44 ip-10-208-73-71 pull-sandbox-image.sh: goroutine 1 [running]:
Aug 15 11:51:44 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/containerd/console.Current(...)
Aug 15 11:51:44 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/containerd/console/console.go:70
Aug 15 11:51:44 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/cmd/ctr/commands.passwordPrompt(0x0, 0x0, 0x0, 0x0)
Aug 15 11:51:44 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/cmd/ctr/commands/resolver.go:40 +0x3fc
Aug 15 11:51:44 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/cmd/ctr/commands.GetResolver(0x557e4b206588, 0xc0000dc6c0, 0xc00016d1e0, 0x557e4b206588, 0x557e4b20a730, 0xc0000b8060, 0xc0000361b0)
Aug 15 11:51:44 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/cmd/ctr/commands/resolver.go:70 +0x3f6
Aug 15 11:51:44 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/cmd/ctr/commands/content.NewFetchConfig(0x557e4b206588, 0xc0000dc6c0, 0xc00016d1e0, 0x0, 0x0, 0x0)
Aug 15 11:51:44 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/cmd/ctr/commands/content/fetch.go:115 +0x5c
Aug 15 11:51:44 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/cmd/ctr/commands/images.glob..func7(0xc00016d1e0, 0x0, 0x0)
Aug 15 11:51:44 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/cmd/ctr/commands/images/pull.go:81 +0x1e5
Aug 15 11:51:44 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.HandleAction(0x557e4b0a63e0, 0x557e4b1d92c8, 0xc00016d1e0, 0xc00016d1e0, 0x0)
Aug 15 11:51:44 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:523 +0x107
Aug 15 11:51:44 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.Command.Run(0x557e4ad277cc, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x557e4ad3ed28, 0x1b, 0x0, ...)
Aug 15 11:51:44 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/command.go:174 +0x579
Aug 15 11:51:44 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.(*App).RunAsSubcommand(0xc0003b2540, 0xc00016cf20, 0x0, 0x0)
Aug 15 11:51:44 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:404 +0x8f4
Aug 15 11:51:44 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.Command.startApp(0x557e4ad294d5, 0x6, 0x0, 0x0, 0x557e4b6eb340, 0x2, 0x2, 0x557e4ad310e3, 0xd, 0x0, ...)
Aug 15 11:51:44 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/command.go:329 +0x7ff
Aug 15 11:51:44 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.Command.Run(0x557e4ad294d5, 0x6, 0x0, 0x0, 0x557e4b6eb340, 0x2, 0x2, 0x557e4ad310e3, 0xd, 0x0, ...)
Aug 15 11:51:44 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/command.go:102 +0xa14
Aug 15 11:51:44 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.(*App).Run(0xc0003b2380, 0xc000136000, 0x8, 0x8, 0x0, 0x0)
Aug 15 11:51:44 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:276 +0x7e8
Aug 15 11:51:44 ip-10-208-73-71 pull-sandbox-image.sh: main.main()
Aug 15 11:51:44 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/cmd/ctr/main.go:37 +0x125
Aug 15 11:51:44 ip-10-208-73-71 systemd: Removed slice User Slice of root.
Aug 15 11:52:32 ip-10-208-73-71 dhclient[5585]: XMT: Solicit on eth0, interval 130510ms.
Aug 15 11:54:34 ip-10-208-73-71 pull-sandbox-image.sh: Attempt 5 of 5
Aug 15 11:54:34 ip-10-208-73-71 systemd: Created slice User Slice of root.
Aug 15 11:54:34 ip-10-208-73-71 systemd: Started Session c15 of user root.
Aug 15 11:54:34 ip-10-208-73-71 pull-sandbox-image.sh: Password: panic: provided file is not a console
Aug 15 11:54:34 ip-10-208-73-71 pull-sandbox-image.sh: goroutine 1 [running]:
Aug 15 11:54:34 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/containerd/console.Current(...)
Aug 15 11:54:34 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/containerd/console/console.go:70
Aug 15 11:54:34 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/cmd/ctr/commands.passwordPrompt(0x0, 0x0, 0x0, 0x0)
Aug 15 11:54:34 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/cmd/ctr/commands/resolver.go:40 +0x3fc
Aug 15 11:54:34 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/cmd/ctr/commands.GetResolver(0x5614ca908588, 0xc0003ef290, 0xc00019d760, 0x5614ca908588, 0x5614ca90c730, 0xc00042a050, 0xc0003e2630)
Aug 15 11:54:34 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/cmd/ctr/commands/resolver.go:70 +0x3f6
Aug 15 11:54:34 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/cmd/ctr/commands/content.NewFetchConfig(0x5614ca908588, 0xc0003ef290, 0xc00019d760, 0x0, 0x0, 0x0)
Aug 15 11:54:34 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/cmd/ctr/commands/content/fetch.go:115 +0x5c
Aug 15 11:54:34 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/cmd/ctr/commands/images.glob..func7(0xc00019d760, 0x0, 0x0)
Aug 15 11:54:34 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/cmd/ctr/commands/images/pull.go:81 +0x1e5
Aug 15 11:54:34 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.HandleAction(0x5614ca7a83e0, 0x5614ca8db2c8, 0xc00019d760, 0xc00019d760, 0x0)
Aug 15 11:54:34 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:523 +0x107
Aug 15 11:54:34 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.Command.Run(0x5614ca4297cc, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x5614ca440d28, 0x1b, 0x0, ...)
Aug 15 11:54:34 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/command.go:174 +0x579
Aug 15 11:54:34 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.(*App).RunAsSubcommand(0xc0003dc540, 0xc00019d4a0, 0x0, 0x0)
Aug 15 11:54:34 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:404 +0x8f4
Aug 15 11:54:34 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.Command.startApp(0x5614ca42b4d5, 0x6, 0x0, 0x0, 0x5614caded340, 0x2, 0x2, 0x5614ca4330e3, 0xd, 0x0, ...)
Aug 15 11:54:34 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/command.go:329 +0x7ff
Aug 15 11:54:34 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.Command.Run(0x5614ca42b4d5, 0x6, 0x0, 0x0, 0x5614caded340, 0x2, 0x2, 0x5614ca4330e3, 0xd, 0x0, ...)
Aug 15 11:54:34 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/command.go:102 +0xa14
Aug 15 11:54:34 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/vendor/github.com/urfave/cli.(*App).Run(0xc0003dc380, 0xc000032080, 0x8, 0x8, 0x0, 0x0)
Aug 15 11:54:34 ip-10-208-73-71 pull-sandbox-image.sh: /builddir/build/BUILD/containerd-1.4.13-3.amzn2/src/github.com/containerd/containerd/vendor/github.com/urfave/cli/app.go:276 +0x7e8
Aug 15 11:54:34 ip-10-208-73-71 pull-sandbox-image.sh: main.main()
Aug 15 11:54:34 ip-10-208-73-71 pull-sandbox-image.sh: github.com/containerd/containerd/cmd/ctr/main.go:37 +0x125
Aug 15 11:54:34 ip-10-208-73-71 systemd: sandbox-image.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Aug 15 11:54:34 ip-10-208-73-71 systemd: Failed to start pull sandbox image defined in containerd config.toml.
Aug 15 11:54:34 ip-10-208-73-71 systemd: Dependency failed for Kubernetes Kubelet.
Aug 15 11:54:34 ip-10-208-73-71 user-data: A dependency job for kubelet.service failed. See 'journalctl -xe' for details.
Aug 15 11:54:34 ip-10-208-73-71 systemd: Job kubelet.service/start failed with result 'dependency'.
Aug 15 11:54:34 ip-10-208-73-71 systemd: Unit sandbox-image.service entered failed state.
Aug 15 11:54:34 ip-10-208-73-71 systemd: sandbox-image.service failed.
Aug 15 11:54:34 ip-10-208-73-71 systemd: Reached target Multi-User System.

When I'm manually start the sandbox-image service it works:

Aug 15 13:11:38 ip-10-208-73-71 systemd: Starting pull sandbox image defined in containerd config.toml...
Aug 15 13:11:39 ip-10-208-73-71 systemd: Started Session c17 of user root.
Aug 15 13:11:39 ip-10-208-73-71 containerd: time="2022-08-15T13:11:39.511262111Z" level=info msg="ImageUpdate event &ImageUpdate{Name:602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/pause:3.5,Labels:map[string]string{},XXX_unrecognized:[],}"
Aug 15 13:11:39 ip-10-208-73-71 containerd: time="2022-08-15T13:11:39.518983580Z" level=info msg="ImageUpdate event &ImageUpdate{Name:sha256:6996f8da07bd405c6f82a549ef041deda57d1d658ec20a78584f9f436c9a3bb7,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"
Aug 15 13:11:39 ip-10-208-73-71 containerd: time="2022-08-15T13:11:39.519491618Z" level=info msg="ImageUpdate event &ImageUpdate{Name:602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/pause:3.5,Labels:map[string]string{io.cri-containerd.image: managed,},XXX_unrecognized:[],}"
Aug 15 13:11:39 ip-10-208-73-71 pull-sandbox-image.sh: 602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/pause:3.5: resolving      |#033[32m#033[0m--------------------------------------|
Aug 15 13:11:39 ip-10-208-73-71 pull-sandbox-image.sh: elapsed: 0.1 s                                              total:   0.0 B (0.0 B/s)
Aug 15 13:11:39 ip-10-208-73-71 pull-sandbox-image.sh: 602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/pause:3.5:                       resolved       |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 15 13:11:39 ip-10-208-73-71 pull-sandbox-image.sh: index-sha256:529cf6b1b6e5b76e901abc43aee825badbd93f9c5ee5f1e316d46a83abbce5a2:    done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 15 13:11:39 ip-10-208-73-71 pull-sandbox-image.sh: manifest-sha256:666eebd093e91212426aeba3b89002911d2c981fefd8806b1a0ccb4f1b639a60: done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 15 13:11:39 ip-10-208-73-71 pull-sandbox-image.sh: layer-sha256:0692f38991d53a0c28679148f99de26a44d630fda984b41f63c5e19f839d15a6:    done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 15 13:11:39 ip-10-208-73-71 pull-sandbox-image.sh: config-sha256:6996f8da07bd405c6f82a549ef041deda57d1d658ec20a78584f9f436c9a3bb7:   done           |#033[32m++++++++++++++++++++++++++++++++++++++#033[0m|
Aug 15 13:11:39 ip-10-208-73-71 pull-sandbox-image.sh: elapsed: 0.2 s                                                                    total:   0.0 B (0.0 B/s)
Aug 15 13:11:39 ip-10-208-73-71 pull-sandbox-image.sh: unpacking linux/amd64 sha256:529cf6b1b6e5b76e901abc43aee825badbd93f9c5ee5f1e316d46a83abbce5a2...
Aug 15 13:11:39 ip-10-208-73-71 pull-sandbox-image.sh: done
Aug 15 13:11:39 ip-10-208-73-71 systemd: Started pull sandbox image defined in containerd config.toml.

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• AWS Region: us-east-1
• Instance Type(s): c4.8xlarge (But not only)
• EKS Platform version (use aws eks describe-cluster --name <name> --query cluster.platformVersion): eks.7
• Kubernetes version (use aws eks describe-cluster --name <name> --query cluster.version): 1.20
• AMI Version: v1.20.15-eks-99076b2
• Kernel (e.g. uname -a): Linux ip-10-208-73-71.ec2.internalec2ssa.info 5.4.204-113.362.amzn2.x86_64 #1 SMP Wed Jul 13 21:34:30 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
• Release information (run cat /etc/eks/release on a node):

BASE_AMI_ID="ami-022ef5294726d1825"
BUILD_TIME="Tue Aug  2 17:49:38 UTC 2022"
BUILD_KERNEL="5.4.204-113.362.amzn2.x86_64"
ARCH="x86_64"

[cartermckinnon]

Are you defining your own bootstrap command?

It appears that $ecr_password is empty when the script reaches this line:https://github.com/awslabs/amazon-eks-ami/blob/master/files/pull-sandbox-image.sh#L16

So containerd is prompting for a password from the tty, and panics.

Do you see any stderr from this line? https://github.com/awslabs/amazon-eks-ami/blob/master/files/pull-sandbox-image.sh#L6

[liorfranko]

I did not modify the bootstrap command.
I'm using Karpenter and 99% of the nodes boots just fine, with the same bootstrap command; every couple of days (Without any changes) one of the nodes doesn't boot.

Do you see any stderr from this line? - I already deleted the faulty node so I can't check it now.
What should I check next time it happens?

[cartermckinnon]

My guess is that aws ecr get-login-password is failing and printing an error to stderr, resulting in ecr_password being the empty string. You should see the error in the log.

If you're creating a large number of nodes in a short period of time, you might be hitting a rate limit. I think the relevant Service Quota would be ECR > Rate of GetAuthorizationToken requests.

@suket22 have you seen something like this before?

[cartermckinnon]

Looks like a bug in any case, this script has no handling for get-login-password failures.

[dewjam]

@cartermckinnon Any thoughts on baking the pause image directly into the AMI?

This seems like the right thing to do, but could somewhat challenging given we don't know which container runtime will be used until the bootstrap script runs. I may take a closer look into what it will take.

[cartermckinnon]

@bwagner5 is tackling this in #938 , chime in over there if you can 😄