kubeflow-dashboard charm goes to Error state on removal

[ca-scribner]

During upgrade testing from 1.6 to 1.7, I did:

• deployed kubeflow-dashboard 1.6 and kubeflow-profile 1.6, then related them
• refreshed kubeflow-dashboard to 1.7
• juju remove-application kubeflow-dashboard

The removal got stuck with the following logs:

juju debug-log -i kubeflow-dashboard --replay

application-kubeflow-dashboard: 13:39:13 INFO juju.cmd running jujud [2.9.34 90e2f047763059f0b8a57941ae0907346464aee8 gc go1.19]
application-kubeflow-dashboard: 13:39:13 DEBUG juju.cmd   args: []string{"/var/lib/juju/tools/jujud", "caasoperator", "--application-name=kubeflow-dashboard", "--debug"}
application-kubeflow-dashboard: 13:39:13 DEBUG juju.agent read agent config, format "2.0"
application-kubeflow-dashboard: 13:39:13 INFO juju.worker.upgradesteps upgrade steps for 2.9.34 have already been run.
application-kubeflow-dashboard: 13:39:13 INFO juju.cmd.jujud caas operator application-kubeflow-dashboard start (2.9.34 [gc])
application-kubeflow-dashboard: 13:39:13 DEBUG juju.worker.dependency "clock" manifold worker started at 2023-03-10 18:39:13.161865129 +0000 UTC
application-kubeflow-dashboard: 13:39:13 DEBUG juju.worker.dependency "caas-units-manager" manifold worker started at 2023-03-10 18:39:13.162476937 +0000 UTC
application-kubeflow-dashboard: 13:39:13 DEBUG juju.worker.dependency "agent" manifold worker started at 2023-03-10 18:39:13.162695457 +0000 UTC
application-kubeflow-dashboard: 13:39:13 DEBUG juju.worker.dependency "caas-units-manager" manifold worker completed successfully
application-kubeflow-dashboard: 13:39:13 DEBUG juju.worker.dependency "upgrade-steps-gate" manifold worker started at 2023-03-10 18:39:13.164621536 +0000 UTC
application-kubeflow-dashboard: 13:39:13 DEBUG juju.worker.introspection introspection worker listening on "@jujud-application-kubeflow-dashboard"
application-kubeflow-dashboard: 13:39:13 DEBUG juju.worker.introspection stats worker now serving
application-kubeflow-dashboard: 13:39:13 DEBUG juju.worker.dependency "api-config-watcher" manifold worker started at 2023-03-10 18:39:13.1732378 +0000 UTC
application-kubeflow-dashboard: 13:39:13 DEBUG juju.worker.dependency "caas-units-manager" manifold worker started at 2023-03-10 18:39:13.173334515 +0000 UTC
application-kubeflow-dashboard: 13:39:13 DEBUG juju.worker.dependency "upgrade-steps-flag" manifold worker started at 2023-03-10 18:39:13.175525077 +0000 UTC
application-kubeflow-dashboard: 13:39:13 DEBUG juju.worker.apicaller connecting with old password
application-kubeflow-dashboard: 13:39:13 DEBUG juju.worker.dependency "migration-fortress" manifold worker started at 2023-03-10 18:39:13.186702338 +0000 UTC
application-kubeflow-dashboard: 13:39:13 DEBUG juju.api successfully dialed "wss://10.152.183.59:17070/model/f7e76919-0b19-47cc-8eb0-325dd06f807f/api"
application-kubeflow-dashboard: 13:39:13 INFO juju.api cannot resolve "controller-service.controller-local-microk8s.svc.cluster.local": lookup controller-service.controller-local-microk8s.svc.cluster.local: operation was canceled
application-kubeflow-dashboard: 13:39:13 INFO juju.api connection established to "wss://10.152.183.59:17070/model/f7e76919-0b19-47cc-8eb0-325dd06f807f/api"
application-kubeflow-dashboard: 13:39:13 INFO juju.worker.apicaller [f7e769] "application-kubeflow-dashboard" successfully connected to "10.152.183.59:17070"
application-kubeflow-dashboard: 13:39:13 DEBUG juju.worker.dependency "api-caller" manifold worker started at 2023-03-10 18:39:13.209778329 +0000 UTC
application-kubeflow-dashboard: 13:39:13 DEBUG juju.worker.dependency "caas-units-manager" manifold worker completed successfully
application-kubeflow-dashboard: 13:39:13 DEBUG juju.worker.dependency "caas-units-manager" manifold worker started at 2023-03-10 18:39:13.219684084 +0000 UTC
application-kubeflow-dashboard: 13:39:13 DEBUG juju.worker.dependency "log-sender" manifold worker started at 2023-03-10 18:39:13.220966515 +0000 UTC
application-kubeflow-dashboard: 13:39:13 DEBUG juju.worker.dependency "migration-minion" manifold worker started at 2023-03-10 18:39:13.222056894 +0000 UTC
application-kubeflow-dashboard: 13:39:13 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker started at 2023-03-10 18:39:13.222572404 +0000 UTC
application-kubeflow-dashboard: 13:39:13 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker completed successfully
application-kubeflow-dashboard: 13:39:13 DEBUG juju.worker.dependency "upgrader" manifold worker started at 2023-03-10 18:39:13.222849064 +0000 UTC
application-kubeflow-dashboard: 13:39:13 DEBUG juju.worker.dependency "migration-inactive-flag" manifold worker started at 2023-03-10 18:39:13.23386301 +0000 UTC
application-kubeflow-dashboard: 13:39:13 INFO juju.worker.caasupgrader abort check blocked until version event received
application-kubeflow-dashboard: 13:39:13 DEBUG juju.worker.caasupgrader current agent binary version: 2.9.34
application-kubeflow-dashboard: 13:39:13 INFO juju.worker.migrationminion migration phase is now: NONE
application-kubeflow-dashboard: 13:39:13 DEBUG juju.worker.dependency "api-address-updater" manifold worker started at 2023-03-10 18:39:13.245457317 +0000 UTC
application-kubeflow-dashboard: 13:39:13 INFO juju.worker.caasupgrader unblocking abort check
application-kubeflow-dashboard: 13:39:13 DEBUG juju.worker.dependency "charm-dir" manifold worker started at 2023-03-10 18:39:13.246639115 +0000 UTC
application-kubeflow-dashboard: 13:39:13 DEBUG juju.worker.logger initial log config: "<root>=DEBUG"
application-kubeflow-dashboard: 13:39:13 DEBUG juju.worker.dependency "logging-config-updater" manifold worker started at 2023-03-10 18:39:13.246885772 +0000 UTC
application-kubeflow-dashboard: 13:39:13 DEBUG juju.worker.dependency "proxy-config-updater" manifold worker started at 2023-03-10 18:39:13.247143718 +0000 UTC
application-kubeflow-dashboard: 13:39:13 INFO juju.worker.logger logger worker started
application-kubeflow-dashboard: 13:39:13 DEBUG juju.worker.logger reconfiguring logging from "<root>=DEBUG" to "<root>=INFO"
application-kubeflow-dashboard: 13:39:13 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
application-kubeflow-dashboard: 13:39:13 INFO juju.worker.caasoperator.charm downloading ch:amd64/focal/kubeflow-dashboard-183 from API server
application-kubeflow-dashboard: 13:39:13 INFO juju.downloader downloading from ch:amd64/focal/kubeflow-dashboard-183
application-kubeflow-dashboard: 13:39:13 INFO juju.downloader download complete ("ch:amd64/focal/kubeflow-dashboard-183")
application-kubeflow-dashboard: 13:39:13 INFO juju.downloader download verified ("ch:amd64/focal/kubeflow-dashboard-183")
application-kubeflow-dashboard: 13:39:20 INFO juju.worker.caasoperator operator "kubeflow-dashboard" started
application-kubeflow-dashboard: 13:39:20 INFO juju.worker.caasoperator.runner start "kubeflow-dashboard/0"
application-kubeflow-dashboard: 13:39:20 INFO juju.worker.leadership kubeflow-dashboard/0 promoted to leadership of kubeflow-dashboard
application-kubeflow-dashboard: 13:39:20 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-kubeflow-dashboard-0
application-kubeflow-dashboard: 13:39:20 INFO juju.worker.caasoperator.uniter.kubeflow-dashboard/0 unit "kubeflow-dashboard/0" started
application-kubeflow-dashboard: 13:39:20 INFO juju.worker.caasoperator.uniter.kubeflow-dashboard/0 resuming charm install
application-kubeflow-dashboard: 13:39:20 INFO juju.worker.caasoperator.uniter.kubeflow-dashboard/0.charm downloading ch:amd64/focal/kubeflow-dashboard-183 from API server
application-kubeflow-dashboard: 13:39:20 INFO juju.downloader downloading from ch:amd64/focal/kubeflow-dashboard-183
application-kubeflow-dashboard: 13:39:20 INFO juju.downloader download complete ("ch:amd64/focal/kubeflow-dashboard-183")
application-kubeflow-dashboard: 13:39:20 INFO juju.downloader download verified ("ch:amd64/focal/kubeflow-dashboard-183")
application-kubeflow-dashboard: 13:39:28 INFO juju.worker.caasoperator.uniter.kubeflow-dashboard/0 hooks are retried true
application-kubeflow-dashboard: 13:39:29 INFO juju.worker.caasoperator.uniter.kubeflow-dashboard/0 found queued "install" hook
application-kubeflow-dashboard: 13:39:31 INFO unit.kubeflow-dashboard/0.juju-log Running legacy hooks/install.
application-kubeflow-dashboard: 13:39:36 INFO juju.worker.caasoperator.uniter.kubeflow-dashboard/0.operation ran "install" hook (via hook dispatching script: dispatch)
application-kubeflow-dashboard: 13:39:39 INFO juju.worker.caasoperator.uniter.kubeflow-dashboard/0.operation ran "kubeflow-profiles-relation-created" hook (via hook dispatching script: dispatch)
application-kubeflow-dashboard: 13:39:39 INFO juju.worker.caasoperator.uniter.kubeflow-dashboard/0 found queued "leader-elected" hook
application-kubeflow-dashboard: 13:39:44 INFO juju.worker.caasoperator.uniter.kubeflow-dashboard/0.operation ran "leader-elected" hook (via hook dispatching script: dispatch)
application-kubeflow-dashboard: 13:39:49 INFO juju.worker.caasoperator.uniter.kubeflow-dashboard/0.operation ran "config-changed" hook (via hook dispatching script: dispatch)
application-kubeflow-dashboard: 13:39:49 INFO juju.worker.caasoperator.uniter.kubeflow-dashboard/0 found queued "start" hook
application-kubeflow-dashboard: 13:39:50 INFO unit.kubeflow-dashboard/0.juju-log Running legacy hooks/start.
application-kubeflow-dashboard: 13:39:53 INFO juju.worker.caasoperator.uniter.kubeflow-dashboard/0.operation ran "start" hook (via hook dispatching script: dispatch)
application-kubeflow-dashboard: 13:39:57 INFO juju.worker.caasoperator.uniter.kubeflow-dashboard/0.operation ran "kubeflow-profiles-relation-joined" hook (via hook dispatching script: dispatch)
application-kubeflow-dashboard: 13:40:03 INFO juju.worker.caasoperator.uniter.kubeflow-dashboard/0.operation ran "kubeflow-profiles-relation-changed" hook (via hook dispatching script: dispatch)
application-kubeflow-dashboard: 13:40:08 INFO juju.worker.caasoperator.uniter.kubeflow-dashboard/0.operation ran "kubeflow-profiles-relation-changed" hook (via hook dispatching script: dispatch)
application-kubeflow-dashboard: 13:40:14 INFO juju.worker.caasoperator.uniter.kubeflow-dashboard/0.operation ran "kubeflow-profiles-relation-changed" hook (via hook dispatching script: dispatch)
application-kubeflow-dashboard: 13:40:19 INFO juju.worker.caasoperator.uniter.kubeflow-dashboard/0.operation ran "config-changed" hook (via hook dispatching script: dispatch)
application-kubeflow-dashboard: 13:40:19 INFO juju.worker.caasoperator started pod init on "kubeflow-dashboard/0"
application-kubeflow-dashboard: 13:45:29 INFO juju.worker.caasoperator.uniter.kubeflow-dashboard/0.operation ran "update-status" hook (via hook dispatching script: dispatch)
application-kubeflow-dashboard: 13:50:23 INFO juju.worker.caasoperator.uniter.kubeflow-dashboard/0.charm downloading ch:amd64/focal/kubeflow-dashboard-249 from API server
application-kubeflow-dashboard: 13:50:23 INFO juju.downloader downloading from ch:amd64/focal/kubeflow-dashboard-249
application-kubeflow-dashboard: 13:50:23 INFO juju.worker.caasoperator.charm downloading ch:amd64/focal/kubeflow-dashboard-249 from API server
application-kubeflow-dashboard: 13:50:23 INFO juju.downloader downloading from ch:amd64/focal/kubeflow-dashboard-249
application-kubeflow-dashboard: 13:50:23 INFO juju.downloader download complete ("ch:amd64/focal/kubeflow-dashboard-249")
application-kubeflow-dashboard: 13:50:23 INFO juju.downloader download complete ("ch:amd64/focal/kubeflow-dashboard-249")
application-kubeflow-dashboard: 13:50:23 INFO juju.downloader download verified ("ch:amd64/focal/kubeflow-dashboard-249")
application-kubeflow-dashboard: 13:50:24 INFO juju.downloader download verified ("ch:amd64/focal/kubeflow-dashboard-249")
application-kubeflow-dashboard: 13:50:33 ERROR juju.worker.caasoperator could not get pod "unit-kubeflow-dashboard-0" "c9ebff80-d949-49c3-bafa-d876712b2378" pod "c9ebff80-d949-49c3-bafa-d876712b2378" not found
application-kubeflow-dashboard: 13:50:33 INFO juju.worker.caasoperator.uniter.kubeflow-dashboard/0 found queued "upgrade-charm" hook
application-kubeflow-dashboard: 13:50:35 INFO unit.kubeflow-dashboard/0.juju-log Running legacy hooks/upgrade-charm.
application-kubeflow-dashboard: 13:50:37 ERROR unit.kubeflow-dashboard/0.juju-log Kubernetes service get failed: services "kubeflow-dashboard" not found
application-kubeflow-dashboard: 13:50:37 ERROR unit.kubeflow-dashboard/0.juju-log Kubernetes service patch failed: services "kubeflow-dashboard" not found
application-kubeflow-dashboard: 13:50:37 INFO juju.worker.caasoperator.uniter.kubeflow-dashboard/0.operation ran "upgrade-charm" hook (via hook dispatching script: dispatch)
application-kubeflow-dashboard: 13:50:37 INFO juju.worker.caasoperator.uniter.kubeflow-dashboard/0 found queued "config-changed" hook
application-kubeflow-dashboard: 13:50:39 INFO juju.worker.caasoperator.uniter.kubeflow-dashboard/0.operation ran "config-changed" hook (via hook dispatching script: dispatch)

This might have been in error even before the attempted removal. I didn't check it beforehand.