fix panic while deleting CSI plugins for missing job

[michaeldwan]

This is what we deployed to production to fix the panic described in #7757.

[hashicorp-cla]

All committers have signed the CLA.

[tgross]

LGTM.

But I think I want to tag @langmartin on this to see this was intended as an assert that we shouldn't otherwise be seeing.

[tgross]

Hi @michaeldwan I think the error we made here is that this line should have read:

tg := a.job.LookupTaskGroup(a.TaskGroup)

It seemed strange to me that a job can not have the taskgroup for one of its own allocs, but my colleague @notnoop has pointed out to me that the alloc can have a view of the job that's stale relative to the job we've pulled from the state store, and that's why we're running into this panic.

[michaeldwan]

That's roughly what we think happened too. We were actually working on a network issue that impacted consul but nomad seemed healthy through it.

I can make that change here.

[tgross]

Gah, I typoed my example. It's a.Job (capital letter) 😊

[michaeldwan]

I just made the change. There's a few other places LookupTaskGroup is being called but they appear to get both the job and alloc from the same snapshot and should be okay.

[michaeldwan]

I made the same typo and pushed before waiting for tests in vagrant to run :)

[langmartin]

This fix looks good to me.

[notnoop]

The fix is LGTM. Thank you so much for reporting and fixing the issue! We probably should also follow up with more tests covering this area and we, the Nomad team, can take care of it.

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.