"allocation lifecycle permissions" when trying to restart with a stopped allocation

[neclimdul]

Nomad version

Nomad v0.11.1 (b434570)

Operating system and Environment details

Linux X 5.4.0-28-generic #32-Ubuntu SMP Wed Apr 22 17:40:10 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Docker Engine - Community
Version: 19.03.8
Ubuntu 20.04

Issue

When trying to restart my allocation or my job I get the following error.
Your ACL token does not grant allocation lifecycle permissions.
even when using a management token.

When I click restart I see this in my logs:

 2020-05-05T23:12:20.126-0500 [ERROR] http: request failed: method=PUT path=/v1/client/allocation/31b5957a-7893-5766-83e8-f697e41b6d7d/restart error="Task not running" code=500

So this seem to be caused when an allocation fails to start or is otherwise not running and you try to restart.

A quick search for the error in the code suggests it gets shown for any exception which tracks with the token having permissions and the error in the log so I'm under the assumption that's the cause.

Reproduction steps

I think you can reproduce this by starting a job, killing the allocation till its stops respawning and trying to restart. I haven't nailed down the steps yet.

Job file (if appropriate)

Nomad Client logs (if appropriate)

Above

[neclimdul]

now I'm really having trouble recreating this despite being dead in the water the other night and having to stop and start the job to get things working again which means I don't have a working example anymore. I'll keep trying to figure out what sort of weird edge case I was in.

[tgross]

Hi @neclimdul! Thanks for opening this issue!

When trying to restart my allocation or my job I get the following error.

Was this with the CLI or the web UI? You mentioned "click", so I'm wondering if you ran into some kind of problem with the storage of the token in the web UI.

[neclimdul]

It was in the Web UI. The rest of the UI was working just something about restarting wasn't working. Restarting a running service seemed to work as well

I'm not sure if this was clear in my report, I was trying to start a dead task using the restart button in the UI. That's why I think you might be able to reproduce this by maybe killing the task in Docker directly until it stalls and then trying to restart it. I haven't been able to sit down and figure that out though.

[tgross]

Thanks @neclimdul, I think that detail will help narrow things down.

[cgbaker]

I just ran into this issue; there are a few things going on. One is a UI issue; as noted above, the UI assumes that any failure to restart is a permissions issue:
https://github.com/hashicorp/nomad/blob/v1.0.0-beta3/ui/app/controllers/allocations/allocation/index.js#L86

In fact, there are a number of error that can happen on restart. For example, my use case involved a post-start task that isn't running anymore.

1. The UI calls /restart without a task name in the payload
2. The Nomad API will not allow the alloc to be restarted without a task name and returns a 500: Task not running. This is potentially a bug, which I have filed separately (Alloc restart does not restart one-shot lifecycle tasks #9464).
3. The UI reports the (incorrect) cause:

Job spec:

job "repro7875" {
  type = "service"
  datacenters = ["dc1"]
  group "repro" {
    task "main" {
      driver = "exec"
      config {
        command = "sleep"
        args = ["3600"]
      }  
    }
    task "poststart" {
      driver = "exec"
      config {
        command = "env"
      }
      lifecycle {
        hook = "poststart"
      }
    }
  }
}

[pySilver]

Is this feature is still non functional?

[tgross]

It's not in the changelog so I'm not sure when, but it looks like the button was removed from the UI in a later version of Nomad. Which is why we have this feature request open: #9881

@DingoEatingFuzz maybe we should close this one, as the bug is fixed by removing in the button, in lieu of #9881?

[pySilver]

@tgross I'm on the latest Nomad and this button is here :)

[scyd-cb]

@pySilver did you verify with Nomad latest version 1.0.2 ?
I have updated to 1.0.2 and I still don't see the button when the alloc is in failed state :)

[pySilver]

@SCYD here it is https://take.ms/YzCOT

[scyd-cb]

@pySilver yes it is present only when the allocation is running, however it is missing when it is in failed state

[pySilver]

I am on the latest version, yes

[pySilver]

I'm sorry. You are probably right. My issue is that I'm getting this "allocation lifecycle permissions" error when I'm trying to restart running service which is also odd.

[DingoEatingFuzz]

If I'm reading all of this correctly, there are two issues being discussed.

1. Tasks that are dead, or allocs that are terminal (and therefore have dead tasks) cannot be restarted. This issue is also opened as UI does not support ability to Start/Restart failed Allocation and tasks #9881 (thank you @SCYD)
2. When a task or alloc fails to restart, the UI always reports the error as a permissions issue even though this isn't true.

I want to leave this issue open to track the bad error message and keep the restarting dead tasks discussion in #9881.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.