consul exec is denied even though acl allows

[mosheavni]

Version 0.8.4 both on server and client.
I have acl turned on on this DC, and of course "disable_remote_exec": false on every server for this case.
I try to run consul exec both with token and without token and I get errors of Permission denied.
Attaching the ACL for anonymous and the exec command.
I have no idea what I'm doing wrong.
Anonymous ACL:

key "" {
  policy = "read"
}
event "" {
  policy = "write"
}
event "*" {
  policy = "write"
}

(Master ACL is similar)
the exec commands I tried:

[root@tr-test ~]$ consul exec -verbose -token="792c6304-772c-4672-a956-2e0aadbea205" -node visat ls /tmp/
Created remote execution session: 778caffa-d091-58a7-0116-14206a1854b3
Uploaded remote execution spec
Fired remote execution event: c790aa4d-cbc2-eb82-c24a-2123553aa30b
0 / 0 node(s) completed / acknowledged
Completed in 2.00 seconds
[root@tr-test ~]$ consul exec -verbose -node viast ls /tmp/
Created remote execution session: 377be95b-02f0-928a-b9bf-7efcba1c8851
Failed to create job file: Unexpected response code: 403 (rpc error: Permission denied)

consul monitor on visat (the node I want to exec on)

2017/06/19 07:41:13 [ERR] consul: RPC failed to server 10.2.1.185:8300: rpc error: rpc error: Permission denied
2017/06/19 07:41:13 [ERR] agent: failed to ack remote exec job: rpc error: rpc error: Permission denied

Note that the first exec with the token produces the access denied message on the node consul monitor, while the exec without the token just exits on spot and doesn't prompt log message on the node.

[magiconair]

Did that use to work with previous versions?

[mosheavni]

I didn't try on different versions. Only this. Should I downgrade to test?

[magiconair]

Could you give 0.8.3 and/or 0.8.2 a try since you have the test setup already. This helps us to understand whether this is a regression or a new issue.

[mosheavni]

Same same.

Node           Address          Status  Type    Build  Protocol  DC
as01-test      10.2.1.138:8301  alive   client  0.8.4  2         dc-test
as02-test      10.2.55.13:8301  alive   client  0.8.4  2         dc-test
consul-test01  10.2.1.160:8301  alive   server  0.8.2  2         dc-test
consul-test02  10.2.1.183:8301  alive   server  0.8.2  2         dc-test
consul-test03  10.2.1.185:8301  alive   server  0.8.2  2         dc-test
deploy-test    10.2.1.108:8301  alive   client  0.8.4  2         dc-test
visat          10.2.1.207:8301  alive   client  0.8.2  2         dc-test
tr01-test      10.2.1.28:8301   alive   client  0.8.2  2         dc-test
tr02-test      10.2.55.63:8301  alive   client  0.8.4  2         dc-test

Ran the exec from tr01-test to visat and got on the monitor:

2017/06/19 12:09:49 [ERR] consul: RPC failed to server 10.2.1.160:8300: rpc error: rpc error: Permission denied
2017/06/19 12:09:49 [ERR] agent: failed to ack remote exec job: rpc error: rpc error: Permission denied

[magiconair]

Thank you.

[slackpad]

Hi @MosheM123 you will need to grant the token write access to the KV prefix _rexec which is used to plumb the results of execution back to where the command is running. It looks like this might have gotten dropped as we refactored the documentation recently so we will add this to the API and ACL docs.

[mosheavni]

Maybe I'm missing something because even after adding write access to the _rexec key, I still get access denied.
There are my ACL token's permissions:

event "" {
  policy = "write"
}
event "*" {
  policy = "write"
}
key "*" {
  policy = "write"
}
key "" {
  policy = "write"
}
key "_rexec" {
  policy = "write"
}
event "_rexec" {
  policy = "write"
}

[slackpad]

@MosheM123 tracked this one down - it's a bug. It looks like the remote exec code is always using the acl_token to retrieve the job info from the KV store:

https://github.com/hashicorp/consul/blob/v0.8.4/command/agent/remote_exec.go#L246

And similarly to write the results back:

https://github.com/hashicorp/consul/blob/v0.8.4/command/agent/remote_exec.go#L313

You could configure the acl_token and give it write permissions to the _rexec prefix in KV, but that's not a good configuration since anything talking to the agent will be able to write there, too. Need to think about the correct fix, but it seems like we should pull the acl_agent_token into here, since that's used for internal operations.

[mosheavni]

@slackpad Glad to hear I helped somehow to track a bug James.
Looking forward for a fix :)