Inconsistent HTTP API behavior when renewing RabbitMQ leases

[artiommft]

Summary

When renewing a RabbitMQ lease the HTTP API responds with an incorrect lease_id.
According to the documentation it should return the lease with original rabbitmq prefix like rabbitmq/creds/..., but it returns sys/leases/renew/... instead.

Environment

Vault docker image v0.7.3
RabbitMQ docker image v3.6.10

Host OS: Ubuntu 16.04, Docker 17.05-ce

Steps to reproduce

1. Prepare the environment by creating a network and running Vault and RabbitMQ images:

docker network create --driver bridge vault_test
docker run -d --name=vault --network=vault_test --cap-add=IPC_LOCK -e 'VAULT_DEV_ROOT_TOKEN_ID=root' -p 8200:8200 vault
docker run -d --name=rabbitmq --network=vault_test -p 15672:15672 rabbitmq:management-alpine

2. Connect interactively to Vault image to set it up with RabbitMQ secret backend:

docker exec -it vault sh

3. Now, run the following commands inside the Vault docker image shell:

export VAULT_TOKEN=root                 
export VAULT_ADDR=http://127.0.0.1:8200/

vault mount rabbitmq
vault write rabbitmq/config/connection connection_uri="http://rabbitmq:15672" username="guest" password="guest"

vault write rabbitmq/roles/readwrite vhosts='{"/":{"write": ".*", "read": ".*"}}'
vault read rabbitmq/creds/readwrite

vault list sys/leases/lookup/rabbitmq/creds/readwrite

Note the lease_id from the commands above, for example rabbitmq/creds/readwrite/5a9a77d1-d4be-3854-e532-8dbcf1fc88dd

4. While still in the vault docker shell, try to renew the lease via Vault CLI:

vault renew rabbitmq/creds/readwrite/5a9a77d1-d4be-3854-e532-8dbcf1fc88dd

The output should be like the following:

Key             Value                                                        
---             -----                                                        
lease_id        rabbitmq/creds/readwrite/5a9a77d1-d4be-3854-e532-8dbcf1fc88dd
lease_duration  767h54m25s                                                   
lease_renewable true

5. Exit the docker image shell with the exit command and create a curl payload file named payload.json with the following contents:

{
  "lease_id": "rabbitmq/creds/readwrite/5a9a77d1-d4be-3854-e532-8dbcf1fc88dd",
  "increment": 1800
}

Make sure to adjust the lease_id value as in previous commands.\

6. Run the following command from the host terminal:

curl --header "X-Vault-Token: root" --request PUT --data @payload.json http://localhost:8200/v1/sys/leases/renew

Actual results:

The curl command above outputs something like (note the lease_id prefix):

{
	"request_id":"6a0d6679-7600-6d36-a20a-aef09d699202",
	"lease_id":"sys/leases/renew/a3dc1840-2870-554f-737e-d76f033e557c",
	"renewable":true,
	"lease_duration":1800,
	"data":null,
	"wrap_info":null,
	"warnings":null,
	"auth":null
}

Expected results:

The lease_id from the results above should be the same as when renewing it via Vault CLI.

Can you please confirm whether this is an issue with Vault or it is something specific to my setup?
Maybe it is already fixed in the latest codebase?
Let me know if you need any more details.

Thank you,
Artiom

[artiommft]

FYI

I was able to reproduce the same behavior with PostgreSQL dabatabse plugin.

Steps to reproduce

This assumes you have the environment running from the message above, so only Postgres-specific commadns are shown.

1. Download and run the postgres instance:

docker run -d --name=postgres --network=vault_test -e POSTGRES_PASSWORD=postgres postgres:alpine

2. Connect to the vault docker container shell:

docker exec -it vault sh

3. Run the following in the vault container shell to configure posgres plugin:

export VAULT_TOKEN=root
export VAULT_ADDR=http://127.0.0.1:8200/

vault mount database

vault write database/config/postgresql \
    plugin_name=postgresql-database-plugin \
    allowed_roles="readonly" \
    connection_url="postgresql://postgres:postgres@postgres:5432/?sslmode=disable"
	
vault write database/roles/readonly \
    db_name=postgresql \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
        GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl="1h" \
    max_ttl="24h"

vault read database/creds/readonly

The last command should produce output similar to this:

Key             Value                                                       
---             -----                                                       
lease_id        database/creds/readonly/216567ec-1082-9e61-c004-bf1d423e1a7c
lease_duration  1h0m0s                                                      
lease_renewable true                                                        
password        254edebb-2166-92c6-b9dc-ae813c850fb9                        
username        v-token-readonly-Jss4aQVMFnpKpklqeY6m-1497600587

4. Try to renew the lease via Vault CLI:

vault renew database/creds/readonly/216567ec-1082-9e61-c004-bf1d423e1a7c

This should produce the following output:

Key             Value                                                       
---             -----                                                       
lease_id        database/creds/readonly/216567ec-1082-9e61-c004-bf1d423e1a7c
lease_duration  1h0m0s                                                      
lease_renewable true

5. Exit the vault container shell with the exit command and create the payload-pg.json file with the following contents:

{
  "lease_id": "database/creds/readonly/216567ec-1082-9e61-c004-bf1d423e1a7c",
  "increment": 1800
}

6. Request credentials renewal via HTTP API:

curl --header "X-Vault-Token: root" --request PUT --data @payload-pg.json http://localhost:8200/v1/sys/leases/renew

Actual results:

The response contains an invalid lease_id:

{
	"request_id":"3452bf67-5b23-4f78-46cb-ac42f11c9938",
	"lease_id":"sys/leases/renew/482786df-a0b6-a6f4-99dc-acd2cc3a425a",
	"renewable":true,
	"lease_duration":1800,
	"data":null,
	"wrap_info":null,
	"warnings":null,
	"auth":null
}

Expected results:

The output of the last command should output same result as for the vault cli command.

[artiommft]

Looking at the source code the HTTP API calls should be made against a different endpoint - /v1/sys/renew as opposed to /v1/sys/leases/renew stated in documentation.

Is the documentation out of sync with the actual functionality?

[shayangz]

This behavior is occurring for mysql database plugin as well on 0.7.3.

And it is causing all types of issues because the returned lease_id can not be renewed even though API response says renewable.

{
  "errors": [
    "failed to renew entry: resp:(*logical.Response)(nil) err:secret is unsupported by this backend"
  ]
}

any business logic relying on the lease renew endpoint returning the same lease id or at lease a lease id that can be renewed again is broken due to this issue.

[jefferai]

@artiommft No, the two locations are supposed to behave identically. This is a bug, but one you can work around right now by using /v1/sys/renew.