Add parameter to vault token renewal #274

deuch · 2017-04-26T14:25:00Z

Fabio need to have a parameter in the fabio.properties to let the user set the renewal interval for the Vault Token. It's 1h actually and it's too long :-) We've issue because our tokens have a 20mn lease.

magiconair · 2017-04-26T15:52:29Z

Sure. Was expecting that this would happen at some time.

Add the 'renewtoken' option for vault certificate sources to make the token renew interval configurable. Fixes #274

magiconair · 2017-04-26T16:21:33Z

@deuch I've pushed a change that should make the token renew interval configurable. You have to set the renewtoken option in the proxy.cs statement. Could you check whether that solves the problem for you?

deuch · 2017-04-27T07:39:23Z

Hello,

I've made the test and it doesn't seem to work, fabio doesn't want to start, telling me that the port is already used ... Stop fabio, kill every instance, check with netstat (root) and nothing is listening on that port.

Revert back to 1.4.3 and it works fine (with exact same properties file) ...

Please find the logs :

2017/04/27 09:31:25 [INFO] Runtime config
{
    "Proxy": {
        "Strategy": "rr",
        "Matcher": "prefix",
        "NoRouteStatus": 404,
        "MaxConn": 10000,
        "ShutdownWait": 0,
        "DialTimeout": 30000000000,
        "ResponseHeaderTimeout": 0,
        "KeepAliveTimeout": 0,
        "FlushInterval": 1000000000,
        "LocalIP": "192.163.167.143",
        "ClientIPHeader": "",
        "TLSHeader": "",
        "TLSHeaderValue": "",
        "GZIPContentTypes": null
    },
    "Registry": {
        "Backend": "consul",
        "Static": {
            "Routes": ""
        },
        "File": {
            "Path": ""
        },
        "Consul": {
            "Addr": "dckdevdck075.mydomain.com:8543",
            "Scheme": "https",
            "Token": "xxxxxxxxxxxxxxxxxxxxxxxxx",
            "KVPath": "/l7demo1/config",
            "TagPrefix": "urlprefix-",
            "Register": true,
            "ServiceAddr": ":9998",
            "ServiceName": "l7demo1",
            "ServiceTags": null,
            "ServiceStatus": [
                "passing"
            ],
            "CheckInterval": 1000000000,
            "CheckTimeout": 3000000000,
            "CheckScheme": "http",
            "CheckTLSSkipVerify": false
        },
        "Timeout": 10000000000,
        "Retry": 500000000
    },
    "Listen": [
        {
            "Addr": "dckdevdck075:11443",
            "Proto": "tcp+sni",
            "ReadTimeout": 0,
            "WriteTimeout": 0,
            "CertSource": {
                "Name": "",
                "Type": "",
                "CertPath": "",
                "KeyPath": "",
                "ClientCAPath": "",
                "CAUpgradeCN": "",
                "Refresh": 0,
                "RenewToken": 0,
                "Header": null
            },
            "StrictMatch": false
        },
        {
            "Addr": "dckdevdck075:12443",
            "Proto": "https",
            "ReadTimeout": 0,
            "WriteTimeout": 0,
            "CertSource": {
                "Name": "ssl-vault",
                "Type": "vault",
                "CertPath": "secret/fabiodemo-001/certs",
                "KeyPath": "",
                "ClientCAPath": "",
                "CAUpgradeCN": "",
                "Refresh": 3000000000,
                "RenewToken": 900000000000,
                "Header": null
            },
            "StrictMatch": false
        }
    ],
    "Log": {
        "AccessFormat": "$remote_host - - [$time_common] \"$request\" $request_proto $response_status $response_body_size $upstream_addr $upstream_host $upstream_port $upstream_request_url $upstream_request_uri",
        "AccessTarget": "stdout",
        "RoutesFormat": "delta"
    },
    "Metrics": {
        "Target": "statsd",
        "Prefix": "{{clean .Hostname}}.{{clean .Exec}}",
        "Names": "{{clean .Service}}.{{clean .Host}}.{{clean .Path}}.{{clean .TargetURL.Host}}",
        "Interval": 30000000000,
        "GraphiteAddr": "",
        "StatsDAddr": "184.44.245.74:8125",
        "Circonus": {
            "APIKey": "",
            "APIApp": "fabio",
            "APIURL": "",
            "CheckID": "",
            "BrokerID": ""
        }
    },
    "UI": {
        "Listen": {
            "Addr": ":9998",
            "Proto": "http",
            "ReadTimeout": 0,
            "WriteTimeout": 0,
            "CertSource": {
                "Name": "",
                "Type": "",
                "CertPath": "",
                "KeyPath": "",
                "ClientCAPath": "",
                "CAUpgradeCN": "",
                "Refresh": 0,
                "RenewToken": 0,
                "Header": null
            },
            "StrictMatch": false
        },
        "Color": "light-green",
        "Title": ""
    },
    "Runtime": {
        "GOGC": 800,
        "GOMAXPROCS": 1
    }
}
2017/04/27 09:31:25 [INFO] Version  starting
2017/04/27 09:31:25 [INFO] Go runtime is go1.8.1
2017/04/27 09:31:25 [INFO] Sending metrics to StatsD on 184.44.245.74:8125 as "dckdevdck075.fabio"
2017/04/27 09:31:25 [INFO] Sending metrics to StatsD on 184.44.245.74:8125 as "dckdevdck075.fabio"
2017/04/27 09:31:25 [INFO] Setting GOGC=800
2017/04/27 09:31:25 [INFO] Setting GOMAXPROCS=1
2017/04/27 09:31:25 [INFO] consul: Connecting to "dckdevdck075.mydomain.com:8543" in datacenter "dcfabio"
2017/04/27 09:31:25 [INFO] Admin server listening on ":9998"
2017/04/27 09:31:25 [INFO] Waiting for first routing table
2017/04/27 09:31:25 [INFO] consul: Using dynamic routes
2017/04/27 09:31:25 [INFO] consul: Using tag prefix "urlprefix-"
2017/04/27 09:31:25 [INFO] consul: Watching KV path "/l7demo1/config"
2017/04/27 09:31:25 [INFO] consul: Manual config changed to #1
2017/04/27 09:31:25 [INFO] TCP+SNI proxy listening on dckdevdck075:11443
2017/04/27 09:31:25 [INFO] consul: Health changed to #21965
2017/04/27 09:31:25 [INFO] Config updates
+ route add l7demo1-tomcatfabiosni tomcat-sni.mydomain.com/ http://dckdevdck012:30005/
+ route add l7demo1-tomcatfabio tomcat-fabio.mydomain.com/ https://dckdevdck012:30030 opts "proto=https tlsskipverify=true"
2017/04/27 09:31:25 [INFO] consul: Registered fabio with id "l7demo1-dckdevdck075-9998"
2017/04/27 09:31:25 [INFO] consul: Registered fabio with address "192.163.167.143"
2017/04/27 09:31:25 [INFO] consul: Registered fabio with tags ""
2017/04/27 09:31:25 [INFO] consul: Registered fabio with health check to "http://[192.163.167.143]:9998/health"
2017/04/27 09:31:25 [INFO] HTTPS proxy listening on dckdevdck075:12443
2017/04/27 09:31:25 [INFO] Writing access log to stdout
2017/04/27 09:31:25 [INFO] Using routing strategy "rr"
2017/04/27 09:31:25 [INFO] Using route matching "prefix"
2017/04/27 09:31:25 [FATAL] listen: Fail to listen. listen tcp 192.163.167.143:12443: bind: address already in use
2017/04/27 09:31:25 [FATAL] accept tcp 192.163.167.143:12443: use of closed network connection
2017/04/27 09:31:25 [INFO] consul: Deregistering fabio
2017/04/27 09:31:25 [FATAL] ui: http: Server closed

magiconair · 2017-04-27T08:02:28Z

@deuch Hmm, I could check whether the SO_REUSEADDR is used. How did you kill fabio? kill -9?

Can you try with a different port or wait a bit? I think 2 min is the timeout. I just want to confirm that the vault change works.

deuch · 2017-04-27T08:53:33Z

I try an another port, and no need to wait, the fabio process is immediatly killed with the fatal error.

Please find the commands i use to show you the issue. I've started with the issue-274 branch and just after with the 1.4.3. You will see that the 1.4.3 works fine on the same server with the same config (i've just changed the symbolink link for the fabio binary).

-bash-4.2$ netstat -an | grep 12443
-bash-4.2$ ps -edf | grep fabio
webdev01 27842 26465  0 10:45 pts/2    00:00:00 grep --color=auto fabio
-bash-4.2$ ./fabio.start start
 Starting  Fabio
-bash-4.2$ tail -f logs/fabio.out
2017/04/27 10:45:10 [INFO] consul: Registered fabio with tags ""
2017/04/27 10:45:10 [INFO] consul: Registered fabio with health check to "http://[192.163.167.143]:9998/health"
2017/04/27 10:45:10 [INFO] HTTPS proxy listening on dckdevdck075:12443
2017/04/27 10:45:10 [INFO] Writing access log to stdout
2017/04/27 10:45:10 [INFO] Using routing strategy "rr"
2017/04/27 10:45:10 [INFO] Using route matching "prefix"
2017/04/27 10:45:10 [FATAL] listen: Fail to listen. listen tcp 192.163.167.143:12443: bind: address already in use
2017/04/27 10:45:10 [FATAL] accept tcp 192.163.167.143:12443: use of closed network connection
2017/04/27 10:45:10 [INFO] consul: Deregistering fabio
2017/04/27 10:45:10 [FATAL] ui: http: Server closed
-bash-4.2$ cd bin/
-bash-4.2$ rm fabio
-bash-4.2$ ls
fabio-1.4.2-go1.8.1-linux_amd64  fabio-1.4.3-go1.8.1-linux_amd64  fabio-custom
-bash-4.2$ ln -s fabio-1.4.3-go1.8.1-linux_amd64 fabio
-bash-4.2$ netstat -an | grep 12443
-bash-4.2$ ps -edf | grep fabio
webdev01 28201 26465  0 10:49 pts/2    00:00:00 grep --color=auto fabio
-bash-4.2$ cd ..
-bash-4.2$ ./fabio.start start
 Starting  Fabio
-bash-4.2$ tail -f logs/fabio.out
+ route add l7demo1-tomcatfabio tomcat-fabio.mydomain.com/ https://dckdevdck012:30030 opts "proto=https tlsskipverify=true"
2017/04/27 10:49:41 [INFO] consul: Registered fabio with id "l7demo1-dckdevdck075-9998"
2017/04/27 10:49:41 [INFO] consul: Registered fabio with address "192.163.167.143"
2017/04/27 10:49:41 [INFO] consul: Registered fabio with tags ""
2017/04/27 10:49:41 [INFO] consul: Registered fabio with health check to "http://[192.163.167.143]:9998/health"
2017/04/27 10:49:41 [INFO] HTTPS proxy listening on dckdevdck075:12443
2017/04/27 10:49:41 [INFO] Writing access log to stdout
2017/04/27 10:49:41 [INFO] Using routing strategy "rr"
2017/04/27 10:49:41 [INFO] Using route matching "prefix"
2017/04/27 10:49:41 [INFO] cert: Store has certificates for ["tomcat-fabio.mydomain.com,tomcat-sni.mydomain.com"]

deuch · 2017-04-27T15:24:44Z

Find something with the proxy address :

If i put 2 listeners in proxy.addr fabio fails to start ...

proxy.cs=cs=ssl-vault;type=vault;cert=secret/fabiodemo-001/certs;renewtoken=900s
proxy.addr = dckdevdck075:10080;proto=http,dckdevdck075:10443;proto=https;cs=ssl-vault

With this config it works with your patch (only 1 listener):

proxy.cs=cs=ssl-vault;type=vault;cert=secret/fabiodemo-001/certs;renewtoken=900s
proxy.addr = dckdevdck075:10443;proto=https;cs=ssl-vault

magiconair · 2017-04-28T14:18:47Z

That looks strange. I'll have a look.

magiconair · 2017-04-28T14:21:36Z

also fails like this

./fabio -proxy.addr ':9999,:10000'
...
2017/04/28 07:19:14 [FATAL] listen: Fail to listen. listen tcp :10000: bind: address already in use
2017/04/28 07:19:14 [FATAL] http: Server closed
2017/04/28 07:19:14 [FATAL] ui: http: Server closed

I'll open another ticket. Thanks for finding this.

magiconair · 2017-04-28T14:44:29Z

I've opened #279 and found the issue.

Add the 'renewtoken' option for vault certificate sources to make the token renew interval configurable. Fixes #274

magiconair · 2017-04-28T14:56:49Z

I've fixed #279 and rebased the patch. Could you please try again?

deuch · 2017-05-02T15:25:56Z

Hi, i've made some test and i think it doesn't work as expected. In fact, the renewSelf function is called in the load function (vault_source.go) each time you want to refresh the certificates list.
In my configuration i've a lease of 20mn for a token , i set the refresh to 1m and the renewtoken to 7mn in Fabio :

proxy.cs=cs=ssl-vault;type=vault;cert=secret/fabiodemo-001/certs;renewtoken=7m;refresh=1m

But at the end, the token is renew every minutes :

{"time":"2017-05-02T15:22:10Z","type":"response","error":"","auth":{"client_token":"","accessor":"","display_name":"approle","policies":["default","fabiodemo-001/fabio-certs"],"metadata":{}},"request":{"id":"b849194f-1e95-7209-87da-3158bbba1def","operation":"update","client_token":"hmac-sha256:97a2695fb7001d40834599dee3d3b8fe3b0928747c4e026382c0662e93d04ccf","path":"auth/token/renew-self","data":{"increment":"hmac-sha256:58e823f12d627954719c9004391839f8cb6d39d9efa5269727700c04026b9ee7"},"remote_address":"192.160.120.50","wrap_ttl":0},"response":{"auth":{"client_token":"hmac-sha256:97a2695fb7001d40834599dee3d3b8fe3b0928747c4e026382c0662e93d04ccf","accessor":"hmac-sha256:ffbdbe987b69e3fde640542b39ee04a69ead65572b841a2469716e07125378e5","display_name":"approle","policies":["default","fabiodemo-001/fabio-certs"],"metadata":{}}}}
{"time":"2017-05-02T15:23:10Z","type":"response","error":"","auth":{"client_token":"","accessor":"","display_name":"approle","policies":["default","fabiodemo-001/fabio-certs"],"metadata":{}},"request":{"id":"1a9b65c7-2cde-1d6d-77a3-6c308fed0973","operation":"update","client_token":"hmac-sha256:97a2695fb7001d40834599dee3d3b8fe3b0928747c4e026382c0662e93d04ccf","path":"auth/token/renew-self","data":{"increment":"hmac-sha256:58e823f12d627954719c9004391839f8cb6d39d9efa5269727700c04026b9ee7"},"remote_address":"192.160.120.49","wrap_ttl":0},"response":{"auth":{"client_token":"hmac-sha256:97a2695fb7001d40834599dee3d3b8fe3b0928747c4e026382c0662e93d04ccf","accessor":"hmac-sha256:ffbdbe987b69e3fde640542b39ee04a69ead65572b841a2469716e07125378e5","display_name":"approle","policies":["default","fabiodemo-001/fabio-certs"],"metadata":{}}}}

deuch · 2017-05-09T07:31:15Z

Any update on this topic ?

pschultz · 2017-06-27T19:59:18Z

Renewing the token before further API calls are made is fine. Setting renewtoken=7m just ensures that the token is valid for at least another 7 minutes before the certificates are looked up. If the token's TTL is longer than that, nothing happens. What did you expect to happen instead?

deuch · 2017-06-27T20:18:13Z

I want to decorelate the certificate renew and the token renewal. I can set the renewal token to 20mn for example and certificate every 1mn. But in the current implementation, the token is refreshed everytime the certificates are updated. So the Vault logs are flooded by token renewal. And i do not need to to refresh every minute ...

So i would like to have those parameters working as expected.

Don't attempt Vault token renewals if the token isn't renewable. If it is, don't renew the token with each refresh; only do so if the token would expire shortly. Increase the token's lifetime by its original TTL.

pschultz · 2017-06-28T14:03:40Z

Sorry about the commit noise.

@deuch Mind taking a look at #314?

magiconair added a commit that referenced this issue Apr 26, 2017

Issue #274: Make token renew interval for vault source configurable

a5ebe0e

Add the 'renewtoken' option for vault certificate sources to make the token renew interval configurable. Fixes #274

magiconair added a commit that referenced this issue Apr 26, 2017

Issue #274: Make token renew interval for vault source configurable

7729dd1

Add the 'renewtoken' option for vault certificate sources to make the token renew interval configurable. Fixes #274

magiconair added a commit that referenced this issue Apr 28, 2017

Issue #274: Make token renew interval for vault source configurable

7764eaf

Add the 'renewtoken' option for vault certificate sources to make the token renew interval configurable. Fixes #274

magiconair closed this as completed in d6d0a52 Jun 29, 2017

magiconair added this to the 1.5.1 milestone Oct 10, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add parameter to vault token renewal #274

Add parameter to vault token renewal #274

deuch commented Apr 26, 2017

magiconair commented Apr 26, 2017

magiconair commented Apr 26, 2017

deuch commented Apr 27, 2017 •

edited by magiconair

Loading

magiconair commented Apr 27, 2017

deuch commented Apr 27, 2017 •

edited

Loading

deuch commented Apr 27, 2017 •

edited by magiconair

Loading

magiconair commented Apr 28, 2017

magiconair commented Apr 28, 2017

magiconair commented Apr 28, 2017

magiconair commented Apr 28, 2017

deuch commented May 2, 2017 •

edited

Loading

deuch commented May 9, 2017

pschultz commented Jun 27, 2017

deuch commented Jun 27, 2017

pschultz commented Jun 28, 2017

Add parameter to vault token renewal #274

Add parameter to vault token renewal #274

Comments

deuch commented Apr 26, 2017

magiconair commented Apr 26, 2017

magiconair commented Apr 26, 2017

deuch commented Apr 27, 2017 • edited by magiconair Loading

magiconair commented Apr 27, 2017

deuch commented Apr 27, 2017 • edited Loading

deuch commented Apr 27, 2017 • edited by magiconair Loading

magiconair commented Apr 28, 2017

magiconair commented Apr 28, 2017

magiconair commented Apr 28, 2017

magiconair commented Apr 28, 2017

deuch commented May 2, 2017 • edited Loading

deuch commented May 9, 2017

pschultz commented Jun 27, 2017

deuch commented Jun 27, 2017

pschultz commented Jun 28, 2017

deuch commented Apr 27, 2017 •

edited by magiconair

Loading

deuch commented Apr 27, 2017 •

edited

Loading

deuch commented Apr 27, 2017 •

edited by magiconair

Loading

deuch commented May 2, 2017 •

edited

Loading