fleet makes too many etcd requests

[bcwaldon]

Here's a sample of a log from the engine doing a single reconciliation:

Aug 03 01:22:47 core-01 fleet[3310]: I0803 01:22:47.340010 03310 dumb_reconciler.go:39] Polling Registry for actionable work
Aug 03 01:22:47 core-01 fleet[3310]: I0803 01:22:47.340092 03310 client.go:353] etcd: sending HTTP request GET http://localhost:4001/v2/keys/_coreos.com/fleet/job?consistent=true&recursive=true&sorted=true
Aug 03 01:22:47 core-01 fleet[3310]: I0803 01:22:47.350125 03310 client.go:360] etcd: recv response from GET http://localhost:4001/v2/keys/_coreos.com/fleet/job?consistent=true&recursive=true&sorted=true: 200 OK
Aug 03 01:22:47 core-01 fleet[3310]: I0803 01:22:47.351326 03310 client.go:353] etcd: sending HTTP request GET http://localhost:4001/v2/keys/_coreos.com/fleet/unit/07e4d061aff4cb34d7be5e69d293dfb5dfa4cc44?consistent=true&recursive=true&sorted=false
Aug 03 01:22:47 core-01 fleet[3310]: I0803 01:22:47.351680 03310 client.go:360] etcd: recv response from GET http://localhost:4001/v2/keys/_coreos.com/fleet/unit/07e4d061aff4cb34d7be5e69d293dfb5dfa4cc44?consistent=true&recursive=true&sorted=false: 200 OK
Aug 03 01:22:47 core-01 fleet[3310]: I0803 01:22:47.351738 03310 client.go:353] etcd: sending HTTP request GET http://localhost:4001/v2/keys/_coreos.com/fleet/job/foo.service/target-state?consistent=true&recursive=false&sorted=false
Aug 03 01:22:47 core-01 fleet[3310]: I0803 01:22:47.352400 03310 client.go:360] etcd: recv response from GET http://localhost:4001/v2/keys/_coreos.com/fleet/job/foo.service/target-state?consistent=true&recursive=false&sorted=false: 200 OK
Aug 03 01:22:47 core-01 fleet[3310]: I0803 01:22:47.352438 03310 client.go:353] etcd: sending HTTP request GET http://localhost:4001/v2/keys/_coreos.com/fleet/job/foo.service/target?consistent=true&recursive=true&sorted=false
Aug 03 01:22:47 core-01 fleet[3310]: I0803 01:22:47.352966 03310 client.go:360] etcd: recv response from GET http://localhost:4001/v2/keys/_coreos.com/fleet/job/foo.service/target?consistent=true&recursive=true&sorted=false: 200 OK
Aug 03 01:22:47 core-01 fleet[3310]: I0803 01:22:47.353132 03310 client.go:353] etcd: sending HTTP request GET http://localhost:4001/v2/keys/_coreos.com/fleet/state/foo.service?consistent=true&recursive=true&sorted=false
Aug 03 01:22:47 core-01 fleet[3310]: I0803 01:22:47.354798 03310 client.go:360] etcd: recv response from GET http://localhost:4001/v2/keys/_coreos.com/fleet/state/foo.service?consistent=true&recursive=true&sorted=false: 200 OK
Aug 03 01:22:47 core-01 fleet[3310]: I0803 01:22:47.354918 03310 client.go:353] etcd: sending HTTP request GET http://localhost:4001/v2/keys/_coreos.com/fleet/job/foo.service/target?consistent=true&recursive=true&sorted=false
Aug 03 01:22:47 core-01 fleet[3310]: I0803 01:22:47.356104 03310 client.go:360] etcd: recv response from GET http://localhost:4001/v2/keys/_coreos.com/fleet/job/foo.service/target?consistent=true&recursive=true&sorted=false: 200 OK
Aug 03 01:22:47 core-01 fleet[3310]: I0803 01:22:47.356159 03310 client.go:353] etcd: sending HTTP request GET http://localhost:4001/v2/keys/_coreos.com/fleet/state/foo.service?consistent=true&recursive=true&sorted=false
Aug 03 01:22:47 core-01 fleet[3310]: I0803 01:22:47.356622 03310 client.go:360] etcd: recv response from GET http://localhost:4001/v2/keys/_coreos.com/fleet/state/foo.service?consistent=true&recursive=true&sorted=false: 200 OK
Aug 03 01:22:47 core-01 fleet[3310]: I0803 01:22:47.356636 03310 client.go:353] etcd: sending HTTP request GET http://localhost:4001/v2/keys/_coreos.com/fleet/job/foo.service/job-state?consistent=true&recursive=false&sorted=false
Aug 03 01:22:47 core-01 fleet[3310]: I0803 01:22:47.357487 03310 client.go:360] etcd: recv response from GET http://localhost:4001/v2/keys/_coreos.com/fleet/job/foo.service/job-state?consistent=true&recursive=false&sorted=false: 200 OK
Aug 03 01:22:47 core-01 fleet[3310]: I0803 01:22:47.357540 03310 client.go:353] etcd: sending HTTP request GET http://localhost:4001/v2/keys/_coreos.com/fleet/offer?consistent=true&recursive=true&sorted=true
Aug 03 01:22:47 core-01 fleet[3310]: I0803 01:22:47.357757 03310 client.go:360] etcd: recv response from GET http://localhost:4001/v2/keys/_coreos.com/fleet/offer?consistent=true&recursive=true&sorted=true: 200 OK
Aug 03 01:22:47 core-01 fleet[3310]: I0803 01:22:47.357801 03310 client.go:353] etcd: sending HTTP request GET http://localhost:4001/v2/keys/_coreos.com/fleet/machines?consistent=true&recursive=true&sorted=true
Aug 03 01:22:47 core-01 fleet[3310]: I0803 01:22:47.358501 03310 client.go:360] etcd: recv response from GET http://localhost:4001/v2/keys/_coreos.com/fleet/machines?consistent=true&recursive=true&sorted=true: 200 OK
Aug 03 01:22:47 core-01 fleet[3310]: I0803 01:22:47.358587 03310 engine.go:74] Engine completed reconciliation in 18.573891ms

There is a single job running in the cluster, and it's taking 10 requests to ensure it's okay. It looks like 7 of these are specific to the one job I'm running (foo.service), so I suspect fleet is actually doing 3+7N requests per reconciliation where N is the number of jobs. That's ridiculous. Ideally, this would be a constant number.

[bcwaldon]

Shouldn't have been autoclosed

[bcwaldon]

This has mostly been fixed as of v0.6.2. Remaining issues here: #804