Handoff to pending ingester did not complete cleanly #458

awh · 2017-06-14T09:57:08Z

Again whilst testing the reboot daemon the following situation arose: a cortex ingester ingester-3378111688-nyll4 with address 10.244.228.234 was drained by the reboot daemon, and a new ingester ingester-3378111688-d5lpl with address 10.244.202.96 scheduled to take its place:

From draining ingester:

time="2017-06-14T08:59:37Z" level=info msg="=== received SIGINT/SIGTERM ===
*** exiting" 
2017/06/14 08:59:37 transport: http2Server.HandleStreams failed to read frame: read tcp 10.244.228.234:9095->10.244.253.70:55116: use of closed network connection
time="2017-06-14T08:59:37Z" level=info msg="Changing ingester state from ACTIVE -> LEAVING" source="ingester_lifecycle.go:314" 
time="2017-06-14T08:59:37Z" level=error msg="Error looking for pending ingester: no pending ingesters" source="ingester_lifecycle.go:415" 
time="2017-06-14T08:59:39Z" level=error msg="Failed to flush user 2: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T08:59:42Z" level=error msg="Failed to flush user 2123: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T08:59:47Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T08:59:49Z" level=info msg="Sending chunks to 10.244.202.96:9095" source="ingester_lifecycle.go:349" 
time="2017-06-14T08:59:53Z" level=error msg="Failed to flush user 910: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T08:59:59Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:00:01Z" level=error msg="Failed to flush user 910: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:00:05Z" level=error msg="Failed to flush user 2: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:00:10Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:00:10Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:00:12Z" level=error msg="Failed to flush user 2: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:00:17Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:00:18Z" level=error msg="Failed to flush user 3166: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:00:25Z" level=error msg="Failed to flush user 2: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:00:31Z" level=error msg="Failed to flush user 910: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:00:45Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:00:47Z" level=error msg="Failed to flush user 3711: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:00:51Z" level=error msg="Failed to flush user 3708: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:00:55Z" level=error msg="Failed to flush user 910: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:00:57Z" level=error msg="Failed to flush user 2: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:00:58Z" level=error msg="Failed to flush user 3166: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:01:06Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:01:14Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:01:14Z" level=error msg="Failed to flush user 2: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:01:36Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:01:36Z" level=error msg="Failed to flush user 3165: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:01:39Z" level=error msg="Failed to flush user 3938: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:01:54Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:01:59Z" level=error msg="Failed to flush user 3165: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:02:01Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:02:04Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:02:06Z" level=error msg="Failed to flush user 3681: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:02:07Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:02:11Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:02:14Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:02:23Z" level=error msg="Failed to flush user 3966: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:02:28Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:02:37Z" level=error msg="Failed to flush user 2830: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:02:51Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:02:59Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:03:06Z" level=error msg="Failed to flush user 3630: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:03:19Z" level=error msg="Failed to flush user 3970: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:03:22Z" level=error msg="Failed to flush user 3713: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:03:23Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:03:34Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:03:35Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:03:52Z" level=error msg="Failed to flush user 3531: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:04:15Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:04:17Z" level=error msg="Failed to flush user 2: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:04:18Z" level=error msg="Failed to flush user 1282: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:04:20Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:04:36Z" level=error msg="Failed to flush user 3966: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:04:42Z" level=error msg="Failed to flush user 2848: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:04:44Z" level=error msg="Failed to flush user 3166: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:04:46Z" level=error msg="Failed to flush user 3531: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:04:49Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:04:53Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:05:19Z" level=error msg="Failed to flush user 2867: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:05:25Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:05:29Z" level=error msg="Failed to flush user 3960: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:05:36Z" level=error msg="Failed to flush user 910: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:05:51Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:05:52Z" level=error msg="Failed to flush user 910: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:05:58Z" level=error msg="Failed to flush user 3706: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:06:22Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:06:36Z" level=error msg="Failed to flush user 3531: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:06:56Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:07:11Z" level=error msg="Failed to flush user 1902: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:07:16Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:07:29Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:07:56Z" level=error msg="Failed to flush user 3391: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:07:57Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:08:55Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:08:59Z" level=error msg="Failed to flush user 2123: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:09:02Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:09:16Z" level=error msg="Failed to flush user 3869: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:10:18Z" level=error msg="Failed to flush user 3531: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:10:41Z" level=error msg="Failed to flush user 3965: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:11:00Z" level=error msg="Failed to flush user 2: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:11:02Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:11:14Z" level=error msg="Failed to flush user 3708: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:11:21Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:11:22Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:11:33Z" level=error msg="Failed to flush user 3706: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:11:34Z" level=error msg="Failed to flush user 1282: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:12:05Z" level=error msg="Failed to flush user 2650: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:12:13Z" level=error msg="Failed to flush user 2650: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:12:25Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:12:25Z" level=error msg="Failed to flush user 3968: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:12:36Z" level=error msg="Failed to flush user 3857: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:13:37Z" level=error msg="Failed to flush user 3708: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:13:45Z" level=error msg="Failed to flush user 3913: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:14:13Z" level=error msg="Failed to flush user 3869: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:14:13Z" level=error msg="Failed to flush user 910: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:14:26Z" level=error msg="Failed to flush user 910: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:14:57Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:15:14Z" level=error msg="Failed to flush user 3889: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:15:40Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:16:31Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:17:00Z" level=error msg="Failed to flush user 3871: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:17:14Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:17:16Z" level=error msg="Failed to flush user 3708: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:17:35Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:17:53Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:18:48Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:19:13Z" level=error msg="Failed to flush user 3946: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:19:15Z" level=error msg="Failed to flush user 3630: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:19:36Z" level=error msg="Failed to flush user 2950: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:21:31Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:22:34Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:22:39Z" level=error msg="Failed to flush user 3342: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:23:44Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:24:48Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:24:55Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:25:24Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:25:56Z" level=error msg="Error CASing collectors/ring, trying again 4952650" source="consul_client.go:187" 
time="2017-06-14T09:26:40Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:28:03Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:28:31Z" level=error msg="Error CASing collectors/ring, trying again 4952859" source="consul_client.go:187" 
time="2017-06-14T09:29:52Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:31:54Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:32:10Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:33:25Z" level=error msg="Failed to flush user 3745: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121" 
time="2017-06-14T09:34:44Z" level=error msg="Failed to flush user 2602: failed to write chunk after 20 retries, 1 values remaining" source="ingester_flush.go:121"

From new ingester:

time="2017-06-14T08:59:40Z" level=info msg="Entry not found in ring, adding with no tokens." source="ingester_lifecycle.go:234" 
time="2017-06-14T08:59:49Z" level=info msg="Changing ingester state from PENDING -> JOINING" source="ingester_lifecycle.go:314" 
time="2017-06-14T08:59:57Z" level=warning msg="GET /ready (500) 115.914µs" 
time="2017-06-14T09:00:07Z" level=warning msg="GET /ready (500) 87.622µs" 
time="2017-06-14T09:00:17Z" level=warning msg="GET /ready (500) 107.212µs" 
time="2017-06-14T09:00:27Z" level=warning msg="GET /ready (500) 95.875µs" 
time="2017-06-14T09:00:33Z" level=info msg="Changing ingester state from JOINING -> ACTIVE" source="ingester_lifecycle.go:314"

We can see the the terminating ingester does attempt to send chunks to the new one, but does not manage to remove itself from the ring and exit cleanly before the 40m shutdown limit is reached and it is killed by k8s, leaving the ring in an unhealthy state.

The text was updated successfully, but these errors were encountered:

awh · 2017-06-14T13:37:12Z

Based on the lack of Successfully sent chunks to or Failed to transfer chunks to another ingester log messages it would seem that the terminating ingester is getting stuck in transferChunks; could either be stuck in I/O, or unable to acquire a lock?

awh · 2017-06-14T13:58:46Z

The same thing happened again when kured rebooted the next ingester-hosting node:

https://cloud.weave.works/prom/loud-breeze-77/notebook/8c4e88ba-a6d6-481c-be68-ee83b2056004

The precipitous ramp up of chunks on the new ingester implies that the transfer was largely successful, but the sender never concluded that it was.

awh · 2017-06-14T14:26:02Z

The final step of transferChunks is stream.CloseAndRecv: https://github.com/weaveworks/cortex/blob/1ba9d6d82c0c451835f6b9f8fdeecd12ebda5646/pkg/ingester/ingester_lifecycle.go#L387

On the receiving side, we are seeing messages like this:

2017/06/14 12:25:25 grpc: Server.processUnaryRPC failed to write status stream error: code = Canceled desc = "context canceled"
2017/06/14 12:29:31 grpc: Server.processUnaryRPC failed to write status stream error: code = Canceled desc = "context canceled"
2017/06/14 12:30:17 grpc: Server.processUnaryRPC failed to write status stream error: code = Canceled desc = "context canceled"
2017/06/14 12:30:29 grpc: Server.processUnaryRPC failed to write status stream error: code = Canceled desc = "context canceled"
2017/06/14 12:32:21 grpc: Server.processUnaryRPC failed to write status stream error: code = Canceled desc = "context canceled"
...

The first message corresponds roughly with the assumed end of the transfer based on the graphs above, and continue every few minutes even up until the time of writing (14:25:00Z). Perhaps for some reason the response cannot be written back, perhaps due to the transfer time exceeding a threshold and a deadline exceeded?

awh · 2017-06-14T15:52:52Z

Goroutine dump of terminating ingester:

SIGABRT: abort
PC=0x45a953 m=2 sigcode=0

goroutine 0 [idle]:
runtime.futex(0x1435df8, 0x0, 0x7f57a7a46c48, 0x0, 0x7f5700000000, 0x45a5e6, 0x3c, 0x0, 0x7f57a7a46c90, 0x40ded0, ...)
	/usr/local/go/src/runtime/sys_linux_amd64.s:423 +0x23
runtime.futexsleep(0x1435df8, 0x7f5700000000, 0xdf8475800)
	/usr/local/go/src/runtime/os_linux.go:62 +0xd7
runtime.notetsleep_internal(0x1435df8, 0xdf8475800, 0x0)
	/usr/local/go/src/runtime/lock_futex.go:174 +0xd0
runtime.notetsleep(0x1435df8, 0xdf8475800, 0x7123d2fe46f01)
	/usr/local/go/src/runtime/lock_futex.go:194 +0x56
runtime.sysmon()
	/usr/local/go/src/runtime/proc.go:3805 +0x135
runtime.mstart1()
	/usr/local/go/src/runtime/proc.go:1179 +0x11e
runtime.mstart()
	/usr/local/go/src/runtime/proc.go:1149 +0x64

goroutine 1 [semacquire, 24 minutes]:
sync.runtime_Semacquire(0xc4201b32fc)
	/usr/local/go/src/runtime/sema.go:47 +0x34
sync.(*WaitGroup).Wait(0xc4201b32f0)
	/usr/local/go/src/sync/waitgroup.go:131 +0x7a
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).Shutdown(0xc4201b3180)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_lifecycle.go:126 +0x80
main.main()
	/go/src/github.com/weaveworks/cortex/cmd/ingester/main.go:64 +0x767

goroutine 17 [syscall, 21727 minutes, locked to thread]:
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2197 +0x1

goroutine 8 [syscall, 4 minutes]:
os/signal.signal_recv(0x1400360)
	/usr/local/go/src/runtime/sigqueue.go:116 +0x104
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 54 [select]:
github.com/weaveworks/cortex/pkg/util.(*hashBucketHistogram).loop(0xc420272b40)
	/go/src/github.com/weaveworks/cortex/pkg/util/hash_bucket_histogram.go:87 +0x1e5
created by github.com/weaveworks/cortex/pkg/util.NewHashBucketHistogram
	/go/src/github.com/weaveworks/cortex/pkg/util/hash_bucket_histogram.go:64 +0x18a

goroutine 85 [select]:
github.com/weaveworks/cortex/pkg/chunk.(*MemcacheClient).updateLoop(0xc420113400, 0xdf8475800, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/memcache_client.go:78 +0x1e4
created by github.com/weaveworks/cortex/pkg/chunk.NewMemcacheClient
	/go/src/github.com/weaveworks/cortex/pkg/chunk/memcache_client.go:63 +0x287

goroutine 87 [select, 21727 minutes]:
github.com/weaveworks/cortex/pkg/chunk.(*Cache).writeBackLoop(0xc42036a300)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_cache.go:209 +0x2a9
created by github.com/weaveworks/cortex/pkg/chunk.NewCache
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_cache.go:107 +0x1be

goroutine 86 [select, 21727 minutes]:
github.com/weaveworks/cortex/pkg/chunk.(*Cache).writeBackLoop(0xc42036a300)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_cache.go:209 +0x2a9
created by github.com/weaveworks/cortex/pkg/chunk.NewCache
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_cache.go:107 +0x1be

goroutine 88 [select, 21727 minutes]:
github.com/weaveworks/cortex/pkg/chunk.(*Cache).writeBackLoop(0xc42036a300)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_cache.go:209 +0x2a9
created by github.com/weaveworks/cortex/pkg/chunk.NewCache
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_cache.go:107 +0x1be

goroutine 89 [select, 21727 minutes]:
github.com/weaveworks/cortex/pkg/chunk.(*Cache).writeBackLoop(0xc42036a300)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_cache.go:209 +0x2a9
created by github.com/weaveworks/cortex/pkg/chunk.NewCache
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_cache.go:107 +0x1be

goroutine 90 [select, 21727 minutes]:
github.com/weaveworks/cortex/pkg/chunk.(*Cache).writeBackLoop(0xc42036a300)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_cache.go:209 +0x2a9
created by github.com/weaveworks/cortex/pkg/chunk.NewCache
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_cache.go:107 +0x1be

goroutine 91 [select, 21727 minutes]:
github.com/weaveworks/cortex/pkg/chunk.(*Cache).writeBackLoop(0xc42036a300)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_cache.go:209 +0x2a9
created by github.com/weaveworks/cortex/pkg/chunk.NewCache
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_cache.go:107 +0x1be

goroutine 92 [select, 21727 minutes]:
github.com/weaveworks/cortex/pkg/chunk.(*Cache).writeBackLoop(0xc42036a300)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_cache.go:209 +0x2a9
created by github.com/weaveworks/cortex/pkg/chunk.NewCache
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_cache.go:107 +0x1be

goroutine 93 [select, 21727 minutes]:
github.com/weaveworks/cortex/pkg/chunk.(*Cache).writeBackLoop(0xc42036a300)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_cache.go:209 +0x2a9
created by github.com/weaveworks/cortex/pkg/chunk.NewCache
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_cache.go:107 +0x1be

goroutine 94 [select, 21727 minutes]:
github.com/weaveworks/cortex/pkg/chunk.(*Cache).writeBackLoop(0xc42036a300)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_cache.go:209 +0x2a9
created by github.com/weaveworks/cortex/pkg/chunk.NewCache
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_cache.go:107 +0x1be

goroutine 95 [select, 21727 minutes]:
github.com/weaveworks/cortex/pkg/chunk.(*Cache).writeBackLoop(0xc42036a300)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_cache.go:209 +0x2a9
created by github.com/weaveworks/cortex/pkg/chunk.NewCache
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_cache.go:107 +0x1be

goroutine 96 [sleep]:
time.Sleep(0xa1abff331)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc509c1bbf0, 0xc578762480, 0x1, 0x1, 0xc5a5f99b90, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc509c1bbf0, 0xc578762480, 0x1, 0x1, 0x15ca6150958, 0x15ca6157e85)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc509c1bbf0, 0xa4742f928e0ab7b6, 0xc5a5f99b90, 0xc48db54fe0, 0x1, 0x1, 0xc57886ff28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc42374f5d8, 0x4, 0xa4742f928e0ab7b6, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 97 [sleep]:
time.Sleep(0xd43ca1ef)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc5471016e0, 0xc4bfb88e40, 0x1, 0x1, 0xc50a1e5320, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc5471016e0, 0xc4bfb88e40, 0x1, 0x1, 0x15ca5d46463, 0x15ca5d513df)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc5471016e0, 0x8fbf671c5298dfdb, 0xc50a1e5320, 0xc42186c0d0, 0x1, 0x1, 0xc5d76eff28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc43f1c5ea0, 0x4, 0x8fbf671c5298dfdb, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x1)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 98 [sleep]:
time.Sleep(0xba43b7400)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc4ddebdec0, 0xc486826300, 0x1, 0x1, 0xc4390bb320, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc4ddebdec0, 0xc486826300, 0x1, 0x1, 0x15ca5e5a2cf, 0x15ca5e5a2cf)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc4ddebdec0, 0xf3ae13b47263d3e8, 0xc4390bb320, 0xc505ba8568, 0x1, 0x1, 0xc5dc6f3f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc42374f5d8, 0x4, 0xf3ae13b47263d3e8, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x2)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 99 [sleep]:
time.Sleep(0x7cbc51a61)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc43b560300, 0xc4e5edb920, 0x1, 0x1, 0xc57d50ba70, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc43b560300, 0xc4e5edb920, 0x1, 0x1, 0x15ca6ddbe6f, 0x15ca6ddbe6f)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc43b560300, 0xb4737b910b0b35e1, 0xc57d50ba70, 0xc470b11150, 0x1, 0x1, 0xc5c3d19f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc42374f5d8, 0x4, 0xb4737b910b0b35e1, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x3)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 100 [sleep]:
time.Sleep(0x985fad5c4)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc4342044e0, 0xc502b73440, 0x1, 0x1, 0xc53df83b60, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc4342044e0, 0xc502b73440, 0x1, 0x1, 0x15ca630e455, 0x15ca631598f)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc4342044e0, 0x5e5d1eaf9d7c7b44, 0xc53df83b60, 0xc57bf33d78, 0x1, 0x1, 0xc53625ff28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc42374f5d8, 0x4, 0x5e5d1eaf9d7c7b44, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x4)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 101 [sleep]:
time.Sleep(0x6638c56d9)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc4fdd7dcb0, 0xc45a909080, 0x1, 0x1, 0xc5a5f61500, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc4fdd7dcb0, 0xc45a909080, 0x1, 0x1, 0x15ca6a07c41, 0x15ca6a12c06)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc4fdd7dcb0, 0x62b90803b11d6077, 0xc5a5f61500, 0xc49b5b4430, 0x1, 0x1, 0xc5d65ebf28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc42374f5d8, 0x4, 0x62b90803b11d6077, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x5)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 102 [sleep]:
time.Sleep(0x4ba3c92c)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc4ac683f20, 0xc55b839980, 0x1, 0x1, 0xc4a456d470, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc4ac683f20, 0xc55b839980, 0x1, 0x1, 0x15ca6800b09, 0x15ca6dd6d49)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc4ac683f20, 0x156fd97aae4589f0, 0xc4a456d470, 0xc526daad30, 0x1, 0x2, 0xc5e8ab3f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc4aa8b6718, 0x4, 0x156fd97aae4589f0, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x6)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 103 [sleep]:
time.Sleep(0x10604c87)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc429f0d9b0, 0xc5242c5550, 0x2, 0x2, 0xc4a997a0c0, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc429f0d9b0, 0xc5242c5550, 0x2, 0x2, 0x15ca68ffa0e, 0x15ca6906f3d)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc429f0d9b0, 0xedca7cd7f9935be3, 0xc4a997a0c0, 0xc4af0dc560, 0x2, 0x2, 0xc5dc6f3f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc454eef7c8, 0x4, 0xedca7cd7f9935be3, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x7)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 104 [sleep]:
time.Sleep(0x1a2059c64)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc435e4bdd0, 0xc4839b2de0, 0x1, 0x1, 0xc43ec30f30, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc435e4bdd0, 0xc4839b2de0, 0x1, 0x1, 0x15ca6576cb3, 0x15ca657a74b)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc435e4bdd0, 0xad2c2c30c6d665ec, 0xc43ec30f30, 0xc551077588, 0x1, 0x1, 0xc5d9793f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc42374f5d8, 0x4, 0xad2c2c30c6d665ec, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x8)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 105 [sleep]:
time.Sleep(0xba43b7400)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc51f70ec30, 0xc504bb47e0, 0x1, 0x1, 0xc59b3fa180, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc51f70ec30, 0xc504bb47e0, 0x1, 0x1, 0x15ca66889d8, 0x15ca66889d8)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc51f70ec30, 0x5f94f852095cb58b, 0xc59b3fa180, 0xc467b28a78, 0x1, 0x1, 0xc5ce2b7f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc42374f5d8, 0x4, 0x5f94f852095cb58b, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x9)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 106 [sleep]:
time.Sleep(0xba43b7400)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc4f9e837d0, 0xc4bc2fb380, 0x1, 0x1, 0xc5b4e93410, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc4f9e837d0, 0xc4bc2fb380, 0x1, 0x1, 0x15ca6d5c473, 0x15ca6d5ff0a)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc4f9e837d0, 0xbd599d0b9ae337a8, 0xc5b4e93410, 0xc59b6ed100, 0x1, 0x1, 0xc5cc463f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc42374f5d8, 0x4, 0xbd599d0b9ae337a8, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0xa)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 108 [sleep]:
time.Sleep(0xf3e9ee5c)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc524d5ae70, 0xc57cdb14a0, 0x1, 0x1, 0xc45fba2270, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc524d5ae70, 0xc57cdb14a0, 0x1, 0x1, 0x15ca5ed3605, 0x15ca5ed3605)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc524d5ae70, 0xe7fecee11ddf99ce, 0xc45fba2270, 0xc4e1bf48b8, 0x1, 0x1, 0xc5d4e37f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc42374f5d8, 0x4, 0xe7fecee11ddf99ce, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0xc)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 109 [sleep]:
time.Sleep(0xba43b7400)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc4f5d58570, 0xc4925541e0, 0x1, 0x1, 0xc5cd363c50, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc4f5d58570, 0xc4925541e0, 0x1, 0x1, 0x15ca6c547e6, 0x15ca6c547e6)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc4f5d58570, 0x2b9662ecdc747ed1, 0xc5cd363c50, 0xc485094158, 0x1, 0x1, 0xc5bdc15f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc42374f5d8, 0x4, 0x2b9662ecdc747ed1, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0xd)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 110 [sleep]:
time.Sleep(0xba43b7400)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc4b6073440, 0xc577d43a40, 0x1, 0x1, 0xc49563c300, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc4b6073440, 0xc577d43a40, 0x1, 0x1, 0x15ca6abd46c, 0x15ca6ac49b5)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc4b6073440, 0x2780fb7571f45f48, 0xc49563c300, 0xc420198028, 0x1, 0x1, 0xc58535ff28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc42374f5d8, 0x4, 0x2780fb7571f45f48, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0xe)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 111 [sleep]:
time.Sleep(0xba43b7400)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc42c37b050, 0xc550a30420, 0x4, 0x4, 0xc4823659e0, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc42c37b050, 0xc550a30420, 0x4, 0x4, 0x15ca70a3086, 0x15ca7347e91)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc42c37b050, 0xb6dbc84d78b912db, 0xc4823659e0, 0xc484381440, 0x4, 0x8, 0xc600edff28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc47f159430, 0x4, 0xb6dbc84d78b912db, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0xf)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 112 [sleep]:
time.Sleep(0x17536b17)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc4d4d40510, 0xc5c1927a20, 0x4, 0x4, 0xc53aa13740, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc4d4d40510, 0xc5c1927a20, 0x4, 0x4, 0x15ca6dbf41a, 0x15ca70aa53a)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc4d4d40510, 0x83cada4473e902fc, 0xc53aa13740, 0xc56c517600, 0x4, 0x8, 0xc5d7db5f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc4686d0000, 0x1, 0x83cada4473e902fc, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x10)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 113 [sleep]:
time.Sleep(0xb6b0b56d8)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc551431770, 0xc50b373020, 0x1, 0x1, 0xc4f60a3230, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc551431770, 0xc50b373020, 0x1, 0x1, 0x15ca681c48b, 0x15ca681c48b)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc551431770, 0x2759638eb2cfa7dd, 0xc4f60a3230, 0xc539f8d6e0, 0x1, 0x1, 0xc594b59f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc42374f5d8, 0x4, 0x2759638eb2cfa7dd, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x11)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 114 [sleep]:
time.Sleep(0xa13a45331)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc52b363470, 0xc423b75c20, 0x3, 0x3, 0xc51386c750, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc52b363470, 0xc423b75c20, 0x3, 0x3, 0x15ca6ccfc01, 0x15ca719e381)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc52b363470, 0x4a26639756141c8e, 0xc51386c750, 0xc598e0bb80, 0x3, 0x4, 0xc5d675df28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc475acd030, 0x4, 0x4a26639756141c8e, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x12)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 115 [sleep]:
time.Sleep(0xba43b7400)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc50fde3aa0, 0xc51ed16480, 0x1, 0x1, 0xc4a235b140, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc50fde3aa0, 0xc51ed16480, 0x1, 0x1, 0x15ca6c45f53, 0x15ca6c499eb)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc50fde3aa0, 0x8601cb2639bc9685, 0xc4a235b140, 0xc4eeaa0078, 0x1, 0x1, 0xc5b7ae3f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc42374f5d8, 0x4, 0x8601cb2639bc9685, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x13)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 116 [sleep]:
time.Sleep(0xd26d6da3)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc43fad4150, 0xc451b03560, 0x1, 0x1, 0xc56f0028d0, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc43fad4150, 0xc451b03560, 0x1, 0x1, 0x15ca6cd7107, 0x15ca6cd7107)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc43fad4150, 0xd6bab099e8a99b3c, 0xc56f0028d0, 0xc42020f698, 0x1, 0x1, 0xc5d9303f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc42374f5d8, 0x4, 0xd6bab099e8a99b3c, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x14)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 117 [sleep]:
time.Sleep(0x6c8d2f0b1)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc54e611980, 0xc47d66dda0, 0x1, 0x1, 0xc56d7e0780, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc54e611980, 0xc47d66dda0, 0x1, 0x1, 0x15ca68fc8d0, 0x15ca6907898)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc54e611980, 0x7d8ef36ecaf2a025, 0xc56d7e0780, 0xc569e17b50, 0x1, 0x1, 0xc5e8ab3f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc4686d0000, 0x1, 0x7d8ef36ecaf2a025, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x15)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 118 [sleep]:
time.Sleep(0xba43b7400)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc4af712780, 0xc526ad36e0, 0x1, 0x1, 0xc5312a9fb0, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc4af712780, 0xc526ad36e0, 0x1, 0x1, 0x15ca68fde6f, 0x15ca690538c)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc4af712780, 0x8a5fb952f563cfb0, 0xc5312a9fb0, 0xc57c612c78, 0x1, 0x1, 0xc61ce41f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc49f0a0660, 0x4, 0x8a5fb952f563cfb0, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x16)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 119 [sleep]:
time.Sleep(0xba43b7400)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc452df1080, 0xc541488540, 0x1, 0x1, 0xc535fa9470, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc452df1080, 0xc541488540, 0x1, 0x1, 0x15ca5ea595e, 0x15ca5ea93f6)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc452df1080, 0x69d263927cf2fac5, 0xc535fa9470, 0xc4b7cc96d0, 0x1, 0x1, 0xc5cd7ddf28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc42374f5d8, 0x4, 0x69d263927cf2fac5, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x17)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 120 [sleep]:
time.Sleep(0xba43b7400)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc55c263200, 0xc4b61c0060, 0x1, 0x1, 0xc4f7439590, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc55c263200, 0xc4b61c0060, 0x1, 0x1, 0x15ca68fad00, 0x15ca6905cc8)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc55c263200, 0x9208981af2f460f0, 0xc4f7439590, 0xc56bcbc840, 0x1, 0x1, 0xc5bdab1f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc49f0a0660, 0x4, 0x9208981af2f460f0, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x18)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 121 [sleep]:
time.Sleep(0x327b8be0c)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc4b4e25260, 0xc517299da0, 0x1, 0x1, 0xc427f46630, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc4b4e25260, 0xc517299da0, 0x1, 0x1, 0x15ca5ca7378, 0x15ca64471f0)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc4b4e25260, 0x5dbe1a20ed3d5b69, 0xc427f46630, 0xc4e18e3868, 0x1, 0x1, 0xc5b8253f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc4686d0000, 0x1, 0x5dbe1a20ed3d5b69, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x19)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 122 [sleep]:
time.Sleep(0xba43b7400)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc514151fb0, 0xc5567fd6e0, 0x1, 0x1, 0xc460acc360, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc514151fb0, 0xc5567fd6e0, 0x1, 0x1, 0x15ca67ff361, 0x15ca680a326)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc514151fb0, 0xfd2cc70916f96f54, 0xc460acc360, 0xc47c5d2b18, 0x1, 0x1, 0xc5d7db5f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc44f30b2e0, 0x4, 0xfd2cc70916f96f54, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x1a)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 123 [sleep]:
time.Sleep(0x714956ad7)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc4e5a40030, 0xc53485ee40, 0x1, 0x1, 0xc5a6e787e0, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc4e5a40030, 0xc53485ee40, 0x1, 0x1, 0x15ca68fd26c, 0x15ca690479c)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc4e5a40030, 0xc6f5793c9e1d55bf, 0xc5a6e787e0, 0xc578b9cf80, 0x1, 0x1, 0xc61ce65f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc47f31c730, 0x4, 0xc6f5793c9e1d55bf, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x1b)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 124 [sleep]:
time.Sleep(0xba43b7400)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc518077800, 0xc4fc7faae0, 0x1, 0x1, 0xc44b5fc300, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc518077800, 0xc4fc7faae0, 0x1, 0x1, 0x15ca600a048, 0x15ca6011579)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc518077800, 0xdfba7ab135bff420, 0xc44b5fc300, 0xc549bbe6a8, 0x1, 0x1, 0xc5c7f03f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc42374f5d8, 0x4, 0xdfba7ab135bff420, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x1c)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 125 [sleep]:
time.Sleep(0x2f6874a47)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc4e48fb170, 0xc502be5340, 0x4, 0x4, 0xc44c022c30, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc4e48fb170, 0xc502be5340, 0x4, 0x4, 0x15ca708a130, 0x15ca7232e10)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc4e48fb170, 0xee0f6b3e7ee74cc7, 0xc44c022c30, 0xc50ad88f60, 0x4, 0x6, 0xc5bc1bdf28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc49d37d5b0, 0x4, 0xee0f6b3e7ee74cc7, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x1d)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 127 [sleep]:
time.Sleep(0xb030de323)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc44e9b5b00, 0xc516ec78c0, 0x1, 0x1, 0xc420db2450, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc44e9b5b00, 0xc516ec78c0, 0x1, 0x1, 0x15ca6b04a73, 0x15ca70c4d1c)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc44e9b5b00, 0x2c05949bab544dd9, 0xc420db2450, 0xc5c0abade0, 0x1, 0x2, 0xc5bbd91f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc42001ae64, 0x4, 0x2c05949bab544dd9, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x1f)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 128 [sleep]:
time.Sleep(0x2f6a71ea)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc45b58d410, 0xc5086fbaa0, 0x1, 0x1, 0xc427df4990, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc45b58d410, 0xc5086fbaa0, 0x1, 0x1, 0x15ca6d8345d, 0x15ca731b485)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc45b58d410, 0x895d62425a55d938, 0xc427df4990, 0xc5d60ed5c0, 0x1, 0x2, 0xc5b8ef7f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc42374f5d8, 0x4, 0x895d62425a55d938, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x20)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 129 [sleep]:
time.Sleep(0x1d22415e)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc42ce803c0, 0xc525156540, 0x1, 0x1, 0xc481a306f0, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc42ce803c0, 0xc525156540, 0x1, 0x1, 0x15ca5d47d55, 0x15ca5d52d22)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc42ce803c0, 0x9cc8bc1a79b98dbf, 0xc481a306f0, 0xc430070908, 0x1, 0x1, 0xc5dafc3f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc42001ae64, 0x4, 0x9cc8bc1a79b98dbf, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x21)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 130 [sleep]:
time.Sleep(0x337648607)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc44a339e30, 0xc47c967980, 0x1, 0x1, 0xc4b32337d0, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc44a339e30, 0xc47c967980, 0x1, 0x1, 0x15ca6d1cbbe, 0x15ca6d27b85)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc44a339e30, 0xe7e087a3896dae6a, 0xc4b32337d0, 0xc538fdbf98, 0x1, 0x1, 0xc5db0e9f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc42374f5d8, 0x4, 0xe7e087a3896dae6a, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x22)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 131 [sleep]:
time.Sleep(0x34c0b583)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc42d2c9080, 0xc46caf6360, 0x1, 0x1, 0xc4328f0930, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc42d2c9080, 0xc46caf6360, 0x1, 0x1, 0x15ca63d8c6d, 0x15ca63dc705)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc42d2c9080, 0x48c93a826d42e72b, 0xc4328f0930, 0xc42e134f38, 0x1, 0x1, 0xc5b097ff28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc42374f5d8, 0x4, 0x48c93a826d42e72b, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x23)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 132 [sleep]:
time.Sleep(0x7c01de7af)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc5714fed50, 0xc4f89a72c0, 0x1, 0x1, 0xc452533e00, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc5714fed50, 0xc4f89a72c0, 0x1, 0x1, 0x15ca680a9a7, 0x15ca6eff437)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc5714fed50, 0xb05e0ba5ec39111e, 0xc452533e00, 0xc5319deda0, 0x1, 0x2, 0xc58be4df28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc44d952ed8, 0x4, 0xb05e0ba5ec39111e, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x24)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 133 [sleep]:
time.Sleep(0xafd2eacdd)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc4cfe07d40, 0xc518079c30, 0x2, 0x2, 0xc4cbea19e0, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc4cfe07d40, 0xc518079c30, 0x2, 0x2, 0x15ca6901cf5, 0x15ca690578d)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc4cfe07d40, 0x59f75d10d791b597, 0xc4cbea19e0, 0xc5ba8784b0, 0x2, 0x2, 0xc57a6b1f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc4bf2ab038, 0x4, 0x59f75d10d791b597, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x25)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 134 [sleep]:
time.Sleep(0xba43b7400)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc45d52db60, 0xc45739f980, 0x1, 0x1, 0xc555e07470, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc45d52db60, 0xc45739f980, 0x1, 0x1, 0x15ca6776f07, 0x15ca6781ecf)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc45d52db60, 0x696a8aadf620bb74, 0xc555e07470, 0xc50b2fa928, 0x1, 0x1, 0xc5e7b15f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc42374f5d8, 0x4, 0x696a8aadf620bb74, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x26)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 135 [sleep]:
time.Sleep(0xba43b7400)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc5254c3f80, 0xc515e14720, 0x1, 0x1, 0xc572889e90, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc5254c3f80, 0xc515e14720, 0x1, 0x1, 0x15ca6ab6401, 0x15ca6ae2322)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc5254c3f80, 0xfb0e1bbe091b7c75, 0xc572889e90, 0xc540543fc8, 0x1, 0x1, 0xc5edaf1f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc4c156a6b0, 0x4, 0xfb0e1bbe091b7c75, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x27)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 136 [sleep]:
time.Sleep(0x9e8f67a2e)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc4dddc8d80, 0xc4f950d440, 0x1, 0x1, 0xc49293d1a0, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc4dddc8d80, 0xc4f950d440, 0x1, 0x1, 0x15ca6a7a690, 0x15ca6aa2b17)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc4dddc8d80, 0xc500c7633efd1b9e, 0xc49293d1a0, 0xc558c135b0, 0x1, 0x1, 0xc5d0175f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc4c156a6b0, 0x4, 0xc500c7633efd1b9e, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x28)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 137 [sleep]:
time.Sleep(0xba43b7400)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc48e7b60c0, 0xc46554e300, 0x1, 0x1, 0xc50a41b800, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc48e7b60c0, 0xc46554e300, 0x1, 0x1, 0x15ca68fd196, 0x15ca690815e)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc48e7b60c0, 0xde64ffbd40fc9109, 0xc50a41b800, 0xc598c704b0, 0x1, 0x1, 0xc588b63f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc43480f5b0, 0x4, 0xde64ffbd40fc9109, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x29)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 138 [sleep]:
time.Sleep(0xba43b7400)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc497a48660, 0xc471302840, 0x1, 0x1, 0xc4bc6e46c0, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc497a48660, 0xc471302840, 0x1, 0x1, 0x15ca6a39708, 0x15ca6a40c38)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc497a48660, 0x8e00ee1bdc71e80, 0xc4bc6e46c0, 0xc4564823d0, 0x1, 0x1, 0xc5d6087f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc42374f5d8, 0x4, 0x8e00ee1bdc71e80, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x2a)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 139 [sleep]:
time.Sleep(0x63684885a)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc4ea3f3350, 0xc528544120, 0x1, 0x1, 0xc42aafd4d0, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc4ea3f3350, 0xc528544120, 0x1, 0x1, 0x15ca6ba4b6b, 0x15ca6ba4b6b)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc4ea3f3350, 0x69be85cd34a8b18f, 0xc42aafd4d0, 0xc481798d80, 0x1, 0x1, 0xc5bbd17f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc42374f5d8, 0x4, 0x69be85cd34a8b18f, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x2b)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 140 [sleep]:
time.Sleep(0x732596cb)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc4db3a3ef0, 0xc47ad7f680, 0x1, 0x1, 0xc42338cf90, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc4db3a3ef0, 0xc47ad7f680, 0x1, 0x1, 0x15ca672f017, 0x15ca6736546)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc4db3a3ef0, 0x65687ad45f18c784, 0xc42338cf90, 0xc4238c4078, 0x1, 0x1, 0xc588b63f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc42374f5d8, 0x4, 0x65687ad45f18c784, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x2c)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 141 [sleep]:
time.Sleep(0x3ee891518)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc4b3c63a70, 0xc5470dad80, 0x1, 0x1, 0xc56a3d82a0, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc4b3c63a70, 0xc5470dad80, 0x1, 0x1, 0x15ca5966cc8, 0x15ca596e1f8)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc4b3c63a70, 0x23c53fb9c7f88dc5, 0xc56a3d82a0, 0xc474df3e80, 0x1, 0x1, 0xc5dab19f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc42374f5d8, 0x4, 0x23c53fb9c7f88dc5, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x2d)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 142 [sleep]:
time.Sleep(0xba43b7400)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc57f110ed0, 0xc4408434a0, 0x2, 0x2, 0xc4fd2889f0, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc57f110ed0, 0xc4408434a0, 0x2, 0x2, 0x15ca68ff2e4, 0x15ca6906814)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc57f110ed0, 0x191cc7f9d9bffe80, 0xc4fd2889f0, 0xc46e582890, 0x2, 0x2, 0xc59f957f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc42d1ca828, 0x4, 0x191cc7f9d9bffe80, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x2e)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 143 [sleep]:
time.Sleep(0xba43b7400)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc4efca71a0, 0xc46578eae0, 0x1, 0x1, 0xc52eefecf0, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc4efca71a0, 0xc46578eae0, 0x1, 0x1, 0x15ca6880c76, 0x15ca6880c76)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc4efca71a0, 0x78e5cd3a14c5fde1, 0xc52eefecf0, 0xc539d2d420, 0x1, 0x1, 0xc5c84dbf28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc42374f5d8, 0x4, 0x78e5cd3a14c5fde1, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x2f)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 144 [sleep]:
time.Sleep(0xba43b7400)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc51c90be60, 0xc527392c00, 0x1, 0x1, 0xc55f9527e0, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc51c90be60, 0xc527392c00, 0x1, 0x1, 0x15ca68fc962, 0x15ca690792a)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc51c90be60, 0x61b2e35d6b7a14ea, 0xc55f9527e0, 0xc55a6ee160, 0x1, 0x1, 0xc5dc16df28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc4686d0000, 0x1, 0x61b2e35d6b7a14ea, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x30)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 145 [sleep]:
time.Sleep(0x67889fc52)
	/usr/local/go/src/runtime/time.go:59 +0xf9
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.BatchWrite(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:202 +0x49b
github.com/weaveworks/cortex/pkg/chunk.awsStorageClient.PutChunks(0xc420116d00, 0x4000000000000000, 0x15c5bcd4c00, 0x1, 0x7fff6c44c5da, 0x17, 0x2260ff9290000, 0xc420116c80, 0x140d780, 0xc420198180, ...)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/aws_storage_client.go:568 +0xbf2
github.com/weaveworks/cortex/pkg/chunk.(*awsStorageClient).PutChunks(0xc420117b00, 0x7f57a61cff68, 0xc441b8f740, 0xc45189ed80, 0x1, 0x1, 0xc421005da0, 0x0)
	<autogenerated>:14 +0xce
github.com/weaveworks/cortex/pkg/chunk.(*Store).Put(0xc42014eea0, 0x7f57a61cff68, 0xc441b8f740, 0xc45189ed80, 0x1, 0x1, 0x15ca6c50a48, 0x15ca72e5388)
	/go/src/github.com/weaveworks/cortex/pkg/chunk/chunk_store.go:100 +0xbe
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushChunks(0xc4201b3180, 0x7f57a61cff68, 0xc441b8f740, 0xc007f432ecd2ca67, 0xc421005da0, 0xc5bab4a840, 0x1, 0x2, 0xc5c03e9f28, 0x13fd820)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:196 +0x2fd
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushUserSeries(0xc4201b3180, 0xc42001ae64, 0x4, 0xc007f432ecd2ca67, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:169 +0x21f
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).flushLoop(0xc4201b3180, 0x31)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_flush.go:119 +0xe7
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:263 +0xc89

goroutine 146 [select]:
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).loop(0xc4201b3180)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_lifecycle.go:199 +0x81d
created by github.com/weaveworks/cortex/pkg/ingester.New
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester.go:267 +0xd00

goroutine 148 [IO wait]:
net.runtime_pollWait(0x7f57a6209f70, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4201bc7d8, 0x72, 0x0, 0xc4d7d552c0)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4201bc7d8, 0xffffffffffffffff, 0x0)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc4201bc770, 0x0, 0x13fa720, 0xc4d7d552c0)
	/usr/local/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc4201980c0, 0xc420552ed8, 0x66ddbe, 0x455b40)
	/usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc4201980c0, 0xd9f8c0, 0xc5c4652b40, 0x14034e0, 0xc420430090)
	/usr/local/go/src/net/tcpsock.go:228 +0x49
net/http.(*Server).Serve(0xc420130a50, 0x1402360, 0xc4201980c0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2643 +0x228
created by github.com/weaveworks/cortex/vendor/github.com/weaveworks/common/server.(*Server).Run
	/go/src/github.com/weaveworks/cortex/vendor/github.com/weaveworks/common/server/server.go:148 +0x5f

goroutine 150 [select, 21727 minutes, locked to thread]:
runtime.gopark(0xd9fea0, 0x0, 0xd6f001, 0x6, 0x18, 0x2)
	/usr/local/go/src/runtime/proc.go:271 +0x13a
runtime.selectgoImpl(0xc420362f50, 0x0, 0x18)
	/usr/local/go/src/runtime/select.go:423 +0x1364
runtime.selectgo(0xc420362f50)
	/usr/local/go/src/runtime/select.go:238 +0x1c
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:434 +0x2dd
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2197 +0x1

goroutine 1600111822 [select]:
net/http.(*persistConn).writeLoop(0xc562da3200)
	/usr/local/go/src/net/http/transport.go:1704 +0x43a
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1118 +0xa5a

goroutine 1600112002 [select]:
net/http.(*persistConn).writeLoop(0xc53fe565a0)
	/usr/local/go/src/net/http/transport.go:1704 +0x43a
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1118 +0xa5a

goroutine 1600111821 [IO wait]:
net.runtime_pollWait(0x7f575eff2b00, 0x72, 0x8)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc49fea6308, 0x72, 0x13fc960, 0x13f6ee8)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc49fea6308, 0xc591039000, 0x1000)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc49fea62a0, 0xc591039000, 0x1000, 0x1000, 0x0, 0x13fc960, 0x13f6ee8)
	/usr/local/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc448c84238, 0xc591039000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:181 +0x70
crypto/tls.(*block).readFromUntil(0xc473a96990, 0x7f57a61cff18, 0xc448c84238, 0x5, 0xc448c84238, 0x28)
	/usr/local/go/src/crypto/tls/conn.go:488 +0x98
crypto/tls.(*Conn).readRecord(0xc43d74f880, 0xd9ff17, 0xc43d74f9a0, 0x4553e0)
	/usr/local/go/src/crypto/tls/conn.go:590 +0xc4
crypto/tls.(*Conn).Read(0xc43d74f880, 0xc539b62000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1134 +0x11d
net/http.(*persistConn).Read(0xc562da3200, 0xc539b62000, 0x1000, 0x1000, 0x5, 0x4534a0, 0xc4db0c4300)
	/usr/local/go/src/net/http/transport.go:1316 +0x14b
bufio.(*Reader).fill(0xc489405020)
	/usr/local/go/src/bufio/bufio.go:97 +0x117
bufio.(*Reader).Peek(0xc489405020, 0x1, 0xc5a5636be5, 0x1, 0x0, 0xc4f19fb080, 0x0)
	/usr/local/go/src/bufio/bufio.go:129 +0x67
net/http.(*persistConn).readLoop(0xc562da3200)
	/usr/local/go/src/net/http/transport.go:1474 +0x196
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1117 +0xa35

goroutine 1599933536 [semacquire, 20 minutes]:
sync.runtime_Semacquire(0xc4201b3374)
	/usr/local/go/src/runtime/sema.go:47 +0x34
sync.(*WaitGroup).Wait(0xc4201b3368)
	/usr/local/go/src/sync/waitgroup.go:131 +0x7a
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).processShutdown(0xc4201b3180)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_lifecycle.go:338 +0xa8
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).loop.func2(0xc4201b3180, 0xc50fb877a0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_lifecycle.go:193 +0x2b
created by github.com/weaveworks/cortex/pkg/ingester.(*Ingester).loop
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_lifecycle.go:195 +0x5d5

goroutine 1600111937 [IO wait]:
net.runtime_pollWait(0x7f576171daa0, 0x72, 0x6)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4c7a69728, 0x72, 0x13fc960, 0x13f6ee8)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4c7a69728, 0xc51d6ee800, 0x400)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc4c7a696c0, 0xc51d6ee800, 0x400, 0x400, 0x0, 0x13fc960, 0x13f6ee8)
	/usr/local/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc426a195a8, 0xc51d6ee800, 0x400, 0x400, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:181 +0x70
crypto/tls.(*block).readFromUntil(0xc50081eed0, 0x7f57a61cff18, 0xc426a195a8, 0x5, 0xc426a195a8, 0x28)
	/usr/local/go/src/crypto/tls/conn.go:488 +0x98
crypto/tls.(*Conn).readRecord(0xc51669c700, 0xd9ff17, 0xc51669c820, 0x4553e0)
	/usr/local/go/src/crypto/tls/conn.go:590 +0xc4
crypto/tls.(*Conn).Read(0xc51669c700, 0xc5c3be3000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1134 +0x11d
net/http.(*persistConn).Read(0xc53fe565a0, 0xc5c3be3000, 0x1000, 0x1000, 0x5, 0x4534a0, 0xc4fe48dc80)
	/usr/local/go/src/net/http/transport.go:1316 +0x14b
bufio.(*Reader).fill(0xc4db0c55c0)
	/usr/local/go/src/bufio/bufio.go:97 +0x117
bufio.(*Reader).Peek(0xc4db0c55c0, 0x1, 0xc5a5638be5, 0x1, 0x0, 0xc4d5b6a660, 0x0)
	/usr/local/go/src/bufio/bufio.go:129 +0x67
net/http.(*persistConn).readLoop(0xc53fe565a0)
	/usr/local/go/src/net/http/transport.go:1474 +0x196
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1117 +0xa35

goroutine 1600111964 [IO wait]:
net.runtime_pollWait(0x7f575f3a8f30, 0x72, 0x5)
	/usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4fbec4998, 0x72, 0x13fc960, 0x13f6ee8)
	/usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4fbec4998, 0xc58d0e1000, 0x1000)
	/usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc4fbec4930, 0xc58d0e1000, 0x1000, 0x1000, 0x0, 0x13fc960, 0x13f6ee8)
	/usr/local/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc44bb80548, 0xc58d0e1000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:181 +0x70
net/http.(*persistConn).Read(0xc5514a9680, 0xc58d0e1000, 0x1000, 0x1000, 0x4, 0x4534a0, 0xc484211ce0)
	/usr/local/go/src/net/http/transport.go:1316 +0x14b
bufio.(*Reader).fill(0xc499620f60)
	/usr/local/go/src/bufio/bufio.go:97 +0x117
bufio.(*Reader).Peek(0xc499620f60, 0x1, 0xc5501eebe5, 0x1, 0x0, 0xc499620fc0, 0x0)
	/usr/local/go/src/bufio/bufio.go:129 +0x67
net/http.(*persistConn).readLoop(0xc5514a9680)
	/usr/local/go/src/net/http/transport.go:1474 +0x196
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1117 +0xa35

goroutine 1600111965 [select]:
net/http.(*persistConn).writeLoop(0xc5514a9680)
	/usr/local/go/src/net/http/transport.go:1704 +0x43a
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1118 +0xa5a

rax    0xfffffffffffffffc
rbx    0x0
rcx    0x45a953
rdx    0x0
rdi    0x1435df8
rsi    0x0
rbp    0x7f57a7a46c58
rsp    0x7f57a7a46c10
r8     0x0
r9     0x0
r10    0x7f57a7a46c48
r11    0x246
r12    0x0
r13    0x7fff6c44a89f
r14    0x7f57a7a47700
r15    0x0
rip    0x45a953
rflags 0x246
cs     0x33
fs     0x0
gs     0x0

awh · 2017-06-14T15:57:01Z

goroutine 1599933536 [semacquire, 20 minutes]:
sync.runtime_Semacquire(0xc4201b3374)
	/usr/local/go/src/runtime/sema.go:47 +0x34
sync.(*WaitGroup).Wait(0xc4201b3368)
	/usr/local/go/src/sync/waitgroup.go:131 +0x7a
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).processShutdown(0xc4201b3180)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_lifecycle.go:338 +0xa8
github.com/weaveworks/cortex/pkg/ingester.(*Ingester).loop.func2(0xc4201b3180, 0xc50fb877a0)
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_lifecycle.go:193 +0x2b
created by github.com/weaveworks/cortex/pkg/ingester.(*Ingester).loop
	/go/src/github.com/weaveworks/cortex/pkg/ingester/ingester_lifecycle.go:195 +0x5d5

Corresponds to https://github.com/weaveworks/cortex/blob/1ba9d6d82c0c451835f6b9f8fdeecd12ebda5646/pkg/ingester/ingester_lifecycle.go#L338 - the terminating ingester is blocked waiting for all the flush loops to close.

awh · 2017-06-15T12:55:11Z

Argh - I didn't realise that 105841e wasn't deployed in production.

awh added the type/bug label Jun 14, 2017

awh closed this as completed Jun 15, 2017

awh mentioned this issue Jun 20, 2017

Mysterious flush of underutilised chunks 1hr after ingester rollout #467

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handoff to pending ingester did not complete cleanly #458

Handoff to pending ingester did not complete cleanly #458

awh commented Jun 14, 2017

awh commented Jun 14, 2017

awh commented Jun 14, 2017 •

edited

Loading

awh commented Jun 14, 2017 •

edited

Loading

awh commented Jun 14, 2017

awh commented Jun 14, 2017

awh commented Jun 15, 2017

Handoff to pending ingester did not complete cleanly #458

Handoff to pending ingester did not complete cleanly #458

Comments

awh commented Jun 14, 2017

awh commented Jun 14, 2017

awh commented Jun 14, 2017 • edited Loading

awh commented Jun 14, 2017 • edited Loading

awh commented Jun 14, 2017

awh commented Jun 14, 2017

awh commented Jun 15, 2017

awh commented Jun 14, 2017 •

edited

Loading

awh commented Jun 14, 2017 •

edited

Loading