Long running tests can fail to record metrics

[lalexgap]

It looks like with a long running test the metrics can fail to record due to being too large (example):

Oct  7 05:27:57.426367	�[34mINFO�[0m	654.1094s �[30;47m   MESSAGE�[0m �[33m<< single[006] (d243c0) >>�[0m failed to send batch to InfluxDB; attempt 4; err: {"error":"Request Entity Too Large"}

Oct  7 05:27:57.426526	�[34mINFO�[0m	654.1098s �[30;47m   MESSAGE�[0m �[33m<< single[006] (d243c0) >>�[0m influxdb: failed to upload 116963 points; err: All attempts fail:

We are using results so testground will upload all the metrics at the end of the run. It seems that if we run the test for long enough the size of the metrics becomes too large for influx db to handle in one request.

[lalexgap]

We have a couple options here:

• Look at switching back to diagnositics (this would also let us monitor metrics as the test happens)
• Change the polling period for metrics (I think it's 1 second right now) so we generate less data
• Look at configuring the influx-db to accept larger requests