Output plugin to PostgreSQL (Copy, Batch insert)

[otherpirate]

Hi,

I created these plugins to handling with large number of metrics.
I'm using Isolated transaction and grouping insert by table/metric name

1. Using Copy
Example: https://medium.com/@amoghagarwal/insert-optimisations-in-golang-26884b183b35
Docs: https://www.postgresql.org/docs/9.1/static/sql-copy.html

2. Using Batch
All values are added in same insert line.

Extra:
I already tried #3428 and it is just too slow when we have a lot of metrics.
This PR also solve #3408.

Benchmark between PostgreSQL(svenklemm), InfluxDB, PostgreSQL Copy and PostgreSQL Batch

#First test
Output [postgresql] wrote batch of 10000 metrics in 12.24790546s
Output [influxdb] wrote batch of 10000 metrics in 133.712335ms
Output [postgresql_copy] wrote batch of 10000 metrics in 181.033963ms
Output [postgresql_batch] wrote batch of 10000 metrics in 325.482989ms

#Second test
Output [postgresql] wrote batch of 10000 metrics in 12.110281988s
Output [influxdb] wrote batch of 10000 metrics in 83.051051ms
Output [postgresql_copy] wrote batch of 10000 metrics in 207.410926ms
Output [postgresql_batch] wrote batch of 10000 metrics in 285.334801ms

#Third test
Output [postgresql] wrote batch of 10000 metrics in 11.474782971s
Output [influxdb] wrote batch of 10000 metrics in 77.17222ms
Output [postgresql_copy] wrote batch of 10000 metrics in 158.511947ms
Output [postgresql_batch] wrote batch of 10000 metrics in 286.208872ms	

Obs1: I'm using (TimeScaleDB)
Obs2: All benchmark are running in cloud (each database in equal and isolated machine)
Obs3: This plugin DO NOT create tables (I'm working on it)

Required for all PRs:

• Signed CLA.
• Associated README.md updated.
• Has appropriate unit tests.

[nerzhul]

#3428

[otherpirate]

Hi @nerzhul

I saw this PR, I tested the code, and it's too slow for large number of metrics, please, take a look in my tests.

[glinton]

@otherpirate do you mind updating this branch with master so tests pass (also, we've updated to use dep), thanks.

[endersonmaia]

it would be nice to have the jsonb implementation for tags and fields from #3428 with the COPY implementation from this PR

[otherpirate]

Hi @glinton, could you review this PR again?

[sjwang90]

Can you clarify the discrepancies between this PR and #3428? That PR is a little more active that we may want to go with that one.

[otherpirate]

Can you clarify the discrepancies between this PR and #3428? That PR is a little more active that we may want to go with that one.

In Globo company, we still using this plugin, but the first is more active.
When I write this plugin (2018) the #3428 had no Copy method, and we need the performance of the Copy.
I did not test the latest version of #3428 but probably is very good
My suggestion to you is, try both and find what's fit better in your case :)

[sjwang90]

Looks like PR #8651 (comment) covers copy.

Let's move forward with PR #8651 - feel free to test that PR and offer any feedback.