Metrics returning variable number of rows stopped working

[ondar]

I April I created metric that returns all the jobs that finished in our app and it was working fine. I was just missing the part where I somehow mark the rows that were already exported so I don't export them in every run of the metric. I have waited for pgwatch3 to become production-ready. When I saw beta1 stage, I started testing again. But it seems that now only 1st row returned from metric is saved.

Why was support for multirow metric dropped and is it possible to bring it back?

[ondar]

Maybe the problem is not in support of multiline metric, but in this error:

15:23:19.446 [INFO] [source:pfx] [metric:pfx_jobs] [rows:60] measurements fetched
15:23:19.449 [WARNING] Failed to create a metric fetching helper for demo in cpu_load: ERROR: permission denied for language plpython3u (SQLSTATE 42501)
15:23:19.459 [INFO] [source:demo] [metric:cpu_load] [rows:1] measurements fetched
15:23:19.575 [ERROR] [sink:postgresql://pgwatch@postgres:5432/pgwatch_metrics] [columnNames:[time dbname data tag_data]] [err:ERROR: no partition of relation "pfx_jobs_pfx" found for row (SQLSTATE 23514)] [pid:330] [tableName:[pfx_jobs]] [time:662.344µs] CopyFrom
15:23:19.575 [ERROR] [sink:postgres] [metric:pfx_jobs] [db:pfx] ERROR: no partition of relation "pfx_jobs_pfx" found for row (SQLSTATE 23514)
15:23:19.575 [WARNING] [sink:postgres] [db:pgwatch_metrics] Some metric partitions might have been removed, halting all metric storage. Trying to re-create all needed partitions on next run

What that could mean? Seems that some row causes this problem. Could it be that partition for particular time is missing? Seems that when I return only 10 latest instead of all 60 rows it works.

[ondar]

Ah, now I get it - pgwatch creates partition for 1st row but doesn't expect that 2nd row can be from different week.

See this image, 2 partitions were created. One for 1st row of returned data (when I tried to save then in ascending order and then for this week when I tried to save 10 rows from yesterday):

[ondar]

It is possible to check partition with each row returned from metric?

[pashagolub]

Well, that would be a disaster! pgwatch assumes timestamps are fetched right now! Remember we are gathering snapshots of the system metrics. We are not working with historic data. If your metric returns something old you're doing it wrong. epoch_ns must be the time metric gathered and by gathered I mean gatherd by pgwatch.

All and every metric started with

                select /* pgwatch_generated */
                  (extract(epoch from now()) * 1e9)::int8 as epoch_ns,

We don't expect someone will put there somewhat years old.

[ondar]

Its all jobs that completed. 1st run would return all historic data, then only new ones (after I implement marking rows that were already sent).

[pashagolub]

pgwatch doesn't care of internals of what you're trying to measure. It's up to you. But pgwatch needs to know when the measurement happened and this is epoch_ns. Changing it you blow the system, telling it that some measurement took place 3 years ago.

[pashagolub]

Would you please show the metric. At least the epoch_ns declaration

[ondar]

Its taken from table as time when job completed:

(EXTRACT(epoch FROM jst.processingend) * 1e9)::int8 AS epoch_ns,

[pashagolub]

No, it's wrong. epoch_ns is always server's time at the moment of measurement taken

[pashagolub]

You may add this value as a custom field though. And use it freely in dashboards.

[ondar]

Ah, actually why not 😁 I will try it that way.

[ondar]

Took me quite a while to get it working in pgwatch and grafana, but in the end it seems its all working. Thanks!