convert airbyte-metrics-reporter to micronaut #17365

colesnodgrass · 2022-09-28T23:20:48Z

What

Update airbyte-metrics-reporter to be Micronaut based.

How

Update staticReporterApp.configDatabase and MietricClientFactory.getMetricClient() calls with constructor injected beans MetricRepository and MetricClient dependencies which are defined in the ReporterFactory class.
Move all queries specific to the metric-reporter from metrics-lib/MetricQueries to the MetricRepository class in the reporter module.
- Additionally the visibility of these methods was changed from public to package-private as there is no reason for these to be available outside of this use-case.
Move the metric emitter initialization from the main function to a Micronaut annotated @EventListener in the EventListeners class.
- This was previously handled by the ReporterApp.
- This handles both starting and cleaning stopping the emitters by listening for ApplicationStartupEvent and ApplicationShutdownEvent events.
Remove all flyway references.
- They seemed out of place in a service like this.
Replace ToEmit enum with a sealed Emitter class.
- Each implementation of the Emitter class is responsible for a single metric.
- Each implementation of the Emitter class has its dependencies injected via it's constructor.
- Each implementation of the Emitter class is annotated with @Singleton and supports being injected.
Add tests for the classes that implement the Emitter class.

Note

I will be updating some of the queries used to gather these metrics in a follow-up PR as they are inefficient. This was not done as part of this PR as it's worth reviewing those changes independent of the Micronaut changes.

…reporter

jdpgrailsdev · 2022-09-29T14:54:52Z

airbyte-metrics/reporter/src/main/resources/application.yml

+      jackson-converter-enabled: true
+      sql-dialect: POSTGRES
+
+docker:


Is this needed for this application?

jdpgrailsdev · 2022-09-29T14:55:29Z

airbyte-metrics/reporter/src/main/resources/application.yml

+        access:
+          - isAnonymous()
+  server:
+    port: 9000


9000 is the port that airbyte-workers used before the conversion. I'm not sure if this will be an issue when running via Docker Compose, as that port is already mapped to something.

@jdpgrailsdev looks like we're using the same port in a few places:

cron

workers

It was originally added to airbyte-workers, as it was already using port 9000. I believe it is copy/pasta in airbyte-cron. This is probably fine because the containers/pods will handle mapping to it and we can cycle back later if we want to have the services on different ports for identification, etc.

…reporter

jdpgrailsdev · 2022-10-04T18:25:42Z

airbyte-metrics/reporter/build.gradle

+    testAnnotationProcessor libs.bundles.micronaut.test.annotation.processor
+
+//    integrationTestJavaAnnotationProcessor platform(libs.micronaut.bom)
+//    integrationTestJavaAnnotationProcessor libs.bundles.micronaut.test.annotation.processor


Nit: remove the commented out dependencies if not needed

jdpgrailsdev

with a few minor nits/comments.

jdpgrailsdev · 2022-10-04T18:28:42Z

@colesnodgrass Bonus points if you want to also include a startup banner: https://github.com/airbytehq/airbyte/blob/master/airbyte-workers/src/main/resources/micronaut-banner.txt

…reporter

benmoriceau · 2022-10-03T18:47:51Z

airbyte-metrics/reporter/src/main/java/io/airbyte/metrics/reporter/Emitter.java

+   * @return Duration of how often this metric should report.
+   */
+  public Duration getDuration() {
+    return Duration.ofSeconds(15);


Why do you we need that? Isn't the Datadog agent supposed to do that for us? Won't this be an issue if we have metrics that are different than a gauge?

As these are stats which are emitted to the datadog statsd agent, there is no way (that I'm currently aware of) to tell this agent to fetch these values on a period basis. They are not polled by the agent, but rather are pushed to the agent. This is why these are registered to run in the EventListeners class at a fixed rate. The rate of which is determined by what duration is returned from this getDuration method.

Not every value is currently running every 15 seconds, some run every hour.

As for the non-gauge values, all of these emitters should match the existing behavior from the now defunct ToEmit enum. If they were working before they should still be working.

benmoriceau · 2022-10-10T20:21:09Z

airbyte-metrics/reporter/src/main/resources/micronaut-banner.txt

@@ -1 +1,8 @@
-: airbyte-metrics-reporter :
+


No strong opinon here but seeing the Airbyte banner, was the sign for me that my local deployment can be use. Having many of them will be confusing. If we decide on keeping them we should add it to the cron app as well.

benmoriceau · 2022-10-10T20:23:24Z

airbyte-metrics/reporter/src/main/java/io/airbyte/metrics/reporter/MetricRepository.java

+  }
+
+  List<Long> numberOfActiveConnPerWorkspace() {
+    final var query = """


Is there any reason to use not use JOOQ here?

This is copied verbatim from the existing implementation. I didn't want to move and change them within the same PR as I find that makes it's difficult to actually track the query changes. I plan to do a follow-up PR that cleans these queries up a bit.

benmoriceau · 2022-10-10T20:25:46Z

airbyte-metrics/reporter/src/main/java/io/airbyte/metrics/reporter/MetricRepository.java

+    return ctx.fetch(query).getValues("num_conn", long.class);
+  }
+
+  long numScheduledActiveConnectionsInLastDay() {


Aren't we going to miss some connection here? expecially the cron and the manual ones that have been triggered.

Possibly? This was also copied verbatim from the existing implementation. See my comments above about a follow-up PR.

benmoriceau · 2022-10-10T20:27:15Z

airbyte-metrics/reporter/src/main/java/io/airbyte/metrics/reporter/MetricRepository.java

+                                                         and j.created_at > now() - interval '24 hours 1 minutes'
+                                                         and c.status = 'active'
+                                                         and j.config_type = 'sync'
+                                                         and c.updated_at < now() - interval '24 hours 1 minutes'


Why do we need the filter on the updated_at of the connection table?

This too was copied verbatim from the existing implementation. See my comments above about a follow-up PR.

benmoriceau

Make sense about the files that got moved.

This reverts commit d30de1b.

* Revert "improve query performance (#17862)" This reverts commit e0db09b. * Revert "fix metric reporter not producing metrics (#17863)" This reverts commit 63e39e6. * Revert "convert airbyte-metrics-reporter to micronaut (#17365)" This reverts commit d30de1b.

* sealed class experiment * micronaut the metric reporter service * code format * move event-listeners to EventListeners class * code format * db cleanup; start adding tests * finish porting over MetricsQueriesTest pieces that apply * dedupe/simplify test code * add EmitterTest; misc code clean-up * address pmd warnings * remove code that was moved to the reporter package * update application.yml * Emitter[] -> List<Emitter> * remove unused singleton * formatting * formatting * remove default test creds * update banner * remove commented out code * remove another commented out line

* Revert "improve query performance (airbytehq#17862)" This reverts commit e0db09b. * Revert "fix metric reporter not producing metrics (airbytehq#17863)" This reverts commit 63e39e6. * Revert "convert airbyte-metrics-reporter to micronaut (airbytehq#17365)" This reverts commit d30de1b.

colesnodgrass added 3 commits September 28, 2022 09:24

sealed class experiment

0f4779e

Merge remote-tracking branch 'origin/master' into cole/mirco-metrics-…

10fce4c

…reporter

micronaut the metric reporter service

e23d2b3

jdpgrailsdev reviewed Sep 29, 2022

View reviewed changes

colesnodgrass added 2 commits September 29, 2022 09:22

code format

18342dc

Merge remote-tracking branch 'origin/master' into cole/mirco-metrics-…

8f7543e

…reporter

colesnodgrass temporarily deployed to more-secrets September 29, 2022 16:26 Inactive

move event-listeners to EventListeners class

f6ebcbd

colesnodgrass temporarily deployed to more-secrets September 29, 2022 18:05 Inactive

colesnodgrass added 2 commits September 29, 2022 12:36

Merge remote-tracking branch 'origin/master' into cole/mirco-metrics-…

f933d5c

…reporter

code format

7abf5b7

colesnodgrass temporarily deployed to more-secrets September 29, 2022 20:10 Inactive

db cleanup; start adding tests

3559044

colesnodgrass temporarily deployed to more-secrets September 29, 2022 23:43 Inactive

finish porting over MetricsQueriesTest pieces that apply

5a1b348

colesnodgrass temporarily deployed to more-secrets September 30, 2022 03:25 Inactive

colesnodgrass added 3 commits September 30, 2022 08:44

dedupe/simplify test code

eeb557f

add EmitterTest; misc code clean-up

a28ff79

Merge remote-tracking branch 'origin/master' into cole/mirco-metrics-…

5e9593e

…reporter

colesnodgrass temporarily deployed to more-secrets September 30, 2022 16:36 Inactive

colesnodgrass added 2 commits September 30, 2022 10:16

address pmd warnings

232c40f

remove code that was moved to the reporter package

1dc2459

colesnodgrass temporarily deployed to more-secrets September 30, 2022 17:25 Inactive

Merge remote-tracking branch 'origin/master' into cole/mirco-metrics-…

d865d50

…reporter

colesnodgrass temporarily deployed to more-secrets September 30, 2022 18:59 Inactive

update application.yml

9b6b05d

colesnodgrass temporarily deployed to more-secrets September 30, 2022 19:15 Inactive

colesnodgrass requested review from alovew and benmoriceau September 30, 2022 19:42

colesnodgrass temporarily deployed to more-secrets September 30, 2022 22:03 Inactive

remove default test creds

d30afc5

colesnodgrass temporarily deployed to more-secrets September 30, 2022 23:15 Inactive

jdpgrailsdev reviewed Oct 4, 2022

View reviewed changes

jdpgrailsdev approved these changes Oct 4, 2022

View reviewed changes

colesnodgrass added 3 commits October 10, 2022 10:31

Merge remote-tracking branch 'origin/master' into cole/mirco-metrics-…

addaaf3

…reporter

update banner

d9f3710

remove commented out code

477451e

colesnodgrass temporarily deployed to more-secrets October 10, 2022 20:07 Inactive

benmoriceau reviewed Oct 10, 2022

View reviewed changes

remove another commented out line

779bbe1

colesnodgrass temporarily deployed to more-secrets October 10, 2022 20:58 Inactive

benmoriceau approved these changes Oct 10, 2022

View reviewed changes

colesnodgrass merged commit d30de1b into master Oct 11, 2022

colesnodgrass deleted the cole/mirco-metrics-reporter branch October 11, 2022 16:13

This was referenced Oct 11, 2022

Convert airbyte-metrics-reporter to a Micronaut service #13272

Closed

Annotate Dependency Services Utilized By airbyte-metrics-reporter #12394

Closed

malikdiarra added a commit that referenced this pull request Oct 13, 2022

Revert "convert airbyte-metrics-reporter to micronaut (#17365)"

7296ff5

This reverts commit d30de1b.

This was referenced Oct 13, 2022

Bump Airbyte version from 0.40.14 to 0.40.15 #17961

Closed

Bump Airbyte version from 0.40.14 to 0.40.15 #17970

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

convert airbyte-metrics-reporter to micronaut #17365

convert airbyte-metrics-reporter to micronaut #17365

colesnodgrass commented Sep 28, 2022 •

edited

Loading

jdpgrailsdev Sep 29, 2022

jdpgrailsdev Sep 29, 2022

colesnodgrass Sep 30, 2022

jdpgrailsdev Oct 4, 2022

jdpgrailsdev Oct 4, 2022

jdpgrailsdev left a comment

jdpgrailsdev commented Oct 4, 2022

benmoriceau Oct 3, 2022

colesnodgrass Oct 10, 2022

benmoriceau Oct 10, 2022

benmoriceau Oct 10, 2022

colesnodgrass Oct 10, 2022

benmoriceau Oct 10, 2022

colesnodgrass Oct 10, 2022

benmoriceau Oct 10, 2022

colesnodgrass Oct 10, 2022

benmoriceau left a comment

convert airbyte-metrics-reporter to micronaut #17365

convert airbyte-metrics-reporter to micronaut #17365

Conversation

colesnodgrass commented Sep 28, 2022 • edited Loading

What

How

Recommended reading order

Note

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jdpgrailsdev left a comment

Choose a reason for hiding this comment

jdpgrailsdev commented Oct 4, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benmoriceau left a comment

Choose a reason for hiding this comment

colesnodgrass commented Sep 28, 2022 •

edited

Loading