Move to DogStatsD. #10238

davinchia · 2022-02-10T10:48:25Z

What

We want to move to DogStatsD to make it easier to do more complex metrics collection. The official DD library also does not double count metrics so using this is cheaper overall.

Unfortunately unlike Prometheus, there is no good way to test that metrics are being reported in an automated test so I've omitted that for now. I've tested this manually in the dev env to make sure metrics are being emitted.

How

Rename the old metrics singleton to make it clear it's Prometheus related. Also add a deprecated annotation.
Add a DogStatsD metric singleton class.

I will remove the old Prometheus class in a follow up PR.

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

airbyte-metrics/src/main/java/io/airbyte/metrics/DogstatsdMetricSingleton.java

jrhizor · 2022-02-10T18:23:04Z

airbyte-metrics/src/main/java/io/airbyte/metrics/DogstatsdMetricSingleton.java

+    return instance;
+  }
+
+  public synchronized static void initialize(final String appName, final Map<String, String> mdc, final boolean publish) {


Why is the MDC important here? Shouldn't the initialization be run from some thread that has the MDC configured?

good point - I think it's left over from when I was testing. You should be right that we don't need this anymore. Will remove.

lmossman · 2022-02-10T18:26:13Z

airbyte-metrics/src/main/java/io/airbyte/metrics/DogstatsdMetricSingleton.java

+
+  public static synchronized DogstatsdMetricSingleton getInstance() {
+    if (instance == null) {
+      throw new RuntimeException("You must initialize configuration with the initializeMonitoringServiceDaemon() method before getting an instance.");


Should this say .. with the initialize() method?

Also, what is the downside of combining getInstance() and initialize() into the same method? The behavior would be that if instance == null, it initializes and returns. If instance != null, it just returns. That approach feels like it would be a bit more ergonomic, no?

ah yes good catch on the error message.

this is a small thing - the main downside would be the initialise method has more parameters than the getInstance method. we'd have to combine those parameters if we combined the methods. because of this I prefer to keep these separate for now.

@davinchia I'm curious how you are thinking about consumers of this class. If I want to use the DogstatsdMetricSingleton, how do I know if it has been initialized or not? Is it expected that a consumer would call getInstance wrapped with a try-catch to catch a RuntimeException in the case that it was already initialized, and call initialize in that case? That doesn't feel very ergonomic to me, but let me know if I'm thinking about this wrong

Good question - I'm trying to follow the general singleton pattern where initialize is called in the main method, and getInstance is called everywhere else. Although it's possible to catch, most folks don't really catch the RTE since it's usually understood the singleton is initialised when the JVM first starts up.

I agree with you having to do getInstance is somewhat annoying, but this is a fairly common pattern and I don't really think it's worth trying to get rid of.

I can't see how to not have a simply method signature if we combine initialize and getInstance and I didn't want to spend more time playing with this.

Happy to discuss more/review any ideas/approaches you come up with!

airbyte-scheduler/app/src/main/java/io/airbyte/scheduler/app/SchedulerApp.java

lmossman

Left a couple of fairly small comments but looks good to me overall

Test publishing a gauge.

961b6e1

github-actions bot added area/platform issues related to the platform area/scheduler labels Feb 10, 2022

davinchia temporarily deployed to more-secrets February 10, 2022 10:49 Inactive

tkorenko reviewed Feb 10, 2022

View reviewed changes

airbyte-metrics/src/main/java/io/airbyte/metrics/DogstatsdMetricSingleton.java Outdated Show resolved Hide resolved

tkorenko reviewed Feb 10, 2022

View reviewed changes

airbyte-metrics/src/main/java/io/airbyte/metrics/DogstatsdMetricSingleton.java Outdated Show resolved Hide resolved

davinchia added 2 commits February 10, 2022 19:29

Rename the current metrics singleton by adding a Prometheus prefix.

e186660

Remove prometheus base example class.

83c9cbd

davinchia temporarily deployed to more-secrets February 10, 2022 11:31 Inactive

Checkpoint: add counter type.

0eb580e

davinchia temporarily deployed to more-secrets February 10, 2022 11:41 Inactive

Checkpoint: initial basic implementation of DogStatsD Metrics Singleton.

d2bf1c7

davinchia temporarily deployed to more-secrets February 10, 2022 14:40 Inactive

Remove old one file.

c37f2e0

davinchia temporarily deployed to more-secrets February 10, 2022 14:42 Inactive

Format.

42bec12

davinchia temporarily deployed to more-secrets February 10, 2022 14:56 Inactive

davinchia marked this pull request as ready for review February 10, 2022 15:02

davinchia requested review from jrhizor and lmossman February 10, 2022 18:18

jrhizor approved these changes Feb 10, 2022

View reviewed changes

lmossman reviewed Feb 10, 2022

View reviewed changes

airbyte-scheduler/app/src/main/java/io/airbyte/scheduler/app/SchedulerApp.java Show resolved Hide resolved

lmossman approved these changes Feb 10, 2022

View reviewed changes

davinchia added 2 commits February 11, 2022 23:46

Remove uneeded MDC.

beea7ac

Format.

6501a08

davinchia temporarily deployed to more-secrets February 11, 2022 15:49 Inactive

davinchia merged commit 4b73bb0 into master Feb 11, 2022

davinchia deleted the davinchia/use-dogstats-d branch February 11, 2022 16:25

octavia-squidington-iii mentioned this pull request Feb 12, 2022

Bump Airbyte version from 0.35.27-alpha to 0.35.28-alpha #10292

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move to DogStatsD. #10238

Move to DogStatsD. #10238

davinchia commented Feb 10, 2022 •

edited

Loading

jrhizor Feb 10, 2022

davinchia Feb 11, 2022

lmossman Feb 10, 2022

davinchia Feb 11, 2022

lmossman Feb 11, 2022

davinchia Feb 13, 2022

lmossman left a comment

Move to DogStatsD. #10238

Move to DogStatsD. #10238

Conversation

davinchia commented Feb 10, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

jrhizor Feb 10, 2022

Choose a reason for hiding this comment

davinchia Feb 11, 2022

Choose a reason for hiding this comment

lmossman Feb 10, 2022

Choose a reason for hiding this comment

davinchia Feb 11, 2022

Choose a reason for hiding this comment

lmossman Feb 11, 2022

Choose a reason for hiding this comment

davinchia Feb 13, 2022

Choose a reason for hiding this comment

lmossman left a comment

Choose a reason for hiding this comment

davinchia commented Feb 10, 2022 •

edited

Loading