12423 remove all statsd from initializers

[bosawt]

Summary

• This PR removes unnecessary StatsD reset counters from all initializers. This should help anyone seeing metrics flipping back and forth between count and rate (making metrics appear to be performing unexpectedly)

Related issue(s)

• Remove all statsd from initializers #12423

Testing done

• I started a Rails server and performed various actions to confirm that things worked fine and that StatsD were still reporting as expected

What areas of the site does it impact?

Metric

Acceptance criteria

• Confirm that your own StatsD metrics still log as expected in `log/statsd.log

[github-actions]

1 Warning

⚠️

This PR changes 366 LoC (not counting whitespace/newlines). In order to ensure each PR receives the proper attention it deserves, we recommend not exceeding 200. Expect some delays getting reviews. File Summary Files app/mailers/application_mailer.rb (+2/-0) app/workers/evss/disability_compensation_form/metrics.rb (+2/-0) config/initializers/statsd.rb (+0/-178) lib/evss/reference_data/response_strategy.rb (+2/-0) modules/appeals_api/app/workers/appeals_api/add_icn_updater.rb (+1/-0) modules/appeals_api/app/workers/appeals_api/daily_error_report.rb (+1/-0) modules/appeals_api/app/workers/appeals_api/decision_review_report_daily.rb (+1/-0) modules/appeals_api/app/workers/appeals_api/decision_review_report_weekly.rb (+1/-0) modules/appeals_api/app/workers/appeals_api/monthly_stats_report.rb (+2/-0) modules/appeals_api/app/workers/appeals_api/weekly_error_report.rb (+1/-0) modules/check_in/config/initializers/statsd.rb (+2/-0) modules/claims_api/app/workers/claims_api/claim_establisher.rb (+1/-0) modules/claims_api/spec/requests/v1/power_of_attorney_request_spec.rb (+1/-0) modules/covid_vaccine/config/initializers/statsd.rb (+0/-25) modules/covid_vaccine/spec/request/covid_vaccine/v0/facilities_request_spec.rb (+1/-0) modules/mobile/config/initializers/statsd.rb (+1/-76) modules/vaos/config/initializers/statsd.rb (+0/-22) modules/vaos/config/initializers/statsd_instrument_monkeypatch.rb (+0/-45) spec/lib/sidekiq/form526_job_status_tracker/job_tracker_spec.rb (+1/-0) Note: We exclude files matching the following when considering PR size: *.csv, *.json, *.tsv, *.txt, Gemfile.lock, app/swagger, modules/mobile/docs, spec/fixtures/, spec/support/vcr_cassettes/, modules/mobile/spec/support/vcr_cassettes/, db/seeds, modules/vaos/app/docs, modules/meb_api/app/docs, modules/appeals_api/app/swagger/ Big PRs are difficult to review, often become stale, and cause delays.

Generated by 🚫 Danger

[kpethtel]

I don't know much about statsd, but it looks like you're saying we're misusing the increment method. This looks like it should include some work on each team's part to maintain the metrics you're removing. Do we need any followup actions?

[rmtolmach]

Looks good. Follow-up with @kpethtel required.

[rmtolmach]

oh wait... I swear all the checks were passing a minute ago. But now they're not. Can you look into those @bosawt?

[bosawt]

I don't know much about statsd, but it looks like you're saying we're misusing the increment method. This looks like it should include some work on each team's part to maintain the metrics you're removing. Do we need any followup actions?

I wouldn't go so far as to say we were 'misusing' it, just that there appears to have been a historic reason we were initializing these to 0, but as far as I can tell, that historic reason is no longer true.

Also, doing an increment reset in the initializers was at worst a no-op until the recent EKS switch, it was causing a ton of metric issues for us. And it's possible we could change how EKS reports metrics to fix it, but for now what was happening was:

• EKS instance would increment metric X whenever route on EKS was run, for whatever reason, Datadog would classify this metric as a rate type metric
• Any time BRD instance was deployed, it would 'reset' metric X with that 'increment 0' in the initializer, and for whatever reason, Datadog would classify the metric as count type metric
• The difference in rate versus count type metrics was taking a metric that was like 'X was incremented 100 times', and making it something really messed up like 'X was incremented 15.43 times'

[bosawt]

@rmtolmach Finally made it so specs are no longer breaking

[kpethtel]

Thanks for cleaning this up. I'm going to file a ticket for my team to review our use of statsd and evaluate whether we need to re-add any of the removed metrics (but without issues).