feat: add insights instrumentation - events and metrics

[roelbondoc]

This PR adds automatic Insights data collection to the gem for certain libraries.

By default, all automatic data collection is disabled. To enable collection, add the following to your config/honeybadger.yml

insights:
  enabled: true

The list below are the names of the plugins that contain insight metrics. Some plugins will emit events that contain a duration attribute indicating how long that particular event took to invoke. Other plugins will poll for metrics. The polling occurs every second. As with regular plugins, they only load if the required gem is available, and they may be disabled through the skipped_plugins configuration setting.

rails

events

process_action.action_controller
send_file.action_controller
redirect_to.action_controller
halted_callback.action_controller
write_fragment.action_controller
read_fragment.action_controller
expire_fragment.action_controller
render_template.action_view
render_partial.action_view
render_collection.action_view
sql.active_record
process.action_mailer
service_upload.active_storage
service_download.active_storage

active_job

events

enqueue_at.active_job
enqueue.active_job
enqueue_retry.active_job
enqueue_all.active_job
perform.active_job
retry_stopped.active_job
discard.active_job

sidekiq

events

perform.sidekiq
enqueue.sidekiq

metrics

active_workers
active_processes
jobs_processed
jobs_failed
jobs_scheduled
jobs_enqueued
jobs_dead
jobs_retry
queue_latency
queue_depth
queue_busy
capacity
utilization

karafka

events

consumer.consumed.karafka

solid_queue (new)

metrics

jobs_in_progress
jobs_blocked
jobs_failed
jobs_scheduled
jobs_processed
active_workers
active_dispatchers
queue_depth

net_http (new, disabled by default)

events

request.net_http

To enable automatic instrumentation of Net::HTTP requests, add the following to the config:

net_http:
  insights:
    enabled: true

autotuner (new)

events

report.autotuner

metrics

diff.minor_gc_count
diff.major_gc_count
diff.time
request_time
heap_pages

puma (new)

metrics

pool_capacity
max_threads
requests_count
backlog
running

system (new)

events

report.system

Note: The puma metrics are enabled through the puma plugin. This isn't enabled by default, but must be added to your puma.rb file by adding plugin: honeybadger anywhere in the file.

Further Configuration

You may also disable insights for a specific plugin by adding a setting with the plugin name:

sidekiq:
  insights:
    enabled: false

Some plugins also collect metrics for a cluster. These metrics do not need to be collected from every running instance. To disable the collection of cluster metrics, add the following configuration:

sidekiq:
  insights:
    cluster_collection: false

This is available for the sidekiq and solid_queue plugins. It is recommended to enable this for a single instance. If you are running Sidekiq Enterprise and have this enabled, it will only run on the current leader instance.

By default, polled metrics collection occurs every one second, this can be configured per plugin:

sidekiq:
  insights:
    collection_interval: 10

This will modify the collector so that sidekiq metrics are checked every 10 seconds.

Ignoring events:

events:
  ignore:
    - 'enqueue.sidekiq'
    - !ruby/regexp '/.*.active_storage/'

Metrics and Registry

Components

Honeybadger::Registry - This class stores all tracked metrics registered in the gem. The registry ensures that only one instance of each unique metric is stored at any time. A metric is defined as unique by it's type, name, and set of attributes.

Honeybadger::RegistryExecution - This class is injected into the MetricsWorker and is run on a configurable interval. When called, it iterates over all registered metrics, compiles payloads for each, and sent as an event. The registry is then flushed.

Honeybadger::Metric - A base metric class from which all other metric classes inherit from. These metric classes record data and compute/track values as needed.

Shape of the Data

Each metric type's data is shaped differently. Below are examples of what that data looks like for each type.

counter

{
  "counter": 1
}

gauge

{
  "min": 1,
  "max": 10,
  "avg": 5.0,
  "latest": 5
}

timer

{
  "min": 1,
  "max": 10,
  "avg": 5.0,
  "latest": 5
}

Note: This may look the same as a gauge, however, the interface to use the metric provides a timing mechanism.

Example:

time 'some_operation', -> { code_to_be_timed() }

histogram

{
  "bins": [[10.0, 1], [50.0, 10], [100.0, 5]]
}

Changes to events

The event payloads will now include the hostname by default. This can be changed by setting the events.attach_hostname configuration parameter to false.

The event work now has its own config events.max_queue_size which defaults to 10_000. This will allow the agent to backlog more events. The batch size (events.batch_size) default has also been increased to 1_000.

Before submitting a pull request, please make sure the following is done:

1. If you've fixed a bug or added code that should be tested, add tests!
2. Run rake spec in the repository root.
3. Use a pull request title that conforms to conventional commits.

[roelbondoc]

There has been some changes since the last approvals. Requesting again for a final review.

[rabidpraxis]

This looks awesome @roelbondoc, great work!

I do have one quick question about the agent methods. If I were to instantiate a second agent and then call instrumentation methods on it, would the events report from that agent, or globally?

[rabidpraxis]

Is there a concern with sensitive data in the URI here?

[roelbondoc]

That's a good question. There could be sensitive data here that a user may not know is being logged. Perhaps we need a mechanism to disable this by default.

[roelbondoc]

I added a default so this is disabled by default and must be explicitly enabled:

net_http:
  insights:
    enabled: true

[stympy]

How about leaving it enabled but reporting just the host name?

[roelbondoc]

That works for me. I'll make it configurable to allow the full uri if set to true.

[roelbondoc]

I do have one quick question about the agent methods. If I were to instantiate a second agent and then call instrumentation methods on it, would the events report from that agent, or globally?

That's a good question. Right now some of the internals rely on the global singleton agent, but with a few tweaks this shouldn't be too hard to fix.

[joshuap]

Great work @roelbondoc. Can't wait to start using this.

[joshuap]

Was this supposed to be false by default?

[roelbondoc]

There was a conversation about disabling it by default, but ultimately we decided to enable it by default, but only log the domain, with an optional configuration to log the full path.