fix: Change default ConfigReloadInterval, add jitter, fix docs (#823)

## Which problem is this PR solving? - The default config reload interval was too long, and the documentation for it was wrong. ## Short description of the changes - Change default for config reload to 15s from 5m. The original intent (see #725) was to have a fleet of refineries gossiping news of new configs, but that's tricky to get right and probably not needed, especially since for now at least, almost all installations will be using a config file, not a config url. - Add a random jitter to avoid thundering herds. - Update metadata to explain what it's really doing - Regenerate documentation.
honeycombio · Aug 2, 2023 · 4d3a459 · 4d3a459
1 parent 00fe2b3
commit 4d3a459
Show file tree

Hide file tree

Showing 7 changed files with 38 additions and 33 deletions.
diff --git a/config.md b/config.md
@@ -1,7 +1,7 @@
 # Honeycomb Refinery Configuration Documentation
 
 This is the documentation for the configuration file for Honeycomb's Refinery.
-It was automatically generated on 2023-08-01 at 15:26:34 UTC.
+It was automatically generated on 2023-08-02 at 02:45:17 UTC.
 
 ## The Config file
 
@@ -87,14 +87,14 @@ When Refinery receives telemetry using an API key associated with a Honeycomb Cl
 
 ConfigReloadInterval is the average interval between attempts at reloading the configuration file.
 
-A single instance of Refinery will attempt to read its configuration and check for changes at approximately this interval.
-This time is varied by a random amount to avoid all instances refreshing together.
-Within a cluster, Refinery will gossip information about new configuration so that all instances can reload at close to the same time.
-Disable this feature with a value of `0s`.
+Refinery will attempt to read its configuration and check for changes at approximately this interval.
+This time is varied by a random amount up to 10% to avoid all instances refreshing together.
+In installations where configuration changes are handled by restarting Refinery, which is often the case when using Kubernetes, disable this feature with a value of `0s`.
+If the config file is being loaded from a URL, it may be wise to increase this value to avoid overloading the file server.
 
 - Not eligible for live reload.
 - Type: `duration`
-- Default: `5m`
+- Default: `15s`
 
 ## Network Configuration
 

diff --git a/config/file_config.go b/config/file_config.go
@@ -4,6 +4,7 @@ import (
 	"errors"
 	"fmt"
 	"math"
+	"math/rand"
 	"net"
 	"os"
 	"regexp"
@@ -216,7 +217,7 @@ type GeneralConfig struct {
 	ConfigurationVersion int      `yaml:"ConfigurationVersion"`
 	MinRefineryVersion   string   `yaml:"MinRefineryVersion" default:"v2.0"`
 	DatasetPrefix        string   `yaml:"DatasetPrefix" `
-	ConfigReloadInterval Duration `yaml:"ConfigReloadInterval" default:"5m"`
+	ConfigReloadInterval Duration `yaml:"ConfigReloadInterval" default:"15s"`
 }
 
 type NetworkConfig struct {
@@ -505,7 +506,9 @@ func NewConfig(opts *CmdEnv, errorCallback func(error)) (Config, error) {
 
 func (f *fileConfig) monitor() {
 	f.done = make(chan struct{})
-	f.ticker = time.NewTicker(time.Duration(f.mainConfig.General.ConfigReloadInterval))
+	// adjust the time by +/- 10% to avoid everyone reloading at the same time
+	reload := time.Duration(float64(f.mainConfig.General.ConfigReloadInterval) * (0.9 + 0.2*rand.Float64()))
+	f.ticker = time.NewTicker(time.Duration(reload))
 	for {
 		select {
 		case <-f.done:

diff --git a/config/metadata/configMeta.yaml b/config/metadata/configMeta.yaml
@@ -60,19 +60,20 @@ groups:
       - name: ConfigReloadInterval
         type: duration
         valuetype: nondefault
-        default: 5m
+        default: 15s
         reload: false
         validations:
           - type: minimum
             arg: 0s
         summary: is the average interval between attempts at reloading the configuration file.
         description: >
-          A single instance of Refinery will attempt to read its configuration
-          and check for changes at approximately this interval. This time is
-          varied by a random amount to avoid all instances refreshing together.
-          Within a cluster, Refinery will gossip information about new
-          configuration so that all instances can reload at close to the same
-          time. Disable this feature with a value of `0s`.
+          Refinery will attempt to read its configuration and check for changes
+          at approximately this interval. This time is varied by a random amount
+          up to 10% to avoid all instances refreshing together. In installations
+          where configuration changes are handled by restarting Refinery, which
+          is often the case when using Kubernetes, disable this feature with a
+          value of `0s`. If the config file is being loaded from a URL, it may
+          be wise to increase this value to avoid overloading the file server.
 
   - name: Network
     title: "Network Configuration"

diff --git a/refinery_config.md b/refinery_config.md
@@ -63,14 +63,14 @@ When Refinery receives telemetry using an API key associated with a Honeycomb Cl
 
 `ConfigReloadInterval` is the average interval between attempts at reloading the configuration file.
 
-A single instance of Refinery will attempt to read its configuration and check for changes at approximately this interval.
-This time is varied by a random amount to avoid all instances refreshing together.
-Within a cluster, Refinery will gossip information about new configuration so that all instances can reload at close to the same time.
-Disable this feature with a value of `0s`.
+Refinery will attempt to read its configuration and check for changes at approximately this interval.
+This time is varied by a random amount up to 10% to avoid all instances refreshing together.
+In installations where configuration changes are handled by restarting Refinery, which is often the case when using Kubernetes, disable this feature with a value of `0s`.
+If the config file is being loaded from a URL, it may be wise to increase this value to avoid overloading the file server.
 
 - Not eligible for live reload.
 - Type: `duration`
-- Default: `5m`
+- Default: `15s`
 
 ## Network Configuration
 

diff --git a/refinery_rules.md b/refinery_rules.md
@@ -488,7 +488,7 @@ The comparison operator to use.
 String comparisons are case-sensitive.
 
 - Type: `string`
-- Options: `=`, `!=`, `>`, `<`, `>=`, `<=`, `starts-with`, `contains`, `does-not-contain`, `exists`, `not-exists`
+- Options: `=`, `!=`, `>`, `<`, `>=`, `<=`, `starts-with`, `contains`, `does-not-contain`, `exists`, `not-exists`, `has-root-span`
 
 ### `Value`
 

diff --git a/rules.md b/rules.md
@@ -1,7 +1,7 @@
 # Honeycomb Refinery Rules Documentation
 
 This is the documentation for the rules configuration for Honeycomb's Refinery.
-It was automatically generated on 2023-08-01 at 15:26:34 UTC.
+It was automatically generated on 2023-08-02 at 02:45:17 UTC.
 
 ## The Rules file
 
@@ -528,7 +528,7 @@ String comparisons are case-sensitive.
 
 Type: `string`
 
-- Options: `=`, `!=`, `>`, `<`, `>=`, `<=`, `starts-with`, `contains`, `does-not-contain`, `exists`, `not-exists`
+- Options: `=`, `!=`, `>`, `<`, `>=`, `<=`, `starts-with`, `contains`, `does-not-contain`, `exists`, `not-exists`, `has-root-span`
 
 ### `Value`
 

diff --git a/tools/convert/templates/configV2.tmpl b/tools/convert/templates/configV2.tmpl
@@ -2,7 +2,7 @@
 ## Honeycomb Refinery Configuration ##
 ######################################
 #
-# created {{ now }} from {{ .Input }} using a template generated on 2023-08-01 at 15:26:32 UTC
+# created {{ now }} from {{ .Input }} using a template generated on 2023-08-02 at 02:45:16 UTC
 
 # This file contains a configuration for the Honeycomb Refinery. It is in YAML
 # format, organized into named groups, each of which contains a set of
@@ -58,17 +58,18 @@ General:
     ## ConfigReloadInterval is the average interval between attempts at
     ## reloading the configuration file.
     ##
-    ## A single instance of Refinery will attempt to read its configuration
-    ## and check for changes at approximately this interval. This time is
-    ## varied by a random amount to avoid all instances refreshing together.
-    ## Within a cluster, Refinery will gossip information about new
-    ## configuration so that all instances can reload at close to the same
-    ## time. Disable this feature with a value of `0s`.
+    ## Refinery will attempt to read its configuration and check for changes
+    ## at approximately this interval. This time is varied by a random amount
+    ## up to 10% to avoid all instances refreshing together. In installations
+    ## where configuration changes are handled by restarting Refinery, which
+    ## is often the case when using Kubernetes, disable this feature with a
+    ## value of `0s`. If the config file is being loaded from a URL, it may
+    ## be wise to increase this value to avoid overloading the file server.
     ##
-    ## Accepts a duration string with units, like "5m".
-    ## default: 5m
+    ## Accepts a duration string with units, like "15s".
+    ## default: 15s
     ## Not eligible for live reload.
-    {{ nonDefaultOnly .Data "ConfigReloadInterval" "ConfigReloadInterval" "5m" }}
+    {{ nonDefaultOnly .Data "ConfigReloadInterval" "ConfigReloadInterval" "15s" }}
 
 ###########################
 ## Network Configuration ##