[otelcol] Determine if NewCommand should supply any default providers/converters

[TylerHelmuth]

There is currently a TODO in otelcol here that states:

// Puts command line flags from flags into the CollectorSettings, to be used during config resolution.
func updateSettingsUsingFlags(set *CollectorSettings, flags *flag.FlagSet) error {
	resolverSet := &set.ConfigProviderSettings.ResolverSettings
	configFlags := getConfigFlag(flags)

	if len(configFlags) > 0 {
		resolverSet.URIs = configFlags
	}
	if len(resolverSet.URIs) == 0 {
		return errors.New("at least one config flag must be provided")
	}
	// Provide a default set of providers and converters if none have been specified.
	// TODO: Remove this after CollectorSettings.ConfigProvider is removed and instead
	// do it in the builder.
	if len(resolverSet.ProviderFactories) == 0 && len(resolverSet.ConverterFactories) == 0 {
		set.ConfigProviderSettings = newDefaultConfigProviderSettings(resolverSet.URIs)
	}
	return nil
}

We have removed ConfigProvider via #10281 so we could start working on this todo, except that it would be a breaking change to stop providing default providers/converters.

We have an explicit unit test via #9302 that tests that we supply default providers/converters for jaeger all-in-one.

Should we move forward with the breaking change (which we know would break jaeger)? What problem are we solving by not providing default providers/converters?

[TylerHelmuth]

Setting of default converters is part of the conversation of #10259, where we are deprecating expandconverter in favor of a DefaultScheme: "env".

If we continue to supply defaults, instead of setting the expandconverter the default would be to set DefaultScheme: "env".

[evan-bradley]

I'd like to remove all defaults and have these supplied by distro authors through the APIs or through the builder.

[TylerHelmuth]

@yurishkuro how is Jaeger all-in-one depending on this feature? Is it using the otelcol API (NewCommand) or a community distribution like Core or Contrib?

[TylerHelmuth]

I'm guessing the relevant code that would need to start explicitly defining providers/converters is here

[andrzej-stencel]

Currently we still include all the providers as dependencies because of the newDefaultConfigProviderSettings function, even if distro builders explicitly define a subset of providers to use via the builder config file, correct? 🤔

[evan-bradley]

Currently we still include all the providers as dependencies because of the newDefaultConfigProviderSettings function, even if distro builders explicitly define a subset of providers to use via the builder config file, correct? 🤔

This function is only called here when distro authors don't specify any of these in the otelcol.CollectorSettings struct they give to NewCommand. The builder also sets these by default when they haven't been specified in the config, which I expect to stick around even if we remove newDefaultConfigProviderSettings. Since the builder sets defaults, newDefaultConfigProviderSettings isn't called in distros created by ocb.

[yurishkuro]

Why are we trying to remove default behavior? There is a general principle in stdlib that structs should be largely usable with zero values. The ability to cherry-pick config resolvers seems quite obscure capability to me (e.g. the viper lib doesn't force you to choose resolvers), having a sensible default is much more useful.

[TylerHelmuth]

Another question: what guarantees do we provide for default providers/converters? Are we allowed to change them without a breaking change or do that need to follow the deprecation process?

[evan-bradley]

Why are we trying to remove default behavior? There is a general principle in stdlib that structs should be largely usable with zero values. The ability to cherry-pick config resolvers seems quite obscure capability to me (e.g. the viper lib doesn't force you to choose resolvers), having a sensible default is much more useful.

I would say that we're moving where the default behavior lives rather than removing it. Specifically, we're moving it from our APIs to the Collector builder. I think this change would fit the users of each:

1. Collector builder users are likely end-users who want good defaults.
2. Distro authors who skip the builder and call the code directly are likely to be more advanced and comfortable deciding which providers/converters they want included in their distro.

To me, moving the defaults out of the otelcol package feels cleaner and should result in smaller binaries for users who want to customize which providers they include.

If we were to keep these defaults in our APIs and want to look to to the principle of structs being usable with zero-valued settings, I would argue we should consider moving these to confmap.Resolver.

Another question: what guarantees do we provide for default providers/converters? Are we allowed to change them without a breaking change or do that need to follow the deprecation process?

In my opinion they are considered part of the stability guarantees for the package that includes them and would need to adhere to the deprecation process. I think this applies whether we have them in our APIs or in the builder.

[mx-psi]

Why are we trying to remove default behavior?

Having otelcol be agnostic to specific confmap provider implementations seems desirable to me. It allows us to evolve the two more independently, and seems easier to reason with and maintain. It also allows us to stabilize the otelcol module independently from specific providers.

There is a general principle in stdlib that structs should be largely usable with zero values

This is not something followed by some popular libraries (think zap.Logger, I am pretty sure Go's SDK metric.MeterProvider also would break). It is very restrictive and also not the general direction we are taking (see #9508).

The ability to cherry-pick config resolvers seems quite obscure capability to me (e.g. the viper lib doesn't force you to choose resolvers), having a sensible default is much more useful.

We will have a default on the builder, just no the Go API.