Spec Conformance Review: OTLP Exporter Retries

[MadVikingGod]

The OTLP exporter Must Retry on transient errors.

Acceptance Criteria:

• OTLP/grpc driver retries when it experiences a transient error
• OTLP/http driver retries when it experiences a transient error

[XSAM]

Part of #1603

[MrAlias]

The gRPC driver for the OTLP exporter accepts a config that has a retry policy enabled by default:

opentelemetry-go/exporters/otlp/otlpgrpc/options.go

Lines 25 to 59 in 841d2a5

	// DefaultServiceConfig is the gRPC service config used if none is
	// provided by the user.
	//
	// For more info on gRPC service configs:
	// https://github.com/grpc/proposal/blob/master/A6-client-retries.md
	//
	// For more info on the RetryableStatusCodes we allow here:
	// https://github.com/open-telemetry/oteps/blob/be2a3fcbaa417ebbf5845cd485d34fdf0ab4a2a4/text/0035-opentelemetry-protocol.md#export-response
	//
	// Note: MaxAttempts > 5 are treated as 5. See
	// https://github.com/grpc/proposal/blob/master/A6-client-retries.md#validation-of-retrypolicy
	// for more details.
	DefaultServiceConfig = `{
	"methodConfig":[{
	"name":[
	{ "service":"opentelemetry.proto.collector.metrics.v1.MetricsService" },
	{ "service":"opentelemetry.proto.collector.trace.v1.TraceService" }
	],
	"retryPolicy":{
	"MaxAttempts":5,
	"InitialBackoff":"0.3s",
	"MaxBackoff":"5s",
	"BackoffMultiplier":2,
	"RetryableStatusCodes":[
	"CANCELLED",
	"DEADLINE_EXCEEDED",
	"RESOURCE_EXHAUSTED",
	"ABORTED",
	"OUT_OF_RANGE",
	"UNAVAILABLE",
	"DATA_LOSS"
	]
	}
	}]
	}`

This was added here for reference.

[MadVikingGod]

Yes, but according to GRPC documents you can't enable retry except with an environment variable.

Unless we are going to rewrite the retry logic within our code, I don't think this is going to be fixed. We could maybe have some documentation that you actually get retries with GRPC_GO_RETRY=on.

[MadVikingGod]

I did a bit of experimenting with it. Here is an example of a client not retrying, and how to get it to retry.
https://github.com/MadVikingGod/opentelemetry-go/tree/retry-example/exporters/otlp/example

[MrAlias]

Yes, but according to GRPC documents you can't enable retry except with an environment variable.

Ah, right, we ran into this on the old issues as well. One option that proposed was to copy the collector and write our own implementation: #561 (comment)

[MrAlias]

We may need to write our own back-off/retry algorithm (like the collector) instead of relying on the gRPC method.