Add support for AWS temporary credentials

[JacobHenner]

Add support for AWS temporary credentials by allowing session tokens to
be specified.

Assuming Secrets are kept up-to-date with valid session tokens, scalers
using temporary credentials will error once after token expiration. The
scaler cache for the corresponding ScaledObject will be cleared, the
scaler will be rebuilt using the updated temporary credentials, and the
scaler will resume operation.

Signed-off-by: Jacob Henner code@ventricle.us

Checklist

• Commits are signed with Developer Certificate of Origin (DCO - learn more)
• Tests have been added
• A PR is opened to update the documentation on (repo)
  • Add documentation for using AWS temporary credentials keda-docs#642
• Changelog has been updated

Fixes #2495

[JacobHenner]

Assuming Secrets are kept up-to-date with valid session tokens, scalers using temporary credentials will error once after token expiration. The scaler cache for the corresponding ScaledObject will be cleared, the scaler will be rebuilt using the updated temporary credentials, and the scaler will resume operation.

I am looking for feedback on whether waiting for an error is an acceptable way to trigger a credential refresh (i.e. reread the contents of the k8s Secret, which has presumably been updated with an unexpired session token). This approach worked in my testing, but I do realize there are at least a few potentially unwanted effects:

• Scaler cache needs to be invalidated/rebuilt (probably not a big deal, but worth mentioning)
• An HPA Scaler Error will be recorded in the metrics each time a temporary credential expires (potentially a bigger deal, depending on the expectations of users consuming this metric)

keda/pkg/provider/provider.go

Lines 122 to 132 in 6b72e85

	if err != nil {
	scalerError = true
	logger.Error(err, "error getting metric for scaler", "scaledObject.Namespace", scaledObject.Namespace, "scaledObject.Name", scaledObject.Name, "scaler", scaler)
	} else {
	for _, metric := range metrics {
	metricValue, _ := metric.Value.AsInt64()
	metricsServer.RecordHPAScalerMetric(namespace, scaledObject.Name, scalerName, scalerIndex, metric.MetricName, metricValue)
	}
	matchingMetrics = append(matchingMetrics, metrics...)
	}
	metricsServer.RecordHPAScalerError(namespace, scaledObject.Name, scalerName, scalerIndex, info.Metric, err)

[zroubalik]

* An HPA Scaler Error will be recorded in the metrics each time a temporary credential expires (potentially a bigger deal, depending on the expectations of users consuming this metric)

Are you able to distinguish the error caused by expired credentials? We might want to add a check in here to not include that in the metric or maybe better would be to introduce a new type of metrics for credentials expiration?

[JacobHenner]

Are you able to distinguish the error caused by expired credentials? We might want to add a check in here to not include that in the metric or maybe better would be to introduce a new type of metrics for credentials expiration?

The logs reflect a 403 "ExpiredToken", but I haven't explored if the library that interacts with AWS parses the specifics of the error (or if it's just returning error text from the API), or how the specifics could be propagated back to KEDA.

Assuming the error can be propagated back, how would we prefer to handle it?

We could:

• Create a new type of metric for creds expiration - but I think that'd be similarly confusing (one event at expiration time is expected, > 1 event at expiration time is bad and indicates actual scaler failure).
• Only report a failure for credential expiration errors if there are >= 2 consecutive failures
  • We might want to consider a similar approach for static credentials too, as they can also be rotated (generally with a much lower frequency).
• Build an independent credential refresh mechanism - e.g. Watch created HPA and TriggerAuthentication resources #511, TriggerAuthentication Secret is not reloaded when Secret changes #563
• (something else?)

[zroubalik]

Are you able to distinguish the error caused by expired credentials? We might want to add a check in here to not include that in the metric or maybe better would be to introduce a new type of metrics for credentials expiration?

The logs reflect a 403 "ExpiredToken", but I haven't explored if the library that interacts with AWS parses the specifics of the error (or if it's just returning error text from the API), or how the specifics could be propagated back to KEDA.

Assuming the error can be propagated back, how would we prefer to handle it?

We could:

* Create a new type of metric for creds expiration - but I think that'd be similarly confusing (one event at expiration time is expected, > 1 event at expiration time is bad and indicates actual scaler failure).

* Only report a failure for credential expiration errors if there are >= 2 consecutive failures
  
  * We might want to consider a similar approach for static credentials too, as they can also be rotated (generally with a much lower frequency).

* Build an independent credential refresh mechanism - e.g. [Watch created HPA and TriggerAuthentication resources #511](https://github.com/kedacore/keda/issues/511), [TriggerAuthentication Secret is not reloaded when Secret changes #563](https://github.com/kedacore/keda/issues/563)

* (something else?)

Those are all good questions, I think we can go ahead with this change as it is now and open an issue for the follow up?
I will do a proper review tomorrow, but could you please update the Changelog as well?
Thanks!

[JacobHenner]

Those are all good questions, I think we can go ahead with this change as it is now and open an issue for the follow up?
I will do a proper review tomorrow, but could you please update the Changelog as well?
Thanks!

Yep, that's fine by me. I'll make the changelog addition in a bit. Thanks!

[zroubalik]

LGTM

[zroubalik]

@JacobHenner could you please open follow up issue with the concerns we raised here?

[JacobHenner]

@JacobHenner could you please open follow up issue with the concerns we raised here?

#2578