Add annotations to interceptor service

[rwkarg]

This issue (#36) has been closed but I don't see a way to currently add annotations to the interceptor service.

The specific use case we have is using autoneg-controller to add workload instances from multiple clusters as backends to a single Load Balancer. In GCP, this requires that the service (would be the interceptor service) has two annotations to create the Network Endpoint Group (NEG) and to associate that NEG with a Backend Service on the Load Balancer (not important what it does, just that we need annotations on the interceptor service).

Looks like #206 may complicate this, if there is only one service. We can't add the annotations to individual services any more for autoneg-controller to work in that case.

Is the multi-tenant interceptor something that can be opted out of?

[tomkerkhove]

That's good input for #183 / @arschles @yaron2.

I think the sweet spot here is being able to install dedicated interceptors with multi-tenant interceptor as default. This would allow you to have 1 KEDA HTTP installation, but tailored to your need since multiple installations won't work due to only-one-operator limitation.

[tomkerkhove]

That's what a team is for!

[arschles]

see issue #241

[rwkarg]

The proposal in #241 looks to give the flexibility to allow either one interceptor for all services (the default), configuring a separate interceptor for each service, or anything in between like grouping several services behind one interceptor/annotation set and the rest of the services behind other custom interceptors or the default interceptor.

[rwkarg]

For naming, would HttpScalingInterceptor be a more specifically descriptive name (for a user) for what is named HTTPScalingComponents in #241 ?

I suppose that it is technically both the interceptor resources and the scaler resources are created as a result of an instance of that CR, but from a user standpoint, my initial thought is that I'm asking for a new interceptor which just happens to have the scaler resources that get created along with that as an implementation detail. Naming is hard.

[arschles]

@rwkarg I think HTTPScalingInterceptor is too specific - we need to denote that it's installing the interceptor and more (the scaler, for now), and at the moment, interceptors and scalers have a 1:1 relationship.

another thing that might be coming is #6. if/when we move forward with that, this CRD might be more flexible than just interceptor and scaler. in that world, a new CRD would notify the controller of an existing SMI-compatible service mesh, ingress controller, etc... and it would respond by configuring a scaler to work with that.

generally speaking, I think the name, substance, and structure of this new CRD needs to be flexible enough to handle a non-interceptor world in the future.