Add Sigv4 support for the main connection

[chadmyers]

Is your feature request related to a problem? Please describe.

I would like to connect my OpenSearch Dashboards (2.11.1, hosted in AWS ECS using my own container image based on the main dashboards image but with some tweaks) to my OpenSearch domain (2.11) hosted in AWS OpenSearch Service. But OpenSearch Dashboards doesn't seem to support sigv4 from the opensearch_dashboards.yml file, only from Data Sources once you've got OpenSearch Dashboards up and running. Right now, I need to use a awslabs/aws-sigv4-proxy signing proxy sidecar container in the ECS task definition in order to be able to handle the sigv4 process.

Describe the solution you'd like

Since OpenSearch Dashboards supports sigv4 for Data Sources, could we also add it to the main OpenSearch connection in the yaml file?

Maybe something like:

server.authtype=sigv4

And then support various AWS-SDK auth types:

server.authtype.sigv4.assume_role=(ARN)
-OR-
server.authtype.sigv4.iam_aws_access_key_id=(ID)
server.authtype.sigv4.iam_aws_secret_access_key=(KEY)
-ETC-

Describe alternatives you've considered

awslabs/aws-sigv4-proxy sidecar container

[wbeckler]

This is a great idea. I don't understand how people can connect their self hosted dashboards to the managed service without this feature.

[seraphjiang]

Thanks @chadmyers for asking,

If you are running your self-hosted OpenSearch Dashboards, we do have an out-of-the-box way to support AWS SigV4 with multiple datasource features enabled

You may find this feature here in playground

https://playground.opensearch.org/app/management/opensearch-dashboards/dataSources/create

refer to detail instruction below

https://opensearch.org/docs/latest/dashboards/management/multi-data-sources/

cc: @kgcreative @elfisher @zengyan-amazon @Pallavi-AWS

[chadmyers]

@seraphjiang thank you for your reply. My understanding is that Data Sources is only available after you've already connected OpenSearch Dashboards to an OpenSearch cluster without sigv4. As I said above:

But OpenSearch Dashboards doesn't seem to support sigv4 from the opensearch_dashboards.yml file, only from Data Sources once you've got OpenSearch Dashboards up and running.

Is that a correct statement? How do I use sigv4 to connect to the server host mentioned in the opensearch_dashboards.yml?

[seraphjiang]

@seraphjiang thank you for your reply. My understanding is that Data Sources is only available after you've already connected OpenSearch Dashboards to an OpenSearch cluster without sigv4. As I said above:

yes. you need to connect to a cluster.

The https://playground.opensearch.org/ is setup in this way, and connect to two Amazon OpenSearch Service domain as example.

Is that a correct statement? How do I use sigv4 to connect to the server host mentioned in the opensearch_dashboards.yml?

You may run a local cluster on localhost:9200 together with you dashboards, so dashboards save/load meta data(Saved Objects) into this local cluster.

if you don't want to setup local cluster, other way is to save/load the OpenSearch Dashboards meta data from other storage type like Postgresql, DynamoDB.

We have a separate issue to track this.
#1441

[chadmyers]

@seraphjiang Thanks again for the reply. I think for us, since we host OSD in ECS/Fargate, setting up another OS cluster either in Fargate or in AWS OpenSearch Service would be more complicated and expensive than just using the aws-sigv4-proxy container.

Storing the metadata in DynamoDB would be fantastic! Thanks for making me aware of this!