Improve Opensearch compatibility

[smlx]

Opensearch and Opensearch Dashboards is the open source successor of Opendistro Elasticsearch and Kibana. Since Opendistro is no longer maintained Lagoon will have to migrate to the Opensearch stack going forward.

Lagoon has a couple of compatibility issues dealing with the Opensearch stack vs Opendistro. Here are the problems that I've noticed from running the current opendistro-sync script:

Creating index patterns

[2021-09-21 15:30:29] [error]: Kibana Error during setup of index pattern "application-logs-*": HTTPError: Response code 400 (Bad Request)

I think this one is due to the rename of the kbn-xsrf header to osd-xsrf? This curl command works:

curl -sSL -u "$USER_AUTH" 'https://opensearch.example.com/api/saved_objects/index-pattern' -XPOST -H 'Content-Type: application/json' -H 'securitytenant: exampletenant' -H 'osd-xsrf: true' --data-raw '{"attributes":{"title":"router-logs-*","timeFieldName":"@timestamp"}}'

Configuring default index (?)

I'm not actually sure what this is supposed to do, but this error is logged:

[2021-09-21 15:30:29] [error]: Kibana Error during config of default Index: HTTPError: Response code 404 (Not Found)

Backend role mapping

Opensearch needs the backend role mapped to the regular role otherwise logging in via Keycloak gives no permissions. In practice for the way that Lagoon uses roles this means adding the name of the role as a backend role. e.g. this curl command works:

curl -ksSL -u "$USER_AUTH" -XPUT 'https://localhost:9200/_plugins/_security/api/rolesmapping/example-lagoon-group' -H 'Content-Type: application/json' --data-raw '{"users":[],"backend_roles":["example-lagoon-group"],"hosts":[]}'"

[tobybellwood]

so - looking back at this - and checking the Lagoon that is running Opensearch

1. Has been done
2. Needs some investigation (but I don't think is a deal breaker, as a default index is selected as a backup anyway)
3. Has been done