[BUG] Failed to load indices with correct user permissions only for specific index pattern

[geckiss]

What is the bug?
An internal user with indices:admin/get permission for specific index pattern (e.g. pattern org.service1*) is unable to load indices in Query Workbench.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Deploy OpenSearch and OpenSearch Dashboard 2.13.0 in Docker
2. As admin user, create new custom tenant named my.custom.tenant
3. As admin user, create new internal user named user1, grant him read-write permissions to his new custom tenant, grant him cluster_composite_ops_ro cluster permissions and indices:admin/get index permissions for index pattern org.service1*
4. As admin user, create index org.service1
5. Log in as newly created internal user, switch to your custom tenant and go into Query Workbench
6. Window with indices shows Failed to load indices\nFailed to load OpenSearch indices, please check user permissions

What is the expected behavior?
The user should be able to only see indices matching index pattern org.service1*, in this reproduction example he would only see index org.service1. He shouldn't be able to see other indices.

What is your host/environment?

• OS: Official OpenSearch and OpenSearch Dashboards Docker images
• Version: 2.13.0
• Pre-packaged plugins + Prometheus metrics plugin in cluster

Do you have any screenshots?

Do you have any additional context?
Adding indices:admin/get permission for index pattern * will show all indices. However, this is not desirable. Adding indices:admin/get permission for index pattern .kibana* does not help.

Dashboard container logs 403:

StatusCodeError: Authorization Exception
2024-05-31T10:23:51.143686350+02:00     at respond (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:349:15)
2024-05-31T10:23:51.143701821+02:00     at checkRespForFailure (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:306:7)
2024-05-31T10:23:51.143713502+02:00     at HttpConnector.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
2024-05-31T10:23:51.143725358+02:00     at IncomingMessage.wrapper (/usr/share/opensearch-dashboards/node_modules/lodash/lodash.js:4991:19)
2024-05-31T10:23:51.143736323+02:00     at IncomingMessage.emit (node:events:529:35)
2024-05-31T10:23:51.143748634+02:00     at IncomingMessage.emit (node:domain:489:12)
2024-05-31T10:23:51.143761411+02:00     at endReadableNT (node:internal/streams/readable:1400:12)
    at processTicksAndRejections (node:internal/process/task_queues:82:21) {
2024-05-31T10:23:51.143823600+02:00   status: 403,
2024-05-31T10:23:51.143845513+02:00   displayName: 'AuthorizationException',
2024-05-31T10:23:51.143858831+02:00   path: '/_plugins/_sql',
2024-05-31T10:23:51.143871235+02:00   query: {},
2024-05-31T10:23:51.143882104+02:00   body: '{\n' +
2024-05-31T10:23:51.143894127+02:00     '  "error": {\n' +
2024-05-31T10:23:51.143911382+02:00     '    "reason": "Error occurred in OpenSearch engine: no permissions for [indices:admin/get] and User [name=user1, backend_roles=[], requestedTenant=my.custom.tenant]",\n' +
2024-05-31T10:23:51.143931419+02:00     '    "details": "OpenSearchSecurityException[no permissions for [indices:admin/get] and User [name=user1, backend_roles=[], requestedTenant=my.custom.tenant]]\\nFor more details, please send request for Json format to see the raw response from OpenSearch engine.",\n' +
    '    "type": "OpenSearchSecurityException"\n' +
2024-05-31T10:23:51.143954706+02:00     '  },\n' +
2024-05-31T10:23:51.143965997+02:00     '  "status": 403\n' +
2024-05-31T10:23:51.143976369+02:00     '}',
2024-05-31T10:23:51.143987021+02:00   statusCode: 403,
2024-05-31T10:23:51.143998370+02:00   response: '{\n' +
2024-05-31T10:23:51.144009196+02:00     '  "error": {\n' +
2024-05-31T10:23:51.144020541+02:00     '    "reason": "Error occurred in OpenSearch engine: no permissions for [indices:admin/get] and User [name=user1, backend_roles=[], requestedTenant=my.custom.tenant]",\n' +
2024-05-31T10:23:51.144032396+02:00     '    "details": "OpenSearchSecurityException[no permissions for [indices:admin/get] and User [name=user1, backend_roles=[], requestedTenant=my.custom.tenant]]\\nFor more details, please send request for Json format to see the raw response from OpenSearch engine.",\n' +
    '    "type": "OpenSearchSecurityException"\n' +
2024-05-31T10:23:51.144055207+02:00     '  },\n' +
    '  "status": 403\n' +
    '}',
2024-05-31T10:23:51.144089456+02:00   toString: [Function (anonymous)],
2024-05-31T10:23:51.144101319+02:00   toJSON: [Function (anonymous)]
2024-05-31T10:23:51.144113176+02:00 }

[dblock]

Catch All Triage - 1 2 3 4 5 6