Slowness to retrieve data in dashboard ( POST calls fail repeatedly )

[Ardomotica]

What's happening

Discussed in : https://community.grafana.com/t/grafana-timestream-data-source/132439?u=xbox3535889

I am using Grafana to display data from AWS Timestream, it seems to be very slow to display the data, I have tried changing some datasource configuration but the situation remains the same, it is really very slow, do you have any ideas to improve performance ?

In this picture you can see the dashboard loading, it stays loading for several minutes.

In this second pic you’ll find the datasource configuration

I did some testing by trying to run the same query on grafana on the Timestream query editor and it takes about 2 sec :

Reloading grafana after a few minutes, it started the slow loading again, and opening Chrome’s DevTool you can see how the query calls repeatedly fail until an answer comes after several minutes, as you can see :

Anything else we need to know?: As you can see in the plugin configuration i've used "SDK Default" ad Authentication provider

Grafana version: Grafana v11.2.0 (c57667e448)
Plugin version: 2.9.8
OS Grafana is installed on: EC2 Amazon Linux V2 / Linux/UNIX
User OS & Browser: Linux/UNIX
Others:

[iwysiu]

Hi @Ardomotica ! We still need to take a look at this, but based on the linked community post, it sounds like you're having an issue with the SDK default taking a long time to get the token. Do you still have this issue if you try another authentication method?

[Ardomotica]

Hi @iwysiu I've tried with another Authentication Provider (credentials file) using my account and it works properly! But this solution is not useful for our use-case, our Grafana instance will be deployed on a EC2 and all security is managed by an Instance Profile attached to the EC2.

[iwysiu]

Hi again @Ardomotica ! It looks like this is a known problem with the AWS SDK, but there are workarounds that other people have tried in this issue that you could look at and try. If that doesn't help we can take another look at it.

[Ardomotica]

Yes thank you! but I solved the problem in another way. We have the Grafana instance inside a docker container and reading here : https://mrkaran.dev/tils/aws-ec2-metadata-inside-docker/ we realised that we needed to set the TTL to a high enough number to be able to get the token to the container as the EC2 had IMDSv2 set, this solved the problem for us, I hope it was helpful

[iwysiu]

I'm glad you were able to fix it!