We are having performance issue in Elasticsearch/Kibana after migrating to another server host.

[kyllr-analytics]

Hello,
We are having issue in Elasticsearch/Kibana.
We have AWS Proxy and Kibana 7.1 setup in the same server. After migrating both to another server host and make it live in the load balancer, we are facing request timeout issue when accessing our Kibana application.

Though when using the old server back in the load balancer, it works perfectly fine.
Does anyone know why this is happening? Any related information will be appreciated. Thank you!

PS: Both servers has the same specification except that the New is Windows 2016 and Old is Windows 2012

By the way, here are the settings that we have:
AWS Elasticsearch Proxy
aws-es-proxy-0.9-windows-amd64.exe -endpoint "https://vpc_endpoint " -listen "127.0.0.1:9230"

Kibana.yml
server.port: 5609
server.host: "x.x.x.x"
server.basePath: "/7.1/kibana"
elasticsearch.hosts: "http://127.0.0.1:9230/"
elasticsearch.ssl.verificationMode: none
elasticsearch.requestTimeout: 300000

Its Cluster Health status is OK.

[kyllr-analytics]

Could anyone help me with this? This is a production issue.

Right now, our suspicion would be the aws proxy are having incompatibility with Windows Server 2016 that's why the issue only appears when pointing to it.

Here's the comparison of the instances where we're running Proxy and Kibana.

Old - which having no issue
Windows Server 2012 R2
Processor: Inter(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz 2.90 GHz
Installed memory (RAM): 30.0 GB
System type: 64-bit OS, x64-based processor

New - which has the issue
Windows Server 2016
Processor: Inter(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz 3.00 GHz
Installed memory (RAM): 31.0 GB
System type: 64-bit OS, x64-based processor

[abutaha]

Hi, Can you try a newer version of the proxy? https://github.com/abutaha/aws-es-proxy/releases/

Do you face the same symptoms when you run your queries from the Win2016 directly? Or it only appears when it is behind a Load Balancer?

[abutaha]

@kyllr-analytics one more thing to add, are you using AWS IAM roles? If yes, did you check the same role is attached to the newer instance?

[kyllr-analytics]

Hi @abutaha

Thank you for taking your time to look into this concern.

Already tried using latest version of proxy (v1.1) and got the same result, Kibana did not load properly or sometime Elasticsearch request timed out.

This only happens in Win2016 when it is behind the load balancer with all the load sending logging data to it.

We have a workaround which is to reboot the server, ES/Kibana works perfectly fine after the reboot. However, after an hour, we've got to face the same result.