[BUG] Connection headers mutability and unintended reuse of headers

[mmtdm]

What is the bug?

Connection.js headers property is reused over multiple requests, and due to mutability of headers, this can cause unwanted headers to be sent with subsequent requests.

How can one reproduce the bug?

Using opensearch-project/opensearch with aws-opensearch-connector, call search repeatedly from multiple expressjs processes, being sure to execute queries on es cluster that run for more than 5 minutes (Ultrawarm index in our case).

Eventually, an X-Amz-Date header will be added to Connection.headers, and due to logic in aws4.js, this header will be sent with every request, and never get updated.

Eventually, the permanent X-Amz-Date header will expire, and all subsequent requests will fail with "Signature expired" error.

The issue only resolves itself when you recreate a new Client (which recreates a new Connection). Eventually, the header will be rewritten though, and the process will repeat itself.

What is the expected behavior?

headers should not be reused across multiple distinct requests. Connection class should not use a reference to this.headers when initializing a request, it should make a copy:

headers: Object.assign({}, this.headers), // create a clone

What is your host/environment?

Linux/node 16

Do you have any screenshots?

Do you have any additional context?

This issue seems to only arise in the presence of multiple requests being handled for an es cluster that has queued search threads and long running query executions that exceed 5 minutes.

Is there an initialization or usage pattern that would result in ability to pass in blank headers to each request? Otherwise, it really does seem like the Connection headers property is being passed around by reference but also reused over multiple requests.

[harshavamsi]

Hi, @mmtdm opensearch-js now supports sigv4 authentication support out of the box. We've enhanced the implementation over how requests are made over the transport layer and this takes into consideration credentials expiring. Can you give this a try and let us know if this helps your case? This is still un-released and will require you to try it from src if you are interested.

[mmtdm]

Thank you kindly @harshavamsi I will try your recommendation and report on my findings.

But I humbly request your opinion about whether or not there is any merit to my findings. It seems as though opensearch is initializing request.headers by reference from Connection.headers, and anything done to request.headers will be permanently made to Connection.headers, affecting all subsequent requests.

Is this intentional? Am I missing something here?

Thank you, cheers

[dblock]

I think this is a valid bug as long as headers are manipulated in the library code. Meaning duplicating headers when they get modified would be ok. In either case, @mmtdm want to submit a fix?

I would do a bit of internet searching to see if other HTTP libraries have this problem and how they deal with it.

[mmtdm]

Thank you @dblock I will submit a proposed fix shortly