[filebeat][streaming] - Added support for TLS & forward proxy configs for websockets

[ShourieG]

Type of change

• Enhancement
• Docs

Proposed commit message

Added support for functional TLS config & proxy configs for websockets by implementing a custom transport object and NetDialContext() and leveraging the httpcommon library. Added supporting tests to accompany

Note:

Tried using in memory certs and keys but our current httpcommon library always asks for a file.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

No enduser impact as this is all additive.

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Closes
• Relates [Proofpoint On Demand] Investigate adding forward proxy support integrations#11581

Use cases

Screenshots

Logs

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @ShourieG? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[botelastic]

This pull request doesn't have a Team:<team> label.

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @ShourieG? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b websocket/tls_proxy upstream/websocket/tls_proxy
git merge upstream/main
git push upstream websocket/tls_proxy

[efd6]

Suggested change

	This specifies the time to wait for the `websocket` handshake and the `http.Upgrade()` operation to complete. This timeout occurs at the application layer and not at the TCP layer. The `default value` is `20` seconds. This setting is specific to the `websocket` streaming input type only.
	This specifies the time to wait for the WebSocket handshake and protocol upgrade to complete. This timeout occurs at the application layer and not at the TCP layer. The `default value` is `20` seconds. This setting is specific to the `websocket` streaming input type only.

Referring to a call is unhelpful to users; discuss the intent not the mechanism in user-facing documentation. Also, it was incorrectly referring to http.Upgrade which does not exist. It is used by websocket.Upgrader.Upgrade, but we don't call that except in testing, instead using websocket.Dialer.DialContext (should we refer to upgrading at all? I think this is an implementation detail and only tenuously relevant). Note that the lib's default is 45s; do we want to match that or is there a reason to use 20s?

I'm curious, is it meaningful to have a handshake timeout that is less than the TCP timeout? Looking at the implementation of the client, the handshake timeout is used to construct a deadline context that must lose the race to return from DialContext, and this context is passed into the net.Dialer.DialContext call, meaning that obtaining the netconn must fit within this deadline. Where does the timeout below come in?

[ShourieG]

@efd6,
What I was going for initially was faster failures at the application layer to achieve better program responsiveness. But then I realised that we might want to wait for TCP retires to overcome any intermittent network issues present and setting a handshake timeout lesser than the tcp timeout defeats that purpose.

The timeout option below actually controls the net.Dialer.DialContext timeout, which by default is set to 90s, half of the os default for TCP timeout window.

So my question is, should we extend the DialContext and handshake timeouts to mirror the TCP time out value of the os ?

As for the documentation, I'll update it.

[efd6]

I suggest taking a look at the gorilla code to see how the timeout is propagated.

[ShourieG]

@efd6, I saw the gorilla code just creates a context deadline out of the provided timeout in the dialcontext. So in this case having the handshake value < the dial context doesn't make much sense by default. We can keep them at the same value then if the user feels they need to change they can do that.

[efd6]

I would not bother adding the handshake timeout since we already provide a mechanism to hand in a dial timeout which achieves the same end.

[ShourieG]

@efd6, I've addressed the PR suggestions. I was wondering should we back port this to 8.17 ? I know the FF has already happened but this might help out a lot of users currently facing issues.