Support for HTTP/2

[jowerner]

Nowadays more and more servers talk HTTP/2. Would be great if HtmlUnit would support that protocol as well. Do you have plans to add that feature to HtmlUnit in the near future, now that Apache HttpClient 5.x is available? If so, let me know if we can help with that. Thanks!

[rbri]

Plans - yes, as always the time is a problem. It will be fantastic, if you can provide a PR.

[jowerner]

Yeah, same here. Okay, let's see what we can do. Will get back to you if we have something to review or just need your advice. :)

[rbri]

Great....

[jowerner]

Meanwhile, I spent some time migrating HtmlUnit to Apache HttpClient 5.x and now want to give you a short update.

HC 5.x comes in two flavors, synchronous (or "classic") and asynchronous. I started with the synchronous variant as it is very similar to 4.x and fits in very well. Only when this was done did I realize that the synchronous client does not support HTTP/2 which ultimately is the goal of this exercise. So I went ahead and reworked the code once more to use the asynchronous client.

The asynchronous and the synchronous variants share a bit of code, but are otherwise two different beasts with different sets of support classes. That's why customization is quite different as well. Many things have been moved to other places. For example, we can no longer set socket factories for plain and SSL sockets (or I simply did not find this). However, there are alternative ways to set custom key/trust stores, cipher suites, protocols, etc. That's why some functionality currently located in the socket factories is still missing. Thinking of SOCKS proxy support, for example. Furthermore, there are couple of test cases in package com.gargoylesoftware.htmlunit that are still failing at the moment.

As you can see there is still a bit of work to do. But before putting more effort into this, it is probably time to get feedback from you whether we are on the right track. If you like, I can provide a PR, as a sneak preview for you and a place where we can discuss details? Please let me know what you think. Thanks!

[rbri]

Wow, this sounds really great.

From my point of view we

• can declare the next version as 3.0 (because this is a major change)
• we will try to integrate your code step by step as soon as possible - i only have two other topics to work on - test the new promise support from rhino and build a selenium4 htmlunit webdriver. But i think getting this done can be the top prio.

If you can make a pr i have a look at your code on Friday, and if you like we can have an online meeting about this also (maybe also on Friday).

[rbri]

Looks like we have to switch to slf4j for logging also

[jowerner]

Sounds like a plan! Will make the PR available later this day or maybe tomorrow morning. Would be great if we can have our online meeting tomorrow as well because next week I'll be on vacation. Tomorrow afternoon anytime after 14:00 would suit me best. How about you? Will you or should I take care of the meeting?

[rbri]

please send a private mail to rbri at rbri dot de and i will send you an invitation

14:00 is fine for me - i have a private jitsi server instance .... and will send you the access details.

[rbri]

Have done a quick look at OkHtml and Jetty Client - both look also promising.
Will have a look at the interface between HtmlUnit itself and the HttpClient - maybe we can have a clear interface for the transport layer.

[mbucc]

FYI, Java 11's java.net.http.HttpClient:

supports HTTP/1.1 and HTTP/2, both synchronous and
asynchronous programming models, handles request and
response bodies as reactive-streams, and follows the familiar
builder pattern.

https://openjdk.java.net/groups/net/httpclient/intro.html

[rbri]

see #776