Use WRY for prerender purposes possible?

[marklonquist]

Hi

Is it possible to use Wry to fetch completed html and return the html?

We're currently using prerender (https://github.com/prerender/prerender), which uses headless Chrome - and we all know how bad Chrome actually is when it comes to resource usage.

What I'm thinking is; make a rust application that exposes an API that accepts an URL (the page we want to prerender), run it through the WebView and then get back the html and return it through the API.

[amrbashir]

This is really nice use-case and I think WRY already supports that, just point it to the desired URL or load your HTML/CSS/JS directly, wait for DOMContentLoaded event or a timeout if you want and just use the IPC to send the value of document.documentElement.innerHTML

[ilyazub]

Here's a basic script to render HTML and print it to stdout after window.onload.

use std::fs::{canonicalize, read_to_string};

use wry::{
    application::{
        event::{Event, WindowEvent},
        event_loop::{ControlFlow, EventLoop},
        window::{Window, WindowBuilder},
    },
    webview::WebViewBuilder,
};

const RETURN_OUTER_HTML: &str = r#"
window.onload = function() {
  let html = document.documentElement.outerHTML;
  window.ipc.postMessage(html);
}
"#;

fn main() -> wry::Result<()> {
    enum UserEvents {
        CloseWindow,
    }

    let event_loop = EventLoop::<UserEvents>::with_user_event();

    let window = WindowBuilder::new()
        .with_title("Render HTML to string")
        .with_visible(false)
        .build(&event_loop)?;

    let html = read_to_string(canonicalize("./MY_HTML_FILE.html")?)?;

    let proxy = event_loop.create_proxy();

    let handler = move |_window: &Window, html: String| {
        println!("{}", &html);

        let _ = proxy.send_event(UserEvents::CloseWindow);
    };

    let _webview = WebViewBuilder::new(window)?
        .with_html(&html)?
        .with_initialization_script(RETURN_OUTER_HTML)
        .with_ipc_handler(handler)
        .build()?;

    event_loop.run(move |event, _, control_flow| {
        *control_flow = ControlFlow::Wait;

        match event {
            Event::WindowEvent {
                event: WindowEvent::CloseRequested,
                ..
            }
            | Event::UserEvent(UserEvents::CloseWindow) => {
                *control_flow = ControlFlow::Exit
            }
            _ => (),
        }
    });
}

Compile.

cargo build -r

It takes about one second to render HTML with JS and return back the HTML string. Only started checking the profiler.

time ./target/release/wry_test_render
0.17user 0.07system 0:00.81elapsed 30%CPU (0avgtext+0avgdata 80388maxresident)k
0inputs+0outputs (172major+7466minor)pagefaults 0swaps

Double-checking - source and resulting files are different.

du -sh source.html rendered.html 
688K	source.html
752K	rendered.html

[HaronK]

Interesting solution. Thanks.
Question to the authors of the library: is there a way to make it a WebView method (i.e. load_html) so I can call it instead of WebView::run and avoid those bunch of code above?
I can of course convert it into the function on my side but probably some other users would like to do the same.

[FabianLars]

not one of the core maintainers but this kind of helper function sounds like it's out of scope for wry to me.

[amrbashir]

I agree with @FabianLars it is out of wry scope.

[HaronK]

@amrbashir, @FabianLars Thanks for the answer.
I understand your reasons, it's just pity that it seems wry is the only cross platform library that can load dynamic pages (correct me if I'm wrong). Such a helper function would be nice to have ;)
In any case, thank you. Going to create my own wrapper.

[ilyazub]

Here's an updated code supporting wry v0.34.

use serde::{Deserialize, Serialize};
use serde_json;
use std::env;
use std::fs::{canonicalize, read_to_string};
use std::process::exit;
use wry::{
    application::{
        event::{Event, WindowEvent},
        event_loop::{ControlFlow, EventLoop},
        window::{Window, WindowBuilder},
    },
    webview::WebViewBuilder,
};

#[derive(Debug, Serialize, Deserialize)]
struct MessageParameters {
    message: String,
    data: String,
}

const RETURN_OUTER_HTML: &str = r#"
document.addEventListener("DOMContentLoaded", () => {
    let html = document.documentElement.outerHTML
    const message = JSON.stringify({ message: "dom-loaded", data: html })
    ipc.postMessage(message)
})
"#;

fn main() -> wry::Result<()> {
    let event_loop = EventLoop::new();

    let window = WindowBuilder::new()
        .with_visible(false)
        .build(&event_loop)?;

    let args: Vec<String> = env::args().collect();
    let file_path = &args[1];
    let html = read_to_string(canonicalize(file_path)?)?;

    let handler = move |_window: &Window, req: String| {
        let msg: MessageParameters = serde_json::from_str(&req).unwrap();

        if &msg.message == "dom-loaded" {
            println!("{}", &msg.data);
        }

        exit(0);
    };

    let _webview = Some(
        WebViewBuilder::new(window)?
            .with_html(&html)?
            .with_initialization_script(RETURN_OUTER_HTML)
            .with_ipc_handler(handler)
            .build()?,
    );

    event_loop.run(move |event, _, control_flow| {
        *control_flow = ControlFlow::Wait;

        match event {
            Event::WindowEvent {
                event: WindowEvent::CloseRequested,
                ..
            } => *control_flow = ControlFlow::Exit,

            _ => (),
        }
    })
}

Let's check how it works, using the Google Shopping results page (the HTML file is taken via the our Google Shopping Results scraper).

$ time cargo run -r ~/serpapi/tmp/big_shopping.html 2>/dev/null > ~/serpapi/tmp/big_shopping_rendered.html

real	0m1.193s
user	0m0.360s
sys	0m0.199s

The source and resulting HTML files differ.

$ du -sh ~/serpapi/tmp/big_shopping*.html
1.9M	/home/serpapi/tmp/big_shopping.html
2.4M	/home/serpapi/tmp/big_shopping_rendered.html

Comparing actual HTML.

In the original HTML (on the left), thumbnails are 1x1 placeholder pixels — data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==.

In the resulting HTML (on the right), thumbnails are actual images, that are JavaScript-rendered.

Closing thoughts

1. Now, it's interesting to compare the JS execution performance of wry and webview-based renderers (webview_deno or webview_ruby).

2. It's even more interesting to bundle a HTML prerenderer into a single binary with bundled shared libraries so it can be portable.

[marklonquist]

I still think WRY would be perfect for helping with "prerender" tasks. Current solutions involve using puppeteer and actually opening up a browser, navigate to the page, wait for navigation to finish (among other things), and then getting the content and returning it to the bots.

Seems it would perform way better with WRY if we can get it working correctly (eg. waiting for JS to finish loading completely)

[ilyazub]

Current solutions involve using puppeteer and actually opening up a browser, navigate to the page, wait for navigation to finish (among other things), and then getting the content and returning it to the bots.

The same happens with wry, except it uses webkit instead of chromium.

[marklonquist]

we should also remember, that performance is in play here, and from my local tests Wry seems faster than using chromium

[ilyazub]

we should also remember, that performance is in play here, and from my local tests Wry seems faster than using chromium

That's interesting. May you share the benchmark?

(I didn't notice the difference in the startup, JS execution, and rendering time, when compared wry (webkit) vs puppeteer (chromium).)