Googlebot compatible protobuf ajax request

[robin-anil]

protobuf.js version: any

This is more of a question. I have noticed that Google bot is running some old janky or hacked up version of chrome browser and is unable crawl and parse our sites using protobuf.js. We try to work around it by using prerender.io service. But that to me is a crutch.

Our Ajax implementation is based on https://github.com/dcodeIO/protobuf.js/wiki/How-to-read-binary-data-in-the-browser-or-under-node.js%3F using whatwg-fetch This works in any browser we test except Google bot always have trouble with it. My guess is Google bot is pre-Chrome 26. Any ideas here?

[dcodeIO]

My first guess based on your assumptions would be that googlebot might not support ArrayBuffer / Uint8Array / responseType="arraybuffer" (properly). One way to work around it (or anything else not supporting one of these features) would be to use base64 encoded strings instead of binary data.

[dcodeIO]

This commit tries to add binary data support to protobuf.util.fetch through specifying an options parameter, like this:

protobuf.util.fetch("my/binary/file.bin", { binary: true }, function(err, data) {
   // with data being an Uint8Array if supported, otherwise an Array of octets
});

While I found a reference on how to prevent charset conversion on MDN, I have no idea if this actually works. The key here is to always return a reader/writer compatible Array while making sure that no utf8 whatsoever conversion is performed by ancient browsers without responseType="arraybuffer" support.

[dcodeIO]

This won't be perfect, yet (or ever), probably, but it's on npm as 6.6.2 now.

[robin-anil]

Yeah tried this. Does not work in Google crawler. I decided to do server side rendering of our app and give out HTML for the Google Crawler.

[robin-anil]

Server side rendering works nicely.

[dcodeIO]

Should also work with all sorts of crawlers then :)

[robin-anil]

Yeah. Probably that is the right recommendation, we have had a huge reduction in JS errors (almost flatlined) since crawlers started consuming html. I think this can be closed. The return on investment with weird js hacks for old browsers is very small.