Fetching page as Googlebot displays an error

[DominikLevitsky]

So I'm currently using "next": "^4.1.4", on a nginx server on production. When trying to optimize the website's SEO, I discovered that crawling the pages as googlebot is causing some errors.

As you can see, on the right the page is rendered normally and on the left, some error was displayed. The page response returned looks good, though:

So I am wondering is this an issue with next.js itself or this may be caused by some other elements?

Tech	Version
next	4.1.4
node	6.10.2
OS	Ubuntu
browser	Googlebot

[brettgullan]

I've encountered the same problem when attempting to analyze a static export from Next.js using PageSpeed Insights.

[DominikLevitsky]

Also, if I am trying to copy and save all the code from the response (second screenshot) into an HTML file, the googlebot renders it absolutely fine. That is really weird.

[iamsoorena]

#3181
please read about this problem and solution here.

[DominikLevitsky]

@iamsoorena How to solve this with npm, though?

[iamsoorena]

the only solution for now :
#3181 (comment)

[iamsoorena]

probably you should use yarn as a workaround here, I know that it makes a lot of problems itself :D

[i8ramin]

I've tried the Yarn resolution suggestion, and that doesn't seem to do the trick.
https://developers.google.com/speed/pagespeed/insights/?url=https%3A%2F%2Fdeploy-preview-38--candidco.netlify.com%2F

Can't really tell if yarn is actually doing the right thing, as I still see 4.x versions of strip-ansi in there:
https://gist.github.com/i8ramin/e3d6c8addd934c17f91b05b64f3bd1e4

Any other suggestions or updates on this?

[arunoda]

Closing as we are looking it in a different place.

[DominikLevitsky]

What place?

[timneutkens]

@DominikLevitsky it's actually fixed in 4.2.1 👍

[DominikLevitsky]

@timneutkens Thank you!

[i8ramin]

FYI, the only way I was able to get things working on IE11 (and Google PageSpeed for some odd reason) was the following:

Added to package.json to lock a few modules. Main issue is ansi-regex. Version 3.x starts using arrow functions.

"resolutions": {
    "ansi-regex": "^2.0.0",
    "pretty-format": "20.0.1",
    "string-width": "2.0.0",
    "table": "3.7.9"
  }

Include this in page head
(polyfill service to include things like Object.assign and Array.from in IE11)

<script src="https://cdn.polyfill.io/v2/polyfill.min.js" defer async />

[DominikLevitsky]

I am still getting this error on next 4.3.0-canary.1 on production.

[sheerun]

I still get this issue for 4.3.0-canary.3 please reopen :(

[sheerun]

You can reproduce this issue by opening next.js webapp in nwjs 0.12 that uses the same engine as googlebot: https://dl.nwjs.io/v0.12.0/nwjs-v0.12.0-osx-x64.zip

[sheerun]

I've created https://github.com/sheerun/targets-webpack-plugin to target this issue

Maybe zeit would like to use it in next.js itself for "next build", hm? @timneutkens

[pyankoff]

Had issues with Googlebot / Chrome 41 rendering, because of using react-native-web and Apollo.
#3205 (comment) solution works great.

[Enalmada]

Is fetching page as googlebot issue still a problem in Next 7 for anyone? I am curious if most people probably have the issue and don't realize it or if there is some specific dependency that causes this that users could be warned about. Is the remaining cause something that could be solved by nextjs or is it recommended for everyone to use targets-webpack-plugin?

[sheerun]

I've re-checked my app that uses next 7.0.2 and I can confirm that google fails to render it:

https://imgur.com/a/AVhQFQ5

I cannot provide the source because it's proprietary. I'm pretty sure the error happens when imported code needs some native builtins that are not available in chrome 41. In this case next.js fails to include these in bundles. I think good candidate to check is includes:

https://caniuse.com/#search=includes

[sheerun]

Also it's the node_modules dependency that needs to use it, not the app itself

[sheerun]

@Enalmada @timneutkens I managed to reproduce this error with this repository that uses newest next.js: https://github.com/sheerun/with-apollo The app is available at https://sheerun-with-apollo.now.sh/ though you need to deploy it by yourself if you want to test it with https://www.google.com/webmasters/tools/googlebot-fetch-details

The error happens when app imports https://github.com/sheerun/awesome-includes package I've created that uses [].includes internally (there are other npm packages that do similar things).

Could you reopen this issue?

[pyankoff]

You can also test with Chrome 41 locally, that's what google uses.

[sheerun]

It's next to impossible to download one :)