Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't run on macs with M1 processors #686

Closed
kazazor opened this issue May 6, 2022 · 11 comments · Fixed by #692
Closed

Can't run on macs with M1 processors #686

kazazor opened this issue May 6, 2022 · 11 comments · Fixed by #692
Labels

Comments

@kazazor
Copy link
Contributor

kazazor commented May 6, 2022

This has been discussed here: puppeteer/puppeteer#6622 and here: puppeteer/puppeteer#6634

There are a few things that need to happen here:

  1. Change puppeteer to puppeteer-core
  2. Be able to pass executablePath to the package
@esakal
Copy link
Collaborator

esakal commented May 14, 2022

@kazazor Hi, we already support it. You can use the https://www.npmjs.com/package/israeli-bank-scrapers-core

Does it helps?

@kazazor
Copy link
Contributor Author

kazazor commented May 18, 2022

Awesome, I'll give it a try

@kazazor
Copy link
Contributor Author

kazazor commented Jun 6, 2022

@esakal maybe I'm missing something here.
In the Getting chromium version used by puppeteer-core section of israeli-bank-scrapers-core it says to:

  1. To get the required chromium revision use the following code
import { getPuppeteerConfig } from 'israeli-bank-scrapers-core';

const chromiumVersion = getPuppeteerConfig().chromiumRevision;
  1. Once you have the chromium revision, you can either download it manually or use other liraries like [download-chromium](https://www.npmjs.com/package/download-chromium) to fetch that version

  2. provide the path to chromium to the library using the option key executablePath

I did just that here:

  const options: ScaperOptions = {
    companyId: source,
    startDate,
    combineInstallments: false,
    showBrowser: false,
    executablePath: await getChromiumExecutablePath(), // Requires in order to support also ARM based CPUs (like Apple's M1 CPUs)
  };

  const scraper = createScraper(options);

And here is the getChromiumExecutablePath function:

async function getChromiumExecutablePath(): Promise<string> {
  const browserFetcher = puppeteer.createBrowserFetcher();
  const chromiumRevision = getPuppeteerConfig().chromiumRevision;
  let revisionInfo;

  if ((await browserFetcher.localRevisions()).includes(chromiumRevision)) {
    revisionInfo = browserFetcher.revisionInfo(chromiumRevision);
  } else {
    revisionInfo = await browserFetcher.download(chromiumRevision, (downloadBytes: number, totalBytes: number) =>
      console.log(`Downloading Chromium: ${downloadBytes}/${totalBytes} (${Math.round((downloadBytes / totalBytes) * 100)}%)`)
    );
  }

  return revisionInfo.executablePath;
}

I'm using israeli-bank-scrapers-core and puppeteer-core to make that work.

The problem here is that for M1 CPUs (arm64) I'm getting: The chromium binary is not available for arm64.

What am I missing here?

@esakal
Copy link
Collaborator

esakal commented Jun 8, 2022

when do you get this message, whwen trying to download? when executing scrape?

@kazazor
Copy link
Contributor Author

kazazor commented Jun 8, 2022

During the download. I found this thread online: https://stackoverflow.com/a/66044814

Which points to this fix here: puppeteer/puppeteer@9a8479a

In this release (6.0.0) : https://github.com/puppeteer/puppeteer/releases/tag/v6.0.0

support fetching and launching on Apple M1

Seems to me like updating puppeteer-core to use 6.0.0 may fix that.


Another indication this is the case is this issue: puppeteer/puppeteer#6614

Seems like executablePath is not being passed correctly using version 5.5.0.
I did the steps he suggested here: puppeteer/puppeteer#6614 (comment) and it worked!
So definitely something is wrong with puppeteer-core in version 5.5.0 in the manner of M1.

Looking at the breaking changes of version 6.0.0 seems to me like probably you can just update the version and it will work, right?

@kazazor
Copy link
Contributor Author

kazazor commented Jun 9, 2022

By the way, also updating up until version 7.1.0 seems easy enough. The only breaking change there is around page.screenshot which I don't believe is actually breaking anything for you here:

Even more, looking at the breaking changes all the way to the latest version 14.x.x seems simple enough and mainly renaming, maybe worth considering updating it

@esakal
Copy link
Collaborator

esakal commented Jun 17, 2022

@kazazor thank you for suggestion this fix. I ended up upgrading to v6 as more up-dated versions although works are breaking the typescript in many places.

As you mentioned the newer versions don't seems to have complicated breaking changes fixes but for a reason the @types/puppeteer does introduce many changes in versions later then v6 so as you stated that it is good enough I ended up upgrading to v6.

Before I merge & deploy, can you please pull and confirm the fix for the core version?

@kazazor
Copy link
Contributor Author

kazazor commented Jun 21, 2022

@esakal It seems to be doing the trick :)

@kazazor
Copy link
Contributor Author

kazazor commented Jun 21, 2022

By the way, would be great if in your merge and deploy you could already take this PR: #695

Which will solve #694

@github-actions
Copy link

🎉 This issue has been resolved in version 1.14.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

@vitalybe
Copy link

Is the latest commit should just work on M1?

When I executed yarn I am getting:

[2/3] ⠁ husky
error /Users/vitalyb/git/israeli-bank-scrapers/node_modules/puppeteer: Command failed.
Exit code: 1
Command: node install.js
Arguments:
Directory: /Users/vitalyb/git/israeli-bank-scrapers/node_modules/puppeteer
Output:
The chromium binary is not available for arm64.
If you are on Ubuntu, you can install with:

 sudo apt install chromium


 sudo apt install chromium-browser

/Users/vitalyb/git/israeli-bank-scrapers/node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserFetcher.js:115
                    throw new Error();
                    ^

Error
    at /Users/vitalyb/git/israeli-bank-scrapers/node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserFetcher.js:115:27

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants