Yarn 2: Support Yarn v2 "Berry"

[crubier]

Describe the bug

If you use Yarn v2, then you can't add Storybook to a simple Create React App, it crashes.

See yarnpkg/berry#592 (comment)

And the output here:

https://github.com/crubier/berry/commit/78acb633a0ad411ed173dae0a0037ab82fe346ad/checks?check_suite_id=407781418

To Reproduce

The following process is used in the workflow to reproduce this bug:

1. Install yarn v2: https://next.yarnpkg.com/getting-started/install
2. yarn dlx create-react-app my-cra && cd my-cra to init a crat react app
3. yarn dlx -p @storybook/cli sb init to init storybook, but it fails here
4. Then I would do yarn storybook and that would be it.... ideally..

Expected behavior

Success

Screenshots

Error: Cannot find module 'commander'
Require stack:
- /Users/vincent/.yarn/berry/cache/@storybook-cli-npm-5.3.6-c196adf18e-1.zip/node_modules/@storybook/cli/bin/generate.js
- /Users/vincent/.yarn/berry/cache/@storybook-cli-npm-5.3.6-c196adf18e-1.zip/node_modules/@storybook/cli/bin/index.js
Require stack:
- /Users/vincent/.yarn/berry/cache/@storybook-cli-npm-5.3.6-c196adf18e-1.zip/node_modules/@storybook/cli/bin/generate.js
- /Users/vincent/.yarn/berry/cache/@storybook-cli-npm-5.3.6-c196adf18e-1.zip/node_modules/@storybook/cli/bin/index.js
    at Proxy.module_1.Module._resolveFilename (/private/var/folders/2g/yhlpschd00n71jykx6bgbp1w0000gn/T/tmp-69185QMHt9oBf5169/dlx-69185/.pnp.js:18118:54)
    at Proxy.module_1.Module._load (/private/var/folders/2g/yhlpschd00n71jykx6bgbp1w0000gn/T/tmp-69185QMHt9oBf5169/dlx-69185/.pnp.js:18001:40)
    at Object.<anonymous> (/Users/vincent/.yarn/berry/cache/@storybook-cli-npm-5.3.6-c196adf18e-1.zip/node_modules/@storybook/cli/bin/generate.js:1)
    at Generator.next (<anonymous>)

System:

➜ npx -p @storybook/cli@next sb info

Environment Info:

  System:
    OS: macOS 10.15.2
    CPU: (12) x64 Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
  Binaries:
    Node: 10.15.3 - ~/.nvm/versions/node/v10.15.3/bin/node
    Yarn: 2.0.0-rc.23 - /usr/local/bin/yarn
    npm: 6.4.1 - ~/.nvm/versions/node/v10.15.3/bin/npm
  Browsers:
    Chrome: 79.0.3945.117
    Firefox: 71.0
    Safari: 13.0.4

Additional context
Trying to make Storybook work with yarn v2

[gaetanmaisse]

I will take a look at this one and try to make things work :)
@ndelangen @shilman do you know if another member already planned to work on Yarn v2 compatibility?

[ndelangen]

I was trigged to do it too, but I'm quite swamped; it'd be awesome if you could take this on!

[shilman]

@igor-dv you were looking at yarn-pnp at one point, weren't you?

[gaetanmaisse]

Current situation about the compatibility with Yarn 2 (a.k.a Berry)

@crubier created a repo explaining how to make SB works with Yarn Berry: https://github.com/crubier/yarnv2-storybook, the workaround is to:

• Unplug @storybook/core, for details see https://next.yarnpkg.com/cli/unplug -> yarn unplug @storybook/core
• Create a node_modules folder to be able to create SB Babel cache folder -> mkdir node_modules
• Create a preset to define a custom webpack config that uses Plug'n'Play resolver for Webpack.

Based on this repo and comments, multiple points need to be addressed to make SB works with Yarn Berry out of the box. There are my thoughts about that:

1. It looks like @storybook/core needs to be unplugged because it writes something inside itself. This is working with classical/legacy yarn because
   node_modules/@storybook/core is a folder with writable rights. In Yarn Berry, packages are read-only so no more writings allowed:

   Packages are now kept within their cache archives. For safety and to prevent cache corruptions, those archives are mounted as read-only drives and cannot be modified under normal circumstances

   So we need to find which part of @storybook/core is writing something and update this behavior. Sadly for us @storybook/core is (one of) the last packages still in JS.

2. As there is no node_modules folder anymore, we need to find another place to put SB Babel cache. The file managing this is https://github.com/storybookjs/storybook/blob/next/lib/core/src/server/common/babel-cache-preset.js and it should be pretty easy to fallback/move to another cache folder.
   However, I didn't find any wide used convention for where should be the cache folder for now (some package like Babel have a fallback to os.tmp(), webpack has changed it here...), so I think we need to discuss that.

3. Update the base webpack config to include Plug'n'Play resolver if the user is using Yarn Berry (in a more global point of view is using Plug'n'Play i.e. there is a pnp.js file in its project)

   OR

   Create a preset to handle this out of @storybook/core as it is affecting only a part of SB users. Something like what @igor-dv and @crubier started there: Pnp continue presets#18 (which is only working for TS users).

   My personal opinion is that we should consider adding the Plug'n'Play resolver in SB core as Yarn Berry is now the "stable" version (yarn as we are used to is now legacy). More and more SB users will be using Yarn Berry and SB should work with it without any additional setup.
   According to Yarn 2 support nodejs/docker-node#1180, Yarn Berry will/should be the default version ship with Node 14 Docker Image.

Here is a link of Introducing Yarn 2 article written by its main maintainer.

Any feedback on all this is welcome! @crubier @igor-dv @ndelangen @shilman and anyone else :)

[shilman]

@gaetanmaisse Thanks for this detailed explanation and for taking this on. You're awesome!!

One extra bit of information: @tmeasday was recently talking to me about wanting to rewrite story_store, client_api, and some other key files in lib/core. I wonder if it might be the right time to port it to typescript and fix whatever is causing it to write itself. WDYT?

[gaetanmaisse]

Are you talking about the usages of story_store and client_api in core? because they are already written in TS in lib/client-api.

I just investigated a bit more and the answer to "What core does write inside itself?" was... 🥁 ready? 🥁 ... the bundle generated by webpack 😄 ! See there, with no custom outputDir we are using ../public folder and so it ends in node_modules/@storybook/core/dist/public. So the solution to this will be to use a folder outside node_modules. One more folder to discuss (+ cache folder).

About "3. Update the base webpack config to include Plug'n'Play", it looks like Plug'n'Play will be supported out of the box by Webpack 5. But it is still in beta for now!
-- Note for the future, there is a constant to check if the user is running Pnp, can be useful if we have to add the Plug'n'Play resolver in Webpack SB base config.

In the next days, I will try to open a PR with as few changes as possible to make this works, this way we will have some code to discuss. 😉

[shilman]

LOL I guess rewriting that wouldn't help things then! 🙈 But I think the rewrite was meant to extend to at least some of lib/core, including start.js, which is one of the more complex files in our codebase.

Sounds like you don't need any of that to move forward. Thanks for leading the charge here!

[ndelangen]

We can move the cache dir to <cwd>/.cache/storybook (creating the folder if not present) for both the babel cache and the output dir for webpack.

That would solve these issues?

[arcanis]

3. Update the base webpack config to include Plug'n'Play resolver if the user is using Yarn Berry (in a more global point of view is using Plug'n'Play i.e. there is a pnp.js file in its project)

Note that you don't need to check whether the user is running under Plug'n'Play, as the plugin already does and is a noop for other environments 🙂This is for example what Gatsby and Next do.

[gaetanmaisse]

With Storybook 6.0.0-alpha.6+ SB & Yarn 2 are working together for simple config, as in repro repo provided by @crubier 🎉

I still need to work on:

• Making yarn dlx -p @storybook/cli sb init works, to have out of the box support:
  • PR: CLI: transpile @storybook/cli to CJS #9807
  • PR: CLI: Add Yarn 2 compatibility #9866
• Updating @storybook/preset-create-react-app. it's currently resolving react-script in node_modules/.bin/ directory, so I need to add one more check for SB PnP user and access webpack config file directly in the .zip file with PnP API.
  • PR: Make preset-create-react-app compatible with Yarn PnP/Yarn 2 presets#104
• Checking that all addons are working with the latest alpha of SB 6:
  • PR : Yarn 2: Fix dependencies issues for compatibility #10613

    - ✅ A11y
    - ✅ Actions
    - ✅ Backgrounds
    - ✅ CSS Resources
    - ✅ Design Assets
    - 🚧 Docs
    - 🚧 Essential
    - ✅ Events
    - ✅ Google Analytics
    - ✅ GraphQL
    - ✅ Jest
    - ✅ Knobs
    - ✅ Links
    - ✅ Query Params
    - 🤷‍♂ Storyshots
    - ✅ Storysource
    - ✅ Toolbars
    - ✅ Viewport

• Finding a way of adding examples with Yarn 2 inside SB monorepo. Maybe creating some GitHub actions as it is done on Yarn 2 repo can be a good idea because it really mimics the users' behaviors: https://github.com/yarnpkg/berry/blob/master/.github/workflows/e2e-jest-workflow.yml

  • PR: CLI: Add Yarn 2 compatibility #9866
  • Yarn 2 will be fully integrated into the new E2E tests: Tech/ci examples overhaul #10447

• Handle .storybook/preview.js file properly . See Yarn 2: SB not working when using a .storybook/preview.js file #10094

  • PR: Yarn 2: rework imports in webpack preview virtual module to fix compatibility #10305

@ndelangen For the cache folder, I put it in node_modules/.cache/@storybook for now. An enhancement issue has been created by @arcanis in Yarn Org yarnpkg/berry#918 to discuss the usefulness, pros, and cons of creating and maintaining a cache pkg in the Yarn org. If it ends on the creation of a such package we will surely use it instead of our custom resolve-path-in-sb-cache.js.

[shilman]

Great job @gaetanmaisse !!! 💯

[missing1984]

Love the efforts here! I ran into #7445 in yarn pnp. Not sure if someone caught this before.

Looks like corejs-upgrade-webpack-plugin is not compatible with yarn pnp. Maybe we need to disable it in pnp mode?

[kubijo]

With Storybook 6.0.0-alpha.6+ SB & Yarn 2 are working together for simple config, as in repro repo provided by @crubier

Hi.

Thanks for your great efforts!
I've managed to get our storybook instance to run with alpha.33 after configs migration & I've also had to unplug @storybook/client-api since otherwise it wouldn't be resolved from somewhere deep.

Here's the error message:

ERROR in ./conf/storybook/preview.js-generated-config-entry.js
Module not found: Error: Can't resolve '…/.yarn/$virtual/@storybook-client-api-virtual-4ff893b6d5/0/cache/@storybook-client-api-npm-6.0.0-alpha.33-bfa9fe6573-2.zip/node_modules/@storybook/client-api/dist/index.js' in '…/conf/storybook'
 @ ./conf/storybook/preview.js-generated-config-entry.js 52:15-225
 @ multi ./.yarn/$$virtual/@storybook-core-virtual-2edb463070/0/cache/@storybook-core-npm-6.0.0-alpha.33-85dfb5f3b0-2.zip/node_modules/@storybook/core/dist/server/common/polyfills.js ./.yarn/$$virtual/@storybook-core-virtual-2edb463070/0/cache/@storybook-core-npm-6.0.0-alpha.33-85dfb5f3b0-2.zip/node_modules/@storybook/core/dist/server/preview/globals.js ./conf/storybook/storybook-init-framework-entry.js ./conf/storybook/preview.js-generated-config-entry.js ./conf/storybook/generated-stories-entry.js ./.yarn/cache/webpack-hot-middleware-npm-2.25.0-b59f3763d7-2.zip/node_modules/webpack-hot-middleware/client.js?reload=true&quiet=false

Also I get this one (don't mind that): Could not open http://localhost:6006/ inside a browser. If you're running this command inside a docker container or on a CI, you need to pass the '--ci' flag to prevent opening a browser by default.

[gaetanmaisse]

@kubijo thanks for your feedback! The issue you're facing will be fixed with #10342, I would like to merge it ASAP to have the fix included in the next alpha (in the next few days 🤞).

[gaetanmaisse]

@kubijo alpha.35 is out and should fix many of the Module not found errors. 🚀
Can you give it a try and check if you still face issues with Yarn 2?

[kubijo]

@gaetanmaisse alpha.34 indeed seems to work 👍

[DimitrK]

@gaetanmaisse great news, alpha.35 works for me. Still, alpha.34 @kubijo is mentioning unfortunately didn't work for me.

[kubijo]

I might have been a little to quick as 35 didn't resolve ... So I thought that it's a typo, but maybe I just raced CDNs

edit: nevermind, I think that just IDE didn't see it because it is running from a.35

[shilman]

Good golly!! I just released https://github.com/storybookjs/storybook/releases/tag/v6.0.0-beta.1 containing PR #10613 that references this issue. Upgrade today to try it out!

You can find this prerelease on the @next NPM tag.

Closing this issue. Please re-open if you think there's still more to do.

[jon301]

Hi Storybook Team !
I'm currently trying the latest version of Storybook (6.0.28) with Yarn Berry, and I'm encountering the following issues.

Here are the steps I followed:
yarn dlx -p @storybook/cli sb init
yarn storybook

core-js is not found

ERROR in ./.storybook/preview.js-generated-config-entry.js
Module not found: Error: Your application tried to access core-js, but it isn't declared in your dependencies; this makes the require call ambiguous and unsound.

Required package: core-js (via "core-js/modules/web.dom-collections.for-each")
Required by: /Users/jonjon/work/onitech-mono/packages/libs/my-lib1/.storybook/preview.js-generated-config-entry.js

...

WARN Broken build, fix the error above.
WARN You may need to refresh the browser.

Essential Addon not working

After adding core-js manually to dev dependencies, Storybook starts correctly but the Essentials Addon does not work (there's no Addon panel in the UI, even if Show addons is checked)

There's also an error in the console:

info => Loading presets
WARN   Failed to load preset: {"name":"@storybook/addon-essentials","type":"presets"} on level 1
ERR! Error: @storybook/core tried to access @storybook/addon-essentials, but it isn't declared in its dependencies; this makes the require call ambiguous and unsound.
ERR!
ERR! Required package: @storybook/addon-essentials (via "@storybook/addon-essentials")
ERR! Required by: @storybook/core@virtual:3ced7c9a1cb7b9c553fcb23a084cf86bc2b7b3c18fb1453f5d7e4e78ef02cf878b49023eb9dceaf9c0ff10242ebfc55e1b462bbf81d54d60fb71b1a18460aa06#npm:6.0.28 (via /Users/jonjon/work/onitech-mono/.yarn/$$virtual/@storybook-core-virtual-5e0cc5b3d0/0/cache/@storybook-core-npm-6.0.28-47dac5a288-ee78dbe2de.zip/node_modules/@storybook/core/dist/server/)

Not sure what to do here as@storybook/addon-essentials is installed correctly :(

Warning in Introduction.stories.mdx

WARNING in ./src/stories/Introduction.stories.mdx 11:0
Module parse failed: Unexpected token (11:0)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
| import StackAlt from './assets/stackalt.svg';
|
> <Meta title="Example/Introduction" />
|
| <style>{`
 @ \.)(?=.)[^/]*?\.stories\.mdx)$ (./src sync ^\.(?:(?:^|\/|(?:(?:(?!(?:^|\/)\.).)*?)\/)(?!\.)(?=.)[^/]*?\.stories\.mdx)$) ./stories/Introduction.stories.mdx
 @ ./.storybook/generated-stories-entry.js

[shilman]

cc @gaetanmaisse @merceyz

[jon301]

For info, I digged a bit and created a repro case with some useful information.
Thank you for your help.

https://github.com/jon301/storybook-yarn-berry-monorepo

[merceyz]

Fixed in #11753 which was released in v6.1.0-alpha.8

[jon301]

Thanks @merceyz for the quick answer !

Updating to 6.1.0-alpha.34 has made errors disappear for the basic Storybook configuration.
However when adding a custom webpack config with SASS support, new errors appear:

Module not found: Error: @storybook/core tried to access sass-loader, but it isn't declared in its dependencies; this makes the require call ambiguous and unsound.

Required package: sass-loader (via "sass-loader")
Required by: @storybook/core@npm:6.1.0-alpha.34 (via /Users/jonjon/tmp/storybook-yarn-berry-monorepo/.yarn/cache/@storybook-core-npm-6.1.0-alpha.34-a93cd16488-2d6e16aa2a.zip/node_modules/@storybook/core/)

I've updated the repro case : https://github.com/jon301/storybook-yarn-berry-monorepo

[merceyz]

You need to require.resolve those loaders, otherwise you're telling storybook to resolve them for you, which it can't since it doesn't declare them. Or you can use the scss preset that does this for you https://github.com/storybookjs/presets/tree/master/packages/preset-scss