[WIP]: webpack 4 support #106

mhheise · 2018-02-23T22:34:11Z

WIP: https://github.com/mhheise/autodll-webpack-plugin/tree/feat/webpack-4

Do not merge yet, please!

I would appreciate any feedback you may have 😄

Todo:

Upgrade to webpack 4 and other related dependencies across package.json files.
webpack
webpack-cli (now required to run webpack from the command line)
webpack-dev-server
extract-text-webpack-plugin (now a dependency of html-webpack-plugin)
html-webpack-plugin
Use compiler.hooks.<identifier>.tap('name', step) instead of compiler.plugin(<identifier>, step).
Use compiler.hooks feature detection to support webpack 3 and webpack 4.
Use Set() for compilationDependencies (still need to make this backwards compatible).
Implement new mode configuration property in webpack configs.
Update tests to use the new webpack 4 strategy (and perhaps test backwards compatibility to webpack 3).
Extra credit: Update examples to babel 7 and react 16.

Blockers:

webpack-contrib/html-webpack-plugin#3 stable release

…use a Set()

asfktz · 2018-02-27T06:26:25Z

Hi again, @mhheise!
First, it's really cool what's you're doing here, and much appreciated 😊

About the file you mentioned:

'C:\Git\autodll-webpack-plugin\examples\recommended\node_modules\.cache\autodll-webpack-plugin\development_instance_0_1dc0c813842ea3b964c005e53b6b3ea3\vendor.manifest.json'

This file, generated by the plugin, or the be more accureate, by the DllPlugin, here:
https://github.com/mhheise/autodll-webpack-plugin/blob/68594cb2e689d7116bd7e7c669a40e69f5a91a36/src/createConfig.js#L45

In the end, this plugin is just an abstraction for the DllPlugin & DllReferencePlugin plugin.

I've noticed that it complains about:
TypeError: Cannot read property 'compilation' of undefined

This happens when it calls new DllReferencePlugin(...).apply(compiler)
where the manifestPath file, is the file you mentioned.

Object.keys(dllConfig.entry)
      .map(getManifestPath(settings.hash))
      .forEach(manifestPath => {
        new DllReferencePlugin({
          context: context,
          manifest: manifestPath, 
        }).apply(compiler);
      });

That's very interesting... you see, for the DLL to work you have to add the DllReferencePlugin plugin to the user's config (That is how the original DllPlugin work)

At that point in the AutoDLL plugin's life cycle, the manifest file does not generated yet.
But, at least for webpack 3, the only way the add additional plugins to the original config was to add them at that stage before any lifecycle hook is triggered.

The original DllReferencePlugin had no trouble referencing a manifest file does not exist yet.
But it seems like it does now /:

Btw, here a little tip, if you want to debug the plugin with devtools, you can:

Open a terminal tab
Go to the plugins directory and run: npm run build:watch
Open another terminal tab
Go to the ./examples/recommended and run node --inspect --inspect-brk node_modules/.bin/webpack
Open chrome, and go to chrome://inspect
Click Open dedicated DevTools for Node

And you should be able to debug the plugin now (:

Also, you can install NIM chrome extensions which will popup devtools for you automaticity

asfktz · 2018-02-27T06:28:18Z

src/plugin.js

-      );
+    const beforeCompile = params => {
+      const dependencies = new Set(params.compilationDependencies);
+      [...dependencies].filter(path => !path.startsWith(cacheDir));


I think you missed a return there (:

mhheise · 2018-03-05T17:09:46Z

@asfktz: Thank you so much for the detailed review! I think I am stuck at this point -- I am not sure where and how I would ensure that the manifest file is generated.

EDIT: Perhaps we need an enhancement in webpack to add a hook to the DLL and DLL reference plugins, like what is present in HTML plugin, to indicate when the manifest has been generated?

asfktz · 2018-03-06T14:34:37Z

EDIT: Perhaps we need an enhancement in webpack to add a hook to the DLL and DLL reference plugins, like what is present in HTML plugin, to indicate when the manifest has been generated?

Hmm.. not sure I understand, can you show me an example? thanks!

mhheise · 2018-03-06T19:24:49Z

Something like this: https://github.com/webpack-contrib/html-webpack-plugin#events

asfktz · 2018-03-14T20:07:45Z

Hi @mhheise! Sorry for the delay, I got busy lately.
Lets make it happen 😄

I've played around with your code to see what I can help with.
The good news it that it works! I really didn't do much. its your magic.
The only thing I did was to comment out html-webpack-plugin on the user side.
Seems like it's not working properly.

Here the changes I made: b2db631...mhheise-feat/webpack-4

The interesting parts are examples/recommended/webpack.config.js and src/plugin.js

running npm run build and npm start in the recommended works.
The DLL was built, but no index.html of course.

I haven't got the time to dig into webpack 4 yet,
but I read something about html-webpack-plugin is not compatible with it?

mhheise · 2018-03-15T17:24:59Z

@asfktz Thank you!

html-webpack-plugin is now compatible with webpack@4 (the latest version at this time is 3.0.6, with webpack@4 supported in 3.0.0 and greater). The reason I originally used webpack-contrib/html-webpack-plugin was that the webpack-contrib organization forked html-webpack-plugin until the maintainer returned and the support could be merged back into the upstream repository.

I think I have most of the webpack@4 support complete, but the tests are timing out. I'm pretty sure I'm missing something but I haven't been able to figure out what.

I apologize for all of the back-and-forth on this PR -- I'm still new to JS and to open-source contributions! I do appreciate all of the help you've provided so far.

emmenko · 2018-03-16T13:57:16Z

Cool! Looking forward to seeing this working. If it's ready to be tested I can try it out on my project before we merge this.

asfktz · 2018-03-17T19:17:08Z

I apologize for all of the back-and-forth on this PR -- I'm still new to JS and to open-source contributions! I do appreciate all of the help you've provided so far.

No worries, @mhheise! we're all here to learn 😄

I've found a bug which may relate to the tests being timed out.

The plugin exposes a hook called autodll-stats-retrieved
It used mostly for testing purposes.

I've noticed that while running the recommended example, it throws:

➜  recommended git:(pr/106) ✗ npm run build

> basic@1.0.0 build /Users/asafkatz/dev/autodll-webpack-plugin/examples/recommended
> webpack

(node:36787) DeprecationWarning: Tapable.plugin is deprecated. Use new API on `.hooks` instead
AutoDllPlugin: is valid cache? false
AutoDllPlugin: cleanup
AutoDllPlugin: compile
TypeError: compiler.applyPlugins is not a function
    at /Users/asafkatz/dev/autodll-webpack-plugin/lib/plugin.js:143:20
    at <anonymous>

Which I believe related to those lines:

if (compiler.hooks) {
  compiler.hooks.autodllStatsRetrieved = new SyncHook(['stats', 'source']);
  compiler.hooks.autodllStatsRetrieved.call(stats, source);
} else {
  compiler.applyPlugins('autodll-stats-retrieved', stats, source);
}

Steps to reproduce:

Navigate to the recommended example in the console cd ./examples/recommended
Make sure the cache is empty by running: rm -rf node_modules/.cache
run npm run build

You should see the above error.

asfktz · 2018-03-17T19:22:16Z

Hi, @emmenko. Thanks!
Your help is very welcome 🙏

We'll let you know when its ready.

akuznetsov-gridgain · 2018-03-22T07:09:05Z

Will autodll bring some speedup for webpack4?
Assuming wp4 has built in chunks?

emmenko · 2018-04-03T19:18:24Z

Will autodll bring some speedup for webpack4

It would be good to find out yes 👌

faceyspacey · 2018-04-05T17:20:05Z

TypeError: compiler.applyPlugins is not a function doesn't occur for me. watchRun successfully checks if (compiler.hooks) and uses the new api instead of compiler.applyPlugins.

The whole thing seems to be working great, except html-webpack-plugin, which I personally don't need. If that's the only remaining blocker, I'll look into that and see what's needed there.

the html webpack plugin now works

faceyspacey · 2018-04-05T18:04:59Z

ok, i got the html-webpack-plugin to work via 2 small changes. Here's a PR to @mhheise 's original PR.

https://github.com/mhheise/autodll-webpack-plugin/pull/1

...for me the integration tests hang on testing usage with webpack-dev-server. I get this far:

> basic@1.0.0 test /Users/jamesgillmore/codementor/autodll/specs/fixtures/basic
> ava --serial specs/*.integration.js

Detect package.json change
Ensure stats retrieved from the currect source
clean run (cache deleted)
Starting the development server...

AutoDllPlugin: is valid cache? false
AutoDllPlugin: cleanup
AutoDllPlugin: compile

doesn't go any farther than compile

The fixtures have been installed etc. Perhaps the integration helpers have to be updated.

faceyspacey · 2018-04-05T18:13:56Z

Here it is temporarily if anyone out there wants to give it a try without jumping through any hoops:

https://www.npmjs.com/package/autodll-webpack-plugin-webpack-4
yarn add --dev autodll-webpack-plugin-webpack-4

zhbhun · 2018-04-09T10:30:07Z

@faceyspacey HTML plugin hooks should tap after compiler's afterPlugins

refer to other html plugin https://github.com/waysact/webpack-subresource-integrity/blob/master/index.js#L338

Venryx · 2018-04-23T14:57:15Z

Just wanted to mention this plugin looks nice, but I'm using Webpack 4 so hesitant to try until the official version supports it. (thanks to @faceyspacey though for making most of the code changes needed!)

waterfoul · 2018-04-25T15:49:57Z

@mhheise What are the chances you merge @faceyspacey's work?

fix($htmlWebpackPlugin): add run hook and use tapAsync for doCompilation

mhheise · 2018-04-28T19:53:08Z

Thanks @faceyspacey. PR has been merged.

pleunv · 2018-05-02T08:17:53Z

Here it is temporarily if anyone out there wants to give it a try without jumping through any hoops:
https://www.npmjs.com/package/autodll-webpack-plugin-webpack-4
yarn add --dev autodll-webpack-plugin-webpack-4

Gave this one a try but I'm running into a couple fs-related errors:

× ｢wdm｣: Error: ENOENT: no such file or directory, open 'c:\<project-dir>\node_modules\.cache\autodll-webpack-plugin\development_instance_0_5a54a620c19768a5faf5a15e78daf557\vendor.manifest.json'
AutoDllPlugin: is valid cache? false
AutoDllPlugin: cleanup
AutoDllPlugin: compile
{ Error: ENOENT: no such file or directory, open 'c:\<project-dir>\node_modules\.cache\autodll-webpack-plugin\development_instance_0_5a54a620c19768a5faf5a15e78daf557\stats.json'
    at c:\<project-dir>\node_modules\autodll-webpack-plugin-webpack-4\lib\createHandleStats.js:19:15
    at c:\<project-dir>\node_modules\autodll-webpack-plugin-webpack-4\lib\createHandleStats.js:56:16
    at <anonymous>
  cause:
   { Error: ENOENT: no such file or directory, open 'c:\<project-dir>\node_modules\.cache\autodll-webpack-plugin\development_instance_0_5a54a620c19768a5faf5a15e78daf557\stats.json'
     errno: -4058,
     code: 'ENOENT',
     syscall: 'open',
     path: 'c:\\<project-dir>\\node_modules\\.cache\\autodll-webpack-plugin\\development_instance_0_5a54a620c19768a5faf5a15e78daf557\\stats.json' },
  isOperational: true,
  errno: -4058,
  code: 'ENOENT',
  syscall: 'open',
  path: 'c:\\<project-dir>\\node_modules\\.cache\\autodll-webpack-plugin\\development_instance_0_5a54a620c19768a5faf5a15e78daf557\\stats.json' }

zhbhun · 2018-05-05T11:53:32Z

Base on autodll-webpack-plugin-webpack-4, I got it work with html-webpack-plugin and fix some internal erros.

https://www.npmjs.com/package/autodll-webpack4-plugin

https://github.com/zhbhun/autodll-webpack-plugin

klassicd · 2018-05-05T22:54:08Z

@zhbhun I'm getting ReferenceError: vendor_b9e331f1a5e536a9a124 is not defined.

zhbhun · 2018-05-06T05:04:50Z

@klassicd I cannot reproduce the problem, you could try basic example https://github.com/zhbhun/autodll-webpack-plugin/tree/master/examples/basic.

Vincz · 2018-05-23T01:25:38Z

Hey guys! Any update on this one?

Vincz · 2018-05-23T06:52:48Z

Hey @zhbhun, I tried your repo with all the example. It's working fine (except 'performance' and 'recommanded' because of a repository not found error (https://github.com/webpack-contrib/html-webpack-plugin doesn't exist).

I tried in my own project, and like @pleunv , with your repo or @faceyspacey one, I get errors like that:

{ Error: ENOENT: no such file or directory, open '/Volumes/xxx/node_modules/.cache/autodll-webpack-plugin/development_instance_0_82e345bb31c079df964161da1db1abc7/stats.json'

Hope it'll help!

Vincz · 2018-05-23T07:52:14Z

Ok, I found the problem. The problem occurs when dll compile failed. One of the package was missing so the build failed, and the cache was not generated.

zhbhun · 2018-05-24T01:49:20Z

Oh, dll build does't check compilation error.

https://github.com/asfktz/autodll-webpack-plugin/blob/master/src/createCompileIfNeeded.js#L23

if (err) return reject(err);
if (stats.compilation.errors.length) return reject(stats.compilation.errors);

vovakulikov · 2018-05-26T19:38:07Z

Hey guys! Any update on this one? 😿

asfktz · 2018-05-30T16:56:34Z

I apologize for the long wait.

@mhheise & @faceyspacey, thank you so much for your hard work. It is much appreciated 🙏

I'll take it from here.
In the next few days I'll merge this branch and release a new version to support webpack 4.

asfktz · 2018-06-02T13:36:16Z

Done (:

sudkumar · 2018-06-04T07:01:57Z

Waited for a long time. Thank you. 👍

mhheise · 2018-06-06T20:14:17Z

@asfktz @faceyspacey Thank you for your help!

And thank you to all who tested this out and waited patiently 👍

mhheise added 3 commits February 22, 2018 21:08

feat: first draft

ae3686e

fix(plugin): remove callbacks and migrate compilationDependencies to …

b40e851

…use a Set()

fix(examples): upgrade to webpack 4

a0a1aa7

tpai mentioned this pull request Feb 25, 2018

Revert "Update webpack to the latest version 🚀" tpai/r2-starter-kit#40

Merged

taneliang mentioned this pull request Feb 26, 2018

Webpack 4 plugin support tracker nusmodifications/nusmods#874

Closed

mhheise added 2 commits February 26, 2018 12:21

chore: upgrade to webpack 4 and webpack-dev-server 3 stable

ce714b0

fix: explicitly set peerDependencies to supported webpack versions

68594cb

asfktz reviewed Feb 27, 2018

View reviewed changes

DorianGrey mentioned this pull request Mar 1, 2018

[W.I.P.][Migration] Webpack 4 - detecting outstanding issues. DorianGrey/vue-ts-playground#6

Merged

7 tasks

mhheise added 2 commits March 5, 2018 10:51

fix: reinstate callbacks and use async hooks

50109ab

[WIP] test: update tests to webpack 4

b2db631

mhheise added 3 commits March 14, 2018 22:18

fix: add from tinkering

4cf1d3c

fix: update html-plugin, remove extract-text-plugin

b1968d8

fix: try backwards compatibility in tests

9aea58b

fix: ensure html-webpack-plugin is before autodll-webpack-plugin

e817b6c

fix($htmlWebpackPlugin): add run hook and use tapAsync for doCompilation

3ea5b9c

the html webpack plugin now works

faceyspacey mentioned this pull request Apr 5, 2018

Webpack 4 support #105

Closed

faceyspacey added 5 commits April 5, 2018 11:33

add mode to examples/recommended

e5f869f

add mode conditionally to examples/performance

3b1b2e8

update html-webpack-plugin dep in examples/recommended

7e65e4f

update html-webpack-plugin dep in examples/performance

d5f29a4

fix($travis): remove node_js 4 as it's no longer supported.

d1d856a

javaLuo mentioned this pull request Apr 24, 2018

可以加入dll打包吗 javaLuo/react-luo#18

Closed

Merge pull request #1 from faceyspacey/patch-1

565040d

fix($htmlWebpackPlugin): add run hook and use tapAsync for doCompilation

marcofugaro mentioned this pull request May 21, 2018

Webpack 4 followup facebook/create-react-app#4492

Closed

3 tasks

asfktz merged commit 565040d into asfktz:master Jun 2, 2018

mhheise deleted the feat/webpack-4 branch June 6, 2018 20:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP]: webpack 4 support #106

[WIP]: webpack 4 support #106

mhheise commented Feb 23, 2018

asfktz commented Feb 27, 2018

asfktz Feb 27, 2018

mhheise commented Mar 5, 2018 •

edited

Loading

asfktz commented Mar 6, 2018

mhheise commented Mar 6, 2018

asfktz commented Mar 14, 2018 •

edited

Loading

mhheise commented Mar 15, 2018

emmenko commented Mar 16, 2018

asfktz commented Mar 17, 2018

asfktz commented Mar 17, 2018

akuznetsov-gridgain commented Mar 22, 2018

emmenko commented Apr 3, 2018 •

edited

Loading

faceyspacey commented Apr 5, 2018

faceyspacey commented Apr 5, 2018 •

edited

Loading

faceyspacey commented Apr 5, 2018

zhbhun commented Apr 9, 2018 •

edited

Loading

Venryx commented Apr 23, 2018 •

edited

Loading

waterfoul commented Apr 25, 2018

mhheise commented Apr 28, 2018

pleunv commented May 2, 2018 •

edited

Loading

zhbhun commented May 5, 2018

klassicd commented May 5, 2018

zhbhun commented May 6, 2018

Vincz commented May 23, 2018

Vincz commented May 23, 2018

Vincz commented May 23, 2018

zhbhun commented May 24, 2018

vovakulikov commented May 26, 2018

asfktz commented May 30, 2018 •

edited

Loading

asfktz commented Jun 2, 2018

sudkumar commented Jun 4, 2018

mhheise commented Jun 6, 2018

[WIP]: webpack 4 support #106

[WIP]: webpack 4 support #106

Conversation

mhheise commented Feb 23, 2018

Todo:

Blockers:

asfktz commented Feb 27, 2018

asfktz Feb 27, 2018

Choose a reason for hiding this comment

mhheise commented Mar 5, 2018 • edited Loading

asfktz commented Mar 6, 2018

mhheise commented Mar 6, 2018

asfktz commented Mar 14, 2018 • edited Loading

mhheise commented Mar 15, 2018

emmenko commented Mar 16, 2018

asfktz commented Mar 17, 2018

asfktz commented Mar 17, 2018

akuznetsov-gridgain commented Mar 22, 2018

emmenko commented Apr 3, 2018 • edited Loading

faceyspacey commented Apr 5, 2018

faceyspacey commented Apr 5, 2018 • edited Loading

faceyspacey commented Apr 5, 2018

zhbhun commented Apr 9, 2018 • edited Loading

Venryx commented Apr 23, 2018 • edited Loading

waterfoul commented Apr 25, 2018

mhheise commented Apr 28, 2018

pleunv commented May 2, 2018 • edited Loading

zhbhun commented May 5, 2018

klassicd commented May 5, 2018

zhbhun commented May 6, 2018

Vincz commented May 23, 2018

Vincz commented May 23, 2018

Vincz commented May 23, 2018

zhbhun commented May 24, 2018

vovakulikov commented May 26, 2018

asfktz commented May 30, 2018 • edited Loading

asfktz commented Jun 2, 2018

sudkumar commented Jun 4, 2018

mhheise commented Jun 6, 2018

mhheise commented Mar 5, 2018 •

edited

Loading

asfktz commented Mar 14, 2018 •

edited

Loading

emmenko commented Apr 3, 2018 •

edited

Loading

faceyspacey commented Apr 5, 2018 •

edited

Loading

zhbhun commented Apr 9, 2018 •

edited

Loading

Venryx commented Apr 23, 2018 •

edited

Loading

pleunv commented May 2, 2018 •

edited

Loading

asfktz commented May 30, 2018 •

edited

Loading