"Error: EMFILE: too many open files, open.." on Windows

[TimNZ]

That's pretty much the issue.
Works fine on OS X

Have googled and can't find any way to increase Windows open file limit.
Is now also happening on OS X 'ENFILE: file table overflow'

As far as I can tell this module always works down to an individual file level in async.

I've worked around this by changing the app structure (it will be the ~8000 files in node_modules) and doing a shell cp instead.

Is it something you need to look at handling via something like graceful-fs?

[qiaolb]

I have a same problem.

[kevlened]

This is tricky, because node-glob no longer uses graceful-fs and just including it before node-glob won't work because graceful-fs has stopped monkey-patching fs. The graceful-fs docs say not to patch fs in a library (like this one), so it'd have to be added as an option. The problem is, the option would have to be checked before any other module loaded fs. An environment variable is the only reasonable way I can think to do that, which seems sloppy.

If you're running into this issue, you'll need to monkey-patch fs yourself. Here's how:

// At the top of your webpack config, insert this
var fs = require('fs');
var gracefulFs = require('graceful-fs');
gracefulFs.gracefulify(fs);

I'd add this info to the docs.

[TimNZ]

The irony of me suggesting graceful-fs based on comments I real elsewhere for different issues and then never looking at the module myself and seeing that it would help :)

[kevlened]

No worries! Others had the same issue, so I'm glad it's in the docs now.

[keyiis]

@kevlened i have same issues,i already monkey-patch fs,below is my code:

var fs = require('fs');
var gracefulFs = require('graceful-fs');
gracefulFs.gracefulify(fs);
const webpackMerge = require('webpack-merge');
const ExtractTextPlugin = require('extract-text-webpack-plugin');
const CopyWebpackPlugin = require('copy-webpack-plugin');
const commonConfig = require('./webpack.common.js');
const helpers = require('./helpers');

module.exports = webpackMerge(commonConfig, {
  ......
plugins: [
    new CopyWebpackPlugin([
      { from: helpers.root('node_modules/ckeditor/ckeditor.js'), to: 'node_modules/ckeditor' },
      { from: helpers.root('node_modules/ckeditor/config.js'), to: 'node_modules/ckeditor' },
      { from: helpers.root('node_modules/ckeditor/styles.js'), to: 'node_modules/ckeditor' },
      { from: helpers.root('node_modules/ckeditor/contents.css'), to: 'node_modules/ckeditor' },
      { from: helpers.root('node_modules/ckeditor/skins/moono'), to: 'node_modules/ckeditor/skins/moono' },
      { from: helpers.root('node_modules/ckeditor/lang/zh-cn.js'), to: 'node_modules/ckeditor/lang' },
      { from: helpers.root('node_modules/ckeditor/plugins'), to: 'node_modules/ckeditor/plugins' },
      { from: helpers.root('node_modules/ckeditor/adapters'), to: 'node_modules/ckeditor/adapters' },
    ])
  ]
}

error msg:

95% emittingfs.js:634
  return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
                 ^

Error: EMFILE: too many open files, open 'D:\work\nodejs\angular\angular2-electron\node_modules\ckeditor\plugins\sourcedialog\lang\ar.js'
    at Error (native)
    at Object.fs.openSync (fs.js:634:18)
    at Object.fs.readFileSync (fs.js:502:33)
    at Object.source (D:\work\nodejs\angular\angular2-electron\node_modules\.npminstall\copy-webpack-plugin\3.0.1\copy-webpack-plugin\dist\writeFileToAssets.js:45:31)
    at Compiler.writeOut (D:\work\nodejs\angular\angular2-electron\node_modules\.npminstall\webpack\2.1.0-beta.15\webpack\lib\Compiler.js:290:26)
    at D:\work\nodejs\angular\angular2-electron\node_modules\.npminstall\mkdirp\0.5.1\mkdirp\index.js:48:26
    at FSReqWrap.oncomplete (fs.js:117:15)

[kb3eua]

@kevlened I'm seeing the same issue that @keyiis is reporting. Even after applying this patch, the error still occurs when trying to copy the "ckeditor" folder.

[jgrosso]

+1

[keyiis]

@kb3eua I use gulp to copy file.

[kb3eua]

@keyiis I didn't want to introduce another task runner. I was able to get it working by upgrading to the latest version of this plugin and using the following configuration to ignore certain files:

 new CopyWebpackPlugin([{
    from: './node_modules/ckeditor',
    to: './dist/scripts/ckeditor',
    ignore: [
        '**/samples/**',
        '**/lang/!(en.js)'
    ]
}]);

[tallaxes]

Having the same error on Windows with many files (Angular 2 CLI Webpack ng build, bower_components in assets; yeah, I know). graceful-fs does not help, most likely because it does not patch readFileSync/openSync codepath. Webpack's Compiler.js emitFiles uses unconstrained async.ForEach over assets, hitting the Node limit on open files (2048 on Windows, due to use of CRT).

require("async").forEach(Object.keys(compilation.assets), function(file, callback) {

Workaround: I don't understand webpack/copy-webpack-plugin/graceful-fs well enough to suggest the right way of fixing this, but constraining webpack to a reasonable number of async operations (using forEachLimit) works fine:

require("async").forEachLimit(Object.keys(compilation.assets), 2000,
  require("async").ensureAsync(function(file, callback) {

EDIT: Wrapped callback with async.ensureAsync - fixes stack overflow for when assets are served from memory

[id0Sch]

having this issue on OSX (patch applied and all)

[McPo]

This is still very much an issue, the monkey patch does not work.
Can we get this issue reopened for visibility @kevlened.

Ive looked into it, but am still clueless as to a fix.

To add a bit more detail, this is the exact error Im getting. I have add issues across Windows, OSX and Linux.

5752ms optimize chunk assets
1773ms optimize assets
95% emitfs.js:634
return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
^

Error: EMFILE: too many open files, open '/home/mcp/Sources/saltim-desktop/src/node_modules/lodash/_baseIsEqual.js'
at Error (native)
at Object.fs.openSync (fs.js:634:18)
at Object.fs.readFileSync (fs.js:502:33)
at Object.source (/home/mcp/Sources/saltim-desktop/node_modules/copy-webpack-plugin/dist/writeFileToAssets.js:45:31)
at Compiler.writeOut (/home/mcp/Sources/saltim-desktop/node_modules/webpack/lib/Compiler.js:256:26)
at /home/mcp/Sources/saltim-desktop/node_modules/mkdirp/index.js:48:26
at FSReqWrap.oncomplete (fs.js:117:15)

npm ERR! Linux 4.4.0-42-generic

Worst of all, I simply doe a cp before or after webpack runs, it appears my build doesn't pick up the folder, so thats not a solution for me unfortunately.

[McPo]

@tallaxes path does work, although I needed a much lower limit (100). Patching WebPack itself also isn't ideal.

The file that needs to be patched is ./node_modules/webpack/lib/Compiler.js.
Make sure not to forget the closing bracket.

Thanks @tallaxes, Im still looking for a more elegant solution though. I doubt Webpack themselves will accept a PR when the number is variable.

[tallaxes]

@McPo Webpack: It is still worth bringing it to their attention, I think. And there could be ways of coming up with a reasonable number, or just making it configurable.

100 seems unreasonably low; wonder what would limit it like that... For OSX/Linux: did you check OS-level file descriptor limits (i.e. ulimit -n)?

[McPo]

@tallaxes done.
webpack/webpack#3164

Ive bumped it back up to 1000 it works fine. I suppose my machine is under heavy load, having said that, lowering it to 100 had no adverse effect on the build time.

[keyiis]

My issues looks like the floder has symlink,resulting in a circular dependency,but I'm not sure.

[McPo]

Ive created a PR. for those suffering from this issue you can use it right away by setting your package.json to

"webpack": "McPo/webpack#1.13.2-file-limit"

Ive limited to 1000, but perhaps 500 is more reasonable.
You can to set it to a user defined value using the environment variable WEBPACK_FILE_LIMIT

webpack/webpack#3166

[kevlened]

The upcoming version of copy-webpack-plugin (currently in master) avoids fs.readFileSync, which is the main source of the issue, because it isn't patched by graceful-fs. That should allow graceful-fs to be a proper solution.

For additional control, I've added a concurrency option that will allow changing how many files the plugin attempts to read at once (currently set to a default of 1000, but it sounds like that should be reduced). Does 100 sound like a reasonable default?

[kevlened]

This should be fixed in the latest version (v4.0.0). We avoid fs.readFileSync, so graceful-fs can now patch the issue, but there are also extra measures to ensure we don't read too many files at once (a concurrency option, which defaults to 100). I'm closing this for now, but if you encounter the error after upgrading to v4.0.0, I'll reopen.

[irjigeiler]

still get this error on 4.0.1

[jf990]

I upgraded to Node 8 and the EMFILE error went away.

[Cjhome]

This error can be due to a configuration file error or a parameter filling error while configuring webpack

[joeyparis]

I'm running into this problem with webpack 4.4.1 and node 10.5