-
Notifications
You must be signed in to change notification settings - Fork 192
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix "Too many open files" errors when running on large repo on OSX #283
Conversation
It appears we're running out of file handles when running on large repo on OSX due to waiting until all files have been processed to join/close the subprocesses. This PR joins/closes them as they finish, resolving the issue. ``` $ python3 -m libcst.tool codemod {codemod_name} {direectory} Calculating full-repo metadata... Executing codemod... Codemodding {file} Traceback (most recent call last): File "/Users/rwilliams/src/go/src/github.com/lyft/python-lyft-ingest/venv/lib/python3.6/site-packages/libcst/codemod/_cli.py", line 253, in _parallel_exec_process_stub OSError: [Errno 24] Too many open files: '{file}' Failed to codemod {file} 19.80s 24% complete, 01m 02s estimated for 741 files to go...Traceback (most recent call last): File "/opt/lyft/brew/Cellar/python36/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/opt/lyft/brew/Cellar/python36/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/Users/rwilliams/src/go/src/github.com/lyft/python-lyft-ingest/venv/lib/python3.6/site-packages/libcst/tool.py", line 833, in <module> main(os.environ.get("LIBCST_TOOL_COMMAND_NAME", "libcst.tool"), sys.argv[1:]) File "/Users/rwilliams/src/go/src/github.com/lyft/python-lyft-ingest/venv/lib/python3.6/site-packages/libcst/tool.py", line 828, in main return lookup.get(args.action or None, _invalid_command)(proc_name, command_args) File "/Users/rwilliams/src/go/src/github.com/lyft/python-lyft-ingest/venv/lib/python3.6/site-packages/libcst/tool.py", line 581, in _codemod_impl repo_root=config["repo_root"], File "/Users/rwilliams/src/go/src/github.com/lyft/python-lyft-ingest/venv/lib/python3.6/site-packages/libcst/codemod/_cli.py", line 720, in parallel_exec_transform_with_prettyprint process.start() File "/opt/lyft/brew/Cellar/python36/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 105, in start self._popen = self._Popen(self) File "/opt/lyft/brew/Cellar/python36/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/context.py", line 223, in _Popen return _default_context.get_context().Process._Popen(process_obj) File "/opt/lyft/brew/Cellar/python36/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/context.py", line 277, in _Popen return Popen(process_obj) File "/opt/lyft/brew/Cellar/python36/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_fork.py", line 19, in __init__ self._launch(process_obj) File "/opt/lyft/brew/Cellar/python36/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_fork.py", line 65, in _launch parent_r, child_w = os.pipe() OSError: [Errno 24] Too many open files ```
Codecov Report
@@ Coverage Diff @@
## master #283 +/- ##
==========================================
- Coverage 93.96% 93.92% -0.04%
==========================================
Files 219 219
Lines 21245 21253 +8
==========================================
Hits 19962 19962
- Misses 1283 1291 +8
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
@rowillia , thanks for your fix!
It appears we're running out of file handles when running on large repo on OSX due to waiting until all files have been
processed to join/close the subprocesses.
This PR joins/closes them as they finish, resolving the issue.
Summary
Test Plan