Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Broken Pipe Issue #3

Open
gautamchitnis opened this issue Dec 17, 2019 · 5 comments
Open

Broken Pipe Issue #3

gautamchitnis opened this issue Dec 17, 2019 · 5 comments

Comments

@gautamchitnis
Copy link

Here's what I need help with:

I have been banging my head against a wall trying to get detectron2 to work, but to no avail. Now after I stumbled onto this project, i was finally able to build and install detectron2 properly. I tried replicating this notebook locally, which works as expected, even predicting balloons properly.

But when I reproduce the same code as a python file and try to run it using either pycharm or cmd line I'm hitting a weird error which I've not found anything about either in original Detectron2 repo or simple google search.

This gist has the code I pulled into a single file.

Here's the error log:

Config 'F:\dd\detectron2\configs\COCO-InstanceSegmentation\mask_rcnn_R_50_FPN_3x.yaml' has no VERSION. Assuming it to be compatible with latest v2.
Config 'F:\dd\detectron2\configs\COCO-InstanceSegmentation\mask_rcnn_R_50_FPN_3x.yaml' has no VERSION. Assuming it to be compatible with latest v2.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
Traceback (most recent call last):
  File "sample", line 79, in <module>
    trainer = DefaultTrainer(cfg)
  File "f:\dd\conan_det\detectron2\detectron2\engine\defaults.py", line 249, in __init__
      File "E:\Anaconda\envs\dd\lib\multiprocessing\spawn.py", line 105, in spawn_main
super().__init__(model, data_loader, optimizer)
  File "f:\dd\conan_det\detectron2\detectron2\engine\train_loop.py", line 194, in __init__
    exitcode = _main(fd)
  File "E:\Anaconda\envs\dd\lib\multiprocessing\spawn.py", line 114, in _main
    self._data_loader_iter = iter(data_loader)
  File "E:\Anaconda\envs\dd\lib\site-packages\torch\utils\data\dataloader.py", line 278, in __iter__
    prepare(preparation_data)
  File "E:\Anaconda\envs\dd\lib\multiprocessing\spawn.py", line 225, in prepare
    return _MultiProcessingDataLoaderIter(self)
_fixup_main_from_path(data['init_main_from_path'])  File "E:\Anaconda\envs\dd\lib\site-packages\torch\utils\data\dataloader.py", line 682, in __init__

  File "E:\Anaconda\envs\dd\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
      File "E:\Anaconda\envs\dd\lib\runpy.py", line 263, in run_path
w.start()
  File "E:\Anaconda\envs\dd\lib\multiprocessing\process.py", line 112, in start
    pkg_name=pkg_name, script_name=fname)
self._popen = self._Popen(self)  File "E:\Anaconda\envs\dd\lib\runpy.py", line 96, in _run_module_code

  File "E:\Anaconda\envs\dd\lib\multiprocessing\context.py", line 223, in _Popen
    mod_name, mod_spec, pkg_name, script_name)
return _default_context.get_context().Process._Popen(process_obj)  File "E:\Anaconda\envs\dd\lib\runpy.py", line 85, in _run_code

  File "E:\Anaconda\envs\dd\lib\multiprocessing\context.py", line 322, in _Popen
    exec(code, run_globals)
      File "C:\Users\Gautam\PycharmProjects\detectron2_sample\sample", line 79, in <module>
return Popen(process_obj)
  File "E:\Anaconda\envs\dd\lib\multiprocessing\popen_spawn_win32.py", line 89, in __init__
    trainer = DefaultTrainer(cfg)
reduction.dump(process_obj, to_child)  File "f:\dd\conan_det\detectron2\detectron2\engine\defaults.py", line 249, in __init__

  File "E:\Anaconda\envs\dd\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
    BrokenPipeErrorsuper().__init__(model, data_loader, optimizer):
[Errno 32] Broken pipe  File "f:\dd\conan_det\detectron2\detectron2\engine\train_loop.py", line 194, in __init__

    self._data_loader_iter = iter(data_loader)
  File "E:\Anaconda\envs\dd\lib\site-packages\torch\utils\data\dataloader.py", line 278, in __iter__
    return _MultiProcessingDataLoaderIter(self)
  File "E:\Anaconda\envs\dd\lib\site-packages\torch\utils\data\dataloader.py", line 682, in __init__
    w.start()
  File "E:\Anaconda\envs\dd\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)
  File "E:\Anaconda\envs\dd\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "E:\Anaconda\envs\dd\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "E:\Anaconda\envs\dd\lib\multiprocessing\popen_spawn_win32.py", line 46, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "E:\Anaconda\envs\dd\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
    _check_not_importing_main()
  File "E:\Anaconda\envs\dd\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
    is not going to be frozen to produce an executable.''')
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

Environment

The filename, directory name, or volume label syntax is incorrect.
------------------------  ------------------------------------------------------------------
sys.platform              win32
Python                    3.7.4 (default, Aug  9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)]
Numpy                     1.16.5
Detectron2 Compiler       MSVC 190024215
Detectron2 CUDA Compiler  10.2
DETECTRON2_ENV_MODULE     <not set>
PyTorch                   1.3.1
PyTorch Debug Build       False
torchvision               0.4.2
CUDA available            True
GPU 0                     GeForce GTX 1060 3GB
CUDA_HOME                 C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2
NVCC                      Not Available
Pillow                    6.2.1
cv2                       3.4.2
------------------------  ------------------------------------------------------------------
PyTorch built with:
  - MSVC 191125547
  - Intel(R) Math Kernel Library Version 2019.0.4 Product Build 20190411 for Intel(R) 64 architecture applications
  - OpenMP 200203
  - CUDA Runtime 10.1
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_50,code=compute_50
  - CuDNN 7.5.1
  - Magma 2.5.0
  - Build settings: BLAS=MKL, BUILD_NAMEDTENSOR=OFF, BUILD_TYPE=Release, CXX_FLAGS=/DWIN32 /D_WINDOWS  /GR  /w /EHa /MP /bigobj -openmp, DISABLE_NUMA=1, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, USE_CUDA=True, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF,
@gautamchitnis
Copy link
Author

As I posted this issue, i just noticed my pytorch is built with a different CUDA Runtime, I will try to fix that and see if the error goes away, but i have a feeling that's not gonna make a difference.

@gautamchitnis
Copy link
Author

Tried compiling this repo with CUDA10.1 installed. The compilation errored out as logged here

Important lines of the log are:

C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -DWITH_CUDA -IF:\dd\conan_det\detectron2\detectron2\layers\csrc -IE:\Anaconda\envs\dd\lib\site-packages\torch\include -IE:\Anaconda\envs\dd\lib\site-packages\torch\include\torch\csrc\api\include -IE:\Anaconda\envs\dd\lib\site-packages\torch\include\TH -IE:\Anaconda\envs\dd\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\include" -IE:\Anaconda\envs\dd\include -IE:\Anaconda\envs\dd\include "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\8.1\include\\shared" "-IC:\Program Files (x86)\Windows Kits\8.1\include\\um" "-IC:\Program Files (x86)\Windows Kits\8.1\include\\winrt" /EHsc /TpF:\dd\conan_det\detectron2\detectron2\layers\csrc\ROIAlignRotated\ROIAlignRotated_cpu.cpp /Fobuild\temp.win-amd64-3.7\Release\dd\conan_det\detectron2\detectron2\layers\csrc\ROIAlignRotated\ROIAlignRotated_cpu.obj /MD /wd4819 /EHsc -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0

error: command 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v10.1\\bin\\nvcc.exe' failed with exit status 2

This is as far as a i understand. Please do take a look at mentioned log file to get a clear picture. Any help is appreciated.

@conansherry
Copy link
Owner

Here's what I need help with:

I have been banging my head against a wall trying to get detectron2 to work, but to no avail. Now after I stumbled onto this project, i was finally able to build and install detectron2 properly. I tried replicating this notebook locally, which works as expected, even predicting balloons properly.

But when I reproduce the same code as a python file and try to run it using either pycharm or cmd line I'm hitting a weird error which I've not found anything about either in original Detectron2 repo or simple google search.

This gist has the code I pulled into a single file.

Here's the error log:

Config 'F:\dd\detectron2\configs\COCO-InstanceSegmentation\mask_rcnn_R_50_FPN_3x.yaml' has no VERSION. Assuming it to be compatible with latest v2.
Config 'F:\dd\detectron2\configs\COCO-InstanceSegmentation\mask_rcnn_R_50_FPN_3x.yaml' has no VERSION. Assuming it to be compatible with latest v2.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
Traceback (most recent call last):
  File "sample", line 79, in <module>
    trainer = DefaultTrainer(cfg)
  File "f:\dd\conan_det\detectron2\detectron2\engine\defaults.py", line 249, in __init__
      File "E:\Anaconda\envs\dd\lib\multiprocessing\spawn.py", line 105, in spawn_main
super().__init__(model, data_loader, optimizer)
  File "f:\dd\conan_det\detectron2\detectron2\engine\train_loop.py", line 194, in __init__
    exitcode = _main(fd)
  File "E:\Anaconda\envs\dd\lib\multiprocessing\spawn.py", line 114, in _main
    self._data_loader_iter = iter(data_loader)
  File "E:\Anaconda\envs\dd\lib\site-packages\torch\utils\data\dataloader.py", line 278, in __iter__
    prepare(preparation_data)
  File "E:\Anaconda\envs\dd\lib\multiprocessing\spawn.py", line 225, in prepare
    return _MultiProcessingDataLoaderIter(self)
_fixup_main_from_path(data['init_main_from_path'])  File "E:\Anaconda\envs\dd\lib\site-packages\torch\utils\data\dataloader.py", line 682, in __init__

  File "E:\Anaconda\envs\dd\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
      File "E:\Anaconda\envs\dd\lib\runpy.py", line 263, in run_path
w.start()
  File "E:\Anaconda\envs\dd\lib\multiprocessing\process.py", line 112, in start
    pkg_name=pkg_name, script_name=fname)
self._popen = self._Popen(self)  File "E:\Anaconda\envs\dd\lib\runpy.py", line 96, in _run_module_code

  File "E:\Anaconda\envs\dd\lib\multiprocessing\context.py", line 223, in _Popen
    mod_name, mod_spec, pkg_name, script_name)
return _default_context.get_context().Process._Popen(process_obj)  File "E:\Anaconda\envs\dd\lib\runpy.py", line 85, in _run_code

  File "E:\Anaconda\envs\dd\lib\multiprocessing\context.py", line 322, in _Popen
    exec(code, run_globals)
      File "C:\Users\Gautam\PycharmProjects\detectron2_sample\sample", line 79, in <module>
return Popen(process_obj)
  File "E:\Anaconda\envs\dd\lib\multiprocessing\popen_spawn_win32.py", line 89, in __init__
    trainer = DefaultTrainer(cfg)
reduction.dump(process_obj, to_child)  File "f:\dd\conan_det\detectron2\detectron2\engine\defaults.py", line 249, in __init__

  File "E:\Anaconda\envs\dd\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
    BrokenPipeErrorsuper().__init__(model, data_loader, optimizer):
[Errno 32] Broken pipe  File "f:\dd\conan_det\detectron2\detectron2\engine\train_loop.py", line 194, in __init__

    self._data_loader_iter = iter(data_loader)
  File "E:\Anaconda\envs\dd\lib\site-packages\torch\utils\data\dataloader.py", line 278, in __iter__
    return _MultiProcessingDataLoaderIter(self)
  File "E:\Anaconda\envs\dd\lib\site-packages\torch\utils\data\dataloader.py", line 682, in __init__
    w.start()
  File "E:\Anaconda\envs\dd\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)
  File "E:\Anaconda\envs\dd\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "E:\Anaconda\envs\dd\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "E:\Anaconda\envs\dd\lib\multiprocessing\popen_spawn_win32.py", line 46, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "E:\Anaconda\envs\dd\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
    _check_not_importing_main()
  File "E:\Anaconda\envs\dd\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
    is not going to be frozen to produce an executable.''')
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

Environment

The filename, directory name, or volume label syntax is incorrect.
------------------------  ------------------------------------------------------------------
sys.platform              win32
Python                    3.7.4 (default, Aug  9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)]
Numpy                     1.16.5
Detectron2 Compiler       MSVC 190024215
Detectron2 CUDA Compiler  10.2
DETECTRON2_ENV_MODULE     <not set>
PyTorch                   1.3.1
PyTorch Debug Build       False
torchvision               0.4.2
CUDA available            True
GPU 0                     GeForce GTX 1060 3GB
CUDA_HOME                 C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2
NVCC                      Not Available
Pillow                    6.2.1
cv2                       3.4.2
------------------------  ------------------------------------------------------------------
PyTorch built with:
  - MSVC 191125547
  - Intel(R) Math Kernel Library Version 2019.0.4 Product Build 20190411 for Intel(R) 64 architecture applications
  - OpenMP 200203
  - CUDA Runtime 10.1
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_50,code=compute_50
  - CuDNN 7.5.1
  - Magma 2.5.0
  - Build settings: BLAS=MKL, BUILD_NAMEDTENSOR=OFF, BUILD_TYPE=Release, CXX_FLAGS=/DWIN32 /D_WINDOWS  /GR  /w /EHa /MP /bigobj -openmp, DISABLE_NUMA=1, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, USE_CUDA=True, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF,

which script do you run?

@gautamchitnis
Copy link
Author

@conansherry https://gist.github.com/gautamchitnis/bed3dd638ce068a84bae712ec8feafbb

Here's the .py script I'm running.

@SFraissTU
Copy link

Hi! I solved this problem by adding
if __name__ == "__main__":
in the beginning of my script and putting everything in the body of that if.
For some weird reason, the code was calling itself in a new process.
Now I run into memory issues, but that's another problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@conansherry @gautamchitnis @SFraissTU