🐛 TypeError: cannot pickle sqlite3.Connection object

[nicolay-r]

This problem caused by type:
torch.utils.data.dataloader.DataLoader
here:

ARElight/arelight/pipelines/items/inference_bert_opennre.py

Line 125 in e18e744

for iter, data in enumerate(eval_loader):

Raised #149 related issue.

• torch -- take control over num_workers parameter #149

The most similar issue mentioned there:
https://discuss.pytorch.org/t/dataloader-multiprocessing-error-cant-pickle-odict-keys-objects-when-num-workers-0/43951
which navigates us to:
pytorch/vision#689

from where as we can see, we got no control over the related parameter (its value binded to 8):

ARElight/arelight/third_party/torch.py

Lines 89 to 97 in e18e744

	task_kwargs, num_workers=8, collate_fn=SQLiteSentenceREDataset.collate_fn, **kwargs):
	dataset = SQLiteSentenceREDataset(path=path, table_name=table_name, rel2id=rel2id,
	tokenizer=tokenizer, kwargs=kwargs, task_kwargs=task_kwargs)
	data_loader = data.DataLoader(dataset=dataset,
	batch_size=batch_size,
	shuffle=shuffle,
	pin_memory=True,
	num_workers=num_workers,
	collate_fn=collate_fn)

Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/kmax/PycharmProjects/ARElight-main/arelight/run/infer.py", line 328, in <module>
    BasePipelineLauncher.run(pipeline=pipeline, pipeline_ctx=PipelineResult(merge_dictionaries(settings)),
  File "/Users/kmax/PycharmProjects/ARElight-main/venv/lib/python3.9/site-packages/arekit/common/pipeline/base.py", line 18, in run
    item_result = item.apply(input_data=input_data, pipeline_ctx=pipeline_ctx)
  File "/Users/kmax/PycharmProjects/ARElight-main/venv/lib/python3.9/site-packages/arekit/common/pipeline/items/base.py", line 48, in apply
    output_data = self.apply_core(input_data=input_data, pipeline_ctx=pipeline_ctx)
  File "/Users/kmax/PycharmProjects/ARElight-main/arelight/pipelines/items/inference_writer.py", line 31, in apply_core
    self.__writer.write(header=header, contents_it=contents_it,
  File "/Users/kmax/PycharmProjects/ARElight-main/arelight/predict/writer_sqlite3.py", line 21, in write
    SQLiteProvider.write(
  File "/Users/kmax/PycharmProjects/ARElight-main/venv/lib/python3.9/site-packages/arekit/common/service/sqlite.py", line 26, in write
    for uid, data in data_it:
  File "/Users/kmax/PycharmProjects/ARElight-main/venv/lib/python3.9/site-packages/tqdm/std.py", line 1195, in iter
    for obj in iterable:
  File "/Users/kmax/PycharmProjects/ARElight-main/arelight/predict/provider.py", line 16, in __iter_contents
    for sample_id, uint_label in sample_id_with_uint_labels_iter:
  File "/Users/kmax/PycharmProjects/ARElight-main/arelight/pipelines/items/inference_bert_opennre.py", line 125, in iter_results
    for iter, data in enumerate(eval_loader):
  File "/Users/kmax/PycharmProjects/ARElight-main/venv/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 441, in iter
    @property
  File "/Users/kmax/PycharmProjects/ARElight-main/venv/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 388, in _get_iterator
    def multiprocessing_context(self, multiprocessing_context):
  File "/Users/kmax/PycharmProjects/ARElight-main/venv/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1042, in init
    self._data_queue = queue.Queue()  # type: ignore[var-annotated]
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in init
    super().__init__(process_obj)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_fork.py", line 19, in init
    self._launch(process_obj)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle 'sqlite3.Connection' object

[nicolay-r]

Fixed: reproduced with no issues on Mac