[ONNXRuntimeError] : 7 : INVALID_PROTOBUF : Load model from onnx/bass.onnx failed:Protobuf parsing failed.

[Ma5onic]

I was unable to build leaderboard B by following the instructions in it's README on my host OS (Ubuntu 20.04) but was unable to get it working.
I then decided to try to build it from scratch withing an nvidia-ubuntu18.04 with no success (same errors as the bare metal attempt. will update issue once I replicate those errors). I then decided to try using the the docker_build.sh utility and was able to get past those errors, however, every time the model tries to perform demixing I receive the following errors:

python predict_blend.py
/home/aicrowd/data/test/Matroda - On My Mind/mixture.wav
92.74054026603699
/home/aicrowd/predict_blend.py:50: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:210.)
  mix_waves = torch.tensor(mix_waves, dtype=torch.float32)
Traceback (most recent call last):
  File "/home/aicrowd/evaluator/music_demixing.py", line 145, in run
    self.evaluation()
  File "/home/aicrowd/evaluator/music_demixing.py", line 132, in evaluation
    self.prediction(mixture_file_path=self.get_music_file_location(music_name),
  File "/home/aicrowd/predict_blend.py", line 24, in prediction
    sources = self.demix(mix.T)
  File "/home/aicrowd/predict_blend.py", line 30, in demix
    base_out = self.demix_base(mix)
  File "/home/aicrowd/predict_blend.py", line 53, in demix_base
    _ort = ort.InferenceSession(f'onnx/{model.target_name}.onnx')
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 280, in __init__
    self._create_inference_session(providers, provider_options)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 307, in _create_inference_session
    sess = C.InferenceSession(session_options, self._model_path, True, self._read_config_from_model)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidProtobuf: [ONNXRuntimeError] : 7 : INVALID_PROTOBUF : Load model from onnx/bass.onnx failed:Protobuf parsing failed.

Traceback (most recent call last):
  File "/home/aicrowd/predict_blend.py", line 84, in <module>
    submission.run()
  File "/home/aicrowd/evaluator/music_demixing.py", line 151, in run
    raise e
  File "/home/aicrowd/evaluator/music_demixing.py", line 145, in run
    self.evaluation()
  File "/home/aicrowd/evaluator/music_demixing.py", line 132, in evaluation
    self.prediction(mixture_file_path=self.get_music_file_location(music_name),
  File "/home/aicrowd/predict_blend.py", line 24, in prediction
    sources = self.demix(mix.T)
  File "/home/aicrowd/predict_blend.py", line 30, in demix
    base_out = self.demix_base(mix)        
  File "/home/aicrowd/predict_blend.py", line 53, in demix_base
    _ort = ort.InferenceSession(f'onnx/{model.target_name}.onnx')
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 280, in __init__
    self._create_inference_session(providers, provider_options)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 307, in _create_inference_session
    sess = C.InferenceSession(session_options, self._model_path, True, self._read_config_from_model)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidProtobuf: [ONNXRuntimeError] : 7 : INVALID_PROTOBUF : Load model from onnx/bass.onnx failed:Protobuf parsing failed.

Has anyone tried a fresh install of this lately?
Would I have better luck using: https://github.com/kuielab/mdx-net ? (I have 2 RTX 3070 available for this project, in the future I would like to train the model to be better at recognising electronic instruments/synths)

[Zokhoi]

Have you checked if the onnx models are actually there in ./onnx? git-lfs does not download the models if the repo has reached its quota, only placing pointer files there. They should be around 20 something MB each.

If the onnx models aren't there, get them from here.

[Ma5onic]

Awesome! Thanks, that fixed my problem in each scenario. That might be worth adding to the documentation.

Unrelated question: Is there a reason that GPU hardware acceleration is not being utilized in this implementation to speed up batches of jobs?