Gradio Demo

[AK391]

web demo on Gradio Hub

[ben-hayes]

Hi @AK391, this is really cool! ​Thanks for putting this together so quickly. Two thoughts:

1. As much of the computation time in the gradio demo is being spent on pitch extraction, I think it might be giving the wrong impression about the efficiency of the synthesis model — especially as it's not clear from the interface what computation is taking place. I would suggest maybe trying the extract_f0_with_pyin function in lieu of CREPE:

   neural-waveshaping-synthesis/neural_waveshaping_synthesis/data/utils/f0_extraction.py

   Lines 61 to 70 in cb7aaea

   	def extract_f0_with_pyin(
   	audio: np.ndarray,
   	sample_rate: float,
   	minimum_frequency: float = 65.0, # recommended minimum freq from librosa docs
   	maximum_frequency: float = 2093.0, # recommended maximum freq from librosa docs
   	frame_length: int = 1024,
   	hop_length: int = 128,
   	fill_na: Optional[float] = None,
   	interpolate_fn: Optional[Callable] = linear_interpolation,
   	):

2. I'd like to limit the content of the repo to only what is necessary to reproduce results from the paper/online supplement. With that in mind, whilst I love the demo and I'm very grateful to you for creating it, I'm not sure it makes sense to merge this to the main repo. If you're happy to host this on your account, I'd love to add a link to the demo/code to the readme, along with a credit for you. Let me know what you think of this.

Best

Ben

[AK391]

@ben-hayes thanks, when trying extract_f0_with_pyin getting this error

Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.7/dist-packages/flask_cors/extension.py", line 165, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.7/dist-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functionsrule.endpoint
File "/usr/local/lib/python3.7/dist-packages/gradio/networking.py", line 91, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/gradio/networking.py", line 179, in predict
prediction, durations = app.interface.process(raw_input)
File "/usr/local/lib/python3.7/dist-packages/gradio/interface.py", line 320, in process
predictions, durations = self.run_prediction(processed_input, return_duration=True)
File "/usr/local/lib/python3.7/dist-packages/gradio/interface.py", line 298, in run_prediction
raise exception
File "/usr/local/lib/python3.7/dist-packages/gradio/interface.py", line 293, in run_prediction
prediction = predict_fn(*processed_input)
File "", line 87, in inference
loudness_filtered = loudness * (confidence > loudness_conf_filter)
ValueError: operands could not be broadcast together with shapes (2501,) (320000,)

 with torch.no_grad():
        f0, confidence = extract_f0_with_pyin(
            audio,
            sample_rate=float(rate),
            maximum_frequency=1000
            )

[AK391]

@ben-hayes also adding the extract_f0_with_pyin method to the colab as a option as well as crepe