Error generating samples mid 10-epoch changes

[guicattani]

Hi everyone!

I'm having a trouble generating the samples each 10 epochs. The error I get is this:

Traceback (most recent call last):
  File "J:\Assets\Apps\prism-samplernn\train.py", line 335, in <module>
    main()
  File "J:\Assets\Apps\prism-samplernn\train.py", line 318, in main
    model.fit(
  File "C:\Users\ME\anaconda3\envs\prism-samplernn\lib\site-packages\keras\engine\training.py", line 1230, in fit
    callbacks.on_epoch_end(epoch, epoch_logs)
  File "C:\Users\ME\anaconda3\envs\prism-samplernn\lib\site-packages\keras\callbacks.py", line 413, in on_epoch_end
    callback.on_epoch_end(epoch, logs)
  File "J:\Assets\Apps\prism-samplernn\callbacks.py", line 103, in on_epoch_end
    self._generate(ckpt_path, self.generation_args)
  File "J:\Assets\Apps\prism-samplernn\callbacks.py", line 108, in _generate
    generate(output_path, ckpt_path, args['config'], args['num_seqs'], args['dur'], args['sample_rate'],
  File "J:\Assets\Apps\prism-samplernn\generate.py", line 136, in generate
    write_wav(file_name, audio, sample_rate)
  File "J:\Assets\Apps\prism-samplernn\samplernn\audio_file.py", line 34, in write_wav
    sf.write(path, np.array(audio), sample_rate)
  File "C:\Users\ME\anaconda3\envs\prism-samplernn\lib\site-packages\soundfile.py", line 314, in write
    with SoundFile(file, 'w', samplerate, channels,
  File "C:\Users\ME\anaconda3\envs\prism-samplernn\lib\site-packages\soundfile.py", line 629, in __init__
    self._file = self._open(file, mode_int, closefd)
  File "C:\Users\ME\anaconda3\envs\prism-samplernn\lib\site-packages\soundfile.py", line 1183, in _open
    _error_check(_snd.sf_error(file_ptr),
  File "C:\Users\ME\anaconda3\envs\prism-samplernn\lib\site-packages\soundfile.py", line 1357, in _error_check
    raise RuntimeError(prefix + _ffi.string(err_str).decode('utf-8', 'replace'))
RuntimeError: Error opening './generated\\default/default_e=logdir\\default/27.10.2021_09.59.05\\model.ckpt-10_t=0.95.wav': System error.

Anaconda is current work directory is the project's root. I assume that this something of an error when creating the samples, but it works fine when I use the generate.py script
Generated and logdir currently look like this:

Any help is greatly appreciated, thanks!!!

[relativeflux]

I have seen these runtime errors (system errors) occur when it doesn't like the output filename, or it can't find the file... The filename looks a bit suspicious:

./generated\\default/default_e=logdir\\default/27.10.2021_09.59.05\\model.ckpt-10_t=0.95.wav

It seems garbled, is this working when you use the script yourself? When you say it happens when saving every 10 epochs, are you saving more often than that (say every 5) and it just occurs on epochs that are multiples of 10?

[relativeflux]

@guicattani Hi, did you get anywhere with this?

[Naming-isDifficult]

@relativeflux Hi, I'm sorry to bother you but I run into the same problem when I'm trying to generate samples. I've already checked the code and I believe it is because of the differences in the python version. My version is 3.8 btw. Following are my explanation of this issue.

In generate.py, at around line 134, there're two lines of code that are used to generate the full output file path:

file_name = f'{path}({i})' if model.batch_size > 1 else path
file_name = f'{file_name}_t={temperature[i][0]}.wav'

Unfortunately, these two lines of code fail to work properly. The generated file path looks like something like the following if the given output path is D:\test which is definitely not a file path.

D:\\test_e=D:\\prism-samplernn-master\\logdir\\default\\testmodel\\model.ckpy-20_t=0.95.wav

If I change those two lines into something like the following (basically manually assigning a file path to the program), everything works fine.

file_name=r"D:\just_for_test"

[relativeflux]

@Naming-isDifficult Thank you for bringing this to my attention, I shall fix.

[relativeflux]

@Naming-isDifficult
@guicattani

Looks like the actual culprit is this line:

epoch = ckpt_path.split('/model.ckpt-')[-1]

which hardcodes a posix path. It's meant to pull off the epoch number from the checkpoint, but due to the hardcoded forward slash this is ineffective on Windows.

@Naming-isDifficult I don't currently have access to a Windows machine, would you mind quickly testing it with the forward slash removed from that line? So just:

epoch = ckpt_path.split('model.ckpt-')[-1]

Many thanks!

[Naming-isDifficult]

@relativeflux
Sorry for the late reply. I've just finished testing and it seems that everything works fine after removing that forward slash.

[relativeflux]

Much obliged, thank you for taking the time to pick this issue up and quickly test it. I shall push the fix shortly.

[Naming-isDifficult]

My pleasure :)
Happy Thanksgiving btw, though it's still two days away :)

[relativeflux]

Thank you, same to you (although from the UK!).