AttributeError: 'GRU' object has no attribute '_flat_weights_names'

[jannessauer]

Dear

I am currently running the taiyaki pipeline to train the a Guppy basecaller model. I have run through the first steps already before to generate preliminary files for the Bonito basecaller. Nevertheless, when I want to train a Guppy model using the "train_flipflop.py" flow, I do get following output message.
`* Taiyaki version 5.0.1

• Platform is Linux-5.4.0-42-generic-x86_64-with-debian-buster-sid
• PyTorch version 1.4.0
• Running on CPU
• Command line:
• "./train_flipflop.py ../../../../media/thor/loki/Bonito_Train/MinION/r941_dna_minion.checkpoint ../../../../media/thor/loki/Bonito_Train/MinION/Bonito_Nick.hdf5
• Started on 2020-09-11 09:18:07.393917
• Loading data from ../../../../media/thor/loki/Bonito_Train/MinION/Bonito_Nick.hdf5
• Per read file MD5 9f78647a4f00f9d7d055b1f80f706832
• Reads not filtered by id
• Using alphabet definition: canonical alphabet ACGT and no modified bases
• Loaded 115821 reads.
• Sampled 1000 chunks: median(mean_dwell)=9.97, mad(mean_dwell)=1.17
• Reading network from ../../../../media/thor/loki/Bonito_Train/MinION/r941_dna_minion.checkpoint
  /home/thor/anaconda3/lib/python3.7/site-packages/torch/serialization.py:593: SourceChangeWarning: source code of class 'taiyaki.layers.Serial' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes.
  warnings.warn(msg, SourceChangeWarning)
  /home/thor/anaconda3/lib/python3.7/site-packages/torch/serialization.py:593: SourceChangeWarning: source code of class 'torch.nn.modules.container.ModuleList' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes.
  warnings.warn(msg, SourceChangeWarning)
  /home/thor/anaconda3/lib/python3.7/site-packages/torch/serialization.py:593: SourceChangeWarning: source code of class 'taiyaki.layers.Convolution' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes.
  warnings.warn(msg, SourceChangeWarning)
  /home/thor/anaconda3/lib/python3.7/site-packages/torch/serialization.py:593: SourceChangeWarning: source code of class 'torch.nn.modules.padding.ConstantPad1d' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes.
  warnings.warn(msg, SourceChangeWarning)
  /home/thor/anaconda3/lib/python3.7/site-packages/torch/serialization.py:593: SourceChangeWarning: source code of class 'torch.nn.modules.conv.Conv1d' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes.
  warnings.warn(msg, SourceChangeWarning)
  /home/thor/anaconda3/lib/python3.7/site-packages/torch/serialization.py:593: SourceChangeWarning: source code of class 'taiyaki.layers.Reverse' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes.
  warnings.warn(msg, SourceChangeWarning)
  /home/thor/anaconda3/lib/python3.7/site-packages/torch/serialization.py:593: SourceChangeWarning: source code of class 'taiyaki.layers.GruMod' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes.
  warnings.warn(msg, SourceChangeWarning)
  /home/thor/anaconda3/lib/python3.7/site-packages/torch/serialization.py:593: SourceChangeWarning: source code of class 'torch.nn.modules.rnn.GRU' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes.
  warnings.warn(msg, SourceChangeWarning)
  /home/thor/anaconda3/lib/python3.7/site-packages/torch/serialization.py:593: SourceChangeWarning: source code of class 'taiyaki.layers.GlobalNormFlipFlop' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes.
  warnings.warn(msg, SourceChangeWarning)
  /home/thor/anaconda3/lib/python3.7/site-packages/torch/serialization.py:593: SourceChangeWarning: source code of class 'torch.nn.modules.linear.Linear' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes.
  warnings.warn(msg, SourceChangeWarning)
• Network has 285160 parameters.
• Loaded standard (canonical bases-only) model.
• Dumping initial model
  Traceback (most recent call last):
  File "./train_flipflop.py", line 531, in
  main()
  File "./train_flipflop.py", line 349, in main
  network = network_save_skeleton.to(device)
  File "/home/thor/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 425, in to
  return self._apply(convert)
  File "/home/thor/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 201, in _apply
  module._apply(fn)
  File "/home/thor/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 201, in _apply
  module._apply(fn)
  File "/home/thor/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 201, in _apply
  module._apply(fn)
  [Previous line repeated 1 more time]
  File "/home/thor/anaconda3/lib/python3.7/site-packages/torch/nn/modules/rnn.py", line 137, in _apply
  self._flat_weights = [getattr(self, weight) for weight in self._flat_weights_names]
  File "/home/thor/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 576, in getattr
  type(self).name, name))
  AttributeError: 'GRU' object has no attribute '_flat_weights_names'`

As it is the first time I am running this command, I am wondering if it is a system error or something I am doing wrong. Since the error occurs after reading the checkpoint file, I am not sure where to localize the issue. Using the same .hdf5 mapped-reads file I have successfully trained the Bonito basecaller. So I would be surprised if this file is corrupted.

Thank you in advance.
Best regards,
Nick Vereecke

[tmassingham-ont]

The current release of Taiyaki does not support PyTorch 1.4.0, due to changes in how PyTorch deals with recurrent neural networks. We are working towards releasing an updated version but, in the meantime, please trying running from a virtual environment as described in README.md

[menickname]

dear @tmassingham-ont

Thank you for your response. I will try that! However I noticed last time installation of the environment through "make install" did not result in successfull installation:

In file included from /home/thor/taiyaki-master/venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822:0,from /home/thor/taiyaki-master/venv/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,from /home/thor/taiyaki-master/venv/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,from taiyaki/squiggle_match/squiggle_match.c:643:/home/thor/taiyaki-master/venv/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]#warning "Using deprecated NumPy API, disable it with " ^~~~~~~/home/thor/anaconda3/bin/x86_64-conda_cos6-linux-gnu-cc -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -Wstrict-prototypes -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/thor/anaconda3/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/thor/anaconda3/include -fPIC -I./taiyaki/squiggle_match -I/home/thor/taiyaki-master/venv/lib/python3.7/site-packages/numpy/core/include -I/home/thor/anaconda3/include/python3.7m -c taiyaki/squiggle_match/c_squiggle_match.c -o build/temp.linux-x86_64-3.7/taiyaki/squiggle_match/c_squiggle_match.o -O3 -fopenmp -std=c99 -march=nativetaiyaki/squiggle_match/c_squiggle_match.c: In function 'squiggle_match_viterbi':taiyaki/squiggle_match/c_squiggle_match.c:409:28: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]if(nfstate - 1 != path[sample_max - 1]){^~taiyaki/squiggle_match/c_squiggle_match.c:416:29: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]if(path[sample] >= nfstate){^~creating build/lib.linux-x86_64-3.7creating build/lib.linux-x86_64-3.7/taiyakicreating build/lib.linux-x86_64-3.7/taiyaki/squiggle_matchx86_64-conda_cos6-linux-gnu-gcc -pthread -shared -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-rpath,/home/thor/anaconda3/lib -L/home/thor/anaconda3/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-rpath,/home/thor/anaconda3/lib -L/home/thor/anaconda3/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/thor/anaconda3/lib -Wl,-rpath-link,/home/thor/anaconda3/lib -L/home/thor/anaconda3/lib -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/thor/anaconda3/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/thor/anaconda3/include build/temp.linux-x86_64-3.7/taiyaki/squiggle_match/squiggle_match.o build/temp.linux-x86_64-3.7/taiyaki/squiggle_match/c_squiggle_match.o -o build/lib.linux-x86_64-3.7/taiyaki/squiggle_match/squiggle_match.cpython-37m-x86_64-linux-gnu.so -fopenmp/home/thor/anaconda3/bin/../lib/gcc/x86_64-conda_cos6-linux-gnu/7.3.0/../../../../x86_64-conda_cos6-linux-gnu/bin/ld: cannot find -lgompcollect2: error: ld returned 1 exit statuserror: command 'x86_64-conda_cos6-linux-gnu-gcc' failed with exit status 1make: *** [Makefile:50: install] Error 1

Hope this can be solved easily?

Thank you in advance.

[tmassingham-ont]

The problem here appears to be that your Conda environment does not contain, or Python's setuptools cannot find, the OpenMP library to link against.
/home/thor/anaconda3/bin/../lib/gcc/x86_64-conda_cos6-linux-gnu/7.3.0/../../../../x86_64-conda_cos6-linux-gnu/bin/ld: cannot find -lgomp

You made need to install some additional software in your Conda environment, or add an new include path to setup.py e.g. change every occurrence of extra_link_args to add an library directory extra_link_args=["-fopenmp", "-L path/to/directory/containing/libgomp"]

Taiyaki is research software and we don't support running it in a Conda environment. Are you able to create the virtualenv outside of Conda?