Error when running TFT tutorial: AttributeError: 'functools.partial' object has no attribute '__name__'

[LumingSun]

• PyTorch-Forecasting version: 0.9.0
• PyTorch version: 1.9.0
• Python version: 1.3.8
• Operating System: Debian 9 Linux 4.14

Expected behavior

I created a new virtual env and use pip install pytorch-forecasting and pip install pyarrow to run the tutorial code of TFT model.

Actual behavior

I got the following error:

trainer.fit(...
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
~/qps_old/test.py in 
      165     tft,
      166     train_dataloader=train_dataloader,
----> 167     val_dataloaders=val_dataloader,
      168 )
      169 

~/anaconda3/envs/envtest/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py in fit(self, model, train_dataloader, val_dataloaders, datamodule)
    458         )
    459 
--> 460         self._run(model)
    461 
    462         assert self.state.stopped

~/anaconda3/envs/envtest/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py in _run(self, model)
    753 
    754         # plugin will setup fitting (e.g. ddp will launch child processes)
--> 755         self.pre_dispatch()
    756 
    757         # dispatch `start_training` or `start_evaluating` or `start_predicting`

~/anaconda3/envs/envtest/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py in pre_dispatch(self)
    785             self.logger.log_hyperparams(self.lightning_module.hparams_initial)
    786             self.logger.log_graph(self.lightning_module)
--> 787             self.logger.save()
    788 
    789     def post_dispatch(self):

~/anaconda3/envs/envtest/lib/python3.7/site-packages/pytorch_lightning/utilities/distributed.py in wrapped_fn(*args, **kwargs)
     47     def wrapped_fn(*args, **kwargs):
     48         if rank_zero_only.rank == 0:
---> 49             return fn(*args, **kwargs)
     50 
     51     return wrapped_fn

~/anaconda3/envs/envtest/lib/python3.7/site-packages/pytorch_lightning/loggers/tensorboard.py in save(self)
    236         # save the metatags file if it doesn't exist and the log directory exists
    237         if self._fs.isdir(dir_path) and not self._fs.isfile(hparams_file):
--> 238             save_hparams_to_yaml(hparams_file, self.hparams)
    239 
    240     @rank_zero_only

~/anaconda3/envs/envtest/lib/python3.7/site-packages/pytorch_lightning/core/saving.py in save_hparams_to_yaml(config_yaml, hparams)
    398     for k, v in hparams.items():
    399         try:
--> 400             yaml.dump(v)
    401         except TypeError:
    402             warn(f"Skipping '{k}' parameter because it is not possible to safely dump to YAML.")

~/anaconda3/envs/envtest/lib/python3.7/site-packages/yaml/__init__.py in dump(data, stream, Dumper, **kwds)
    288     If stream is None, return the produced string instead.
    289     """
--> 290     return dump_all([data], stream, Dumper=Dumper, **kwds)
    291 
    292 def safe_dump_all(documents, stream=None, **kwds):

~/anaconda3/envs/envtest/lib/python3.7/site-packages/yaml/__init__.py in dump_all(documents, stream, Dumper, default_style, default_flow_style, canonical, indent, width, allow_unicode, line_break, encoding, explicit_start, explicit_end, version, tags, sort_keys)
    276         dumper.open()
    277         for data in documents:
--> 278             dumper.represent(data)
    279         dumper.close()
    280     finally:

~/anaconda3/envs/envtest/lib/python3.7/site-packages/yaml/representer.py in represent(self, data)
     25 
     26     def represent(self, data):
---> 27         node = self.represent_data(data)
     28         self.serialize(node)
     29         self.represented_objects = {}

~/anaconda3/envs/envtest/lib/python3.7/site-packages/yaml/representer.py in represent_data(self, data)
     50             for data_type in data_types:
     51                 if data_type in self.yaml_multi_representers:
---> 52                     node = self.yaml_multi_representers[data_type](self, data)
     53                     break
     54             else:

~/anaconda3/envs/envtest/lib/python3.7/site-packages/yaml/representer.py in represent_object(self, data)
    341                 and isinstance(state, dict) and newobj:
    342             return self.represent_mapping(
--> 343                     'tag:yaml.org,2002:python/object:'+function_name, state)
    344         if not listitems and not dictitems  \
    345                 and isinstance(state, dict) and not state:

~/anaconda3/envs/envtest/lib/python3.7/site-packages/yaml/representer.py in represent_mapping(self, tag, mapping, flow_style)
    116         for item_key, item_value in mapping:
    117             node_key = self.represent_data(item_key)
--> 118             node_value = self.represent_data(item_value)
    119             if not (isinstance(node_key, ScalarNode) and not node_key.style):
    120                 best_style = False

~/anaconda3/envs/envtest/lib/python3.7/site-packages/yaml/representer.py in represent_data(self, data)
     50             for data_type in data_types:
     51                 if data_type in self.yaml_multi_representers:
---> 52                     node = self.yaml_multi_representers[data_type](self, data)
     53                     break
     54             else:

~/anaconda3/envs/envtest/lib/python3.7/site-packages/yaml/representer.py in represent_object(self, data)
    341                 and isinstance(state, dict) and newobj:
    342             return self.represent_mapping(
--> 343                     'tag:yaml.org,2002:python/object:'+function_name, state)
    344         if not listitems and not dictitems  \
    345                 and isinstance(state, dict) and not state:

~/anaconda3/envs/envtest/lib/python3.7/site-packages/yaml/representer.py in represent_mapping(self, tag, mapping, flow_style)
    116         for item_key, item_value in mapping:
    117             node_key = self.represent_data(item_key)
--> 118             node_value = self.represent_data(item_value)
    119             if not (isinstance(node_key, ScalarNode) and not node_key.style):
    120                 best_style = False

~/anaconda3/envs/envtest/lib/python3.7/site-packages/yaml/representer.py in represent_data(self, data)
     50             for data_type in data_types:
     51                 if data_type in self.yaml_multi_representers:
---> 52                     node = self.yaml_multi_representers[data_type](self, data)
     53                     break
     54             else:

~/anaconda3/envs/envtest/lib/python3.7/site-packages/yaml/representer.py in represent_object(self, data)
    344         if not listitems and not dictitems  \
    345                 and isinstance(state, dict) and not state:
--> 346             return self.represent_sequence(tag+function_name, args)
    347         value = {}
    348         if args:

~/anaconda3/envs/envtest/lib/python3.7/site-packages/yaml/representer.py in represent_sequence(self, tag, sequence, flow_style)
     90         best_style = True
     91         for item in sequence:
---> 92             node_item = self.represent_data(item)
     93             if not (isinstance(node_item, ScalarNode) and not node_item.style):
     94                 best_style = False

~/anaconda3/envs/envtest/lib/python3.7/site-packages/yaml/representer.py in represent_data(self, data)
     46         data_types = type(data).__mro__
     47         if data_types[0] in self.yaml_representers:
---> 48             node = self.yaml_representers[data_types[0]](self, data)
     49         else:
     50             for data_type in data_types:

~/anaconda3/envs/envtest/lib/python3.7/site-packages/yaml/representer.py in represent_tuple(self, data)
    284 
    285     def represent_tuple(self, data):
--> 286         return self.represent_sequence('tag:yaml.org,2002:python/tuple', data)
    287 
    288     def represent_name(self, data):

~/anaconda3/envs/envtest/lib/python3.7/site-packages/yaml/representer.py in represent_sequence(self, tag, sequence, flow_style)
     90         best_style = True
     91         for item in sequence:
---> 92             node_item = self.represent_data(item)
     93             if not (isinstance(node_item, ScalarNode) and not node_item.style):
     94                 best_style = False

~/anaconda3/envs/envtest/lib/python3.7/site-packages/yaml/representer.py in represent_data(self, data)
     50             for data_type in data_types:
     51                 if data_type in self.yaml_multi_representers:
---> 52                     node = self.yaml_multi_representers[data_type](self, data)
     53                     break
     54             else:

~/anaconda3/envs/envtest/lib/python3.7/site-packages/yaml/representer.py in represent_object(self, data)
    329         if dictitems is not None:
    330             dictitems = dict(dictitems)
--> 331         if function.__name__ == '__newobj__':
    332             function = args[0]
    333             args = args[1:]

AttributeError: 'functools.partial' object has no attribute '__name__'

This is really strange, I thought it might be caused by version compatibility of pytorch-lightning and pytorch-forecasting (or tensorboard and PyYAML). I tried to use different version of pytorch-forecating, pytorch-lightning, but no progress.

Code to reproduce the problem

Just install the pytorch-forecasting and follow the turorial.

pip install pytorch-forecasting
pip install pyarrow

#os.chdir("../../..")   #only change in the tutorial code to avoid permission error.

Thank you!

Solution

Found the problem (although I don't know why). The problem lies in pandas version. I tested and found that pandas==1.2.x works fine but pandas==1.3.x will raise the error.
Now pip install pytorch-forecasting and pip install --upgrade pandas==1.2.5 is OK to run.

[seanyboi]

+1 having the same issue, python3.7.

[crayonfu]

It seems like a version issue with pyyaml. I got the same error as you, upgraded pyyaml from 5.3.1 to 5.4.1 and it is working fine. pytorch lightning 1.3.8, pytorch-forecasting 0.9.0, torch 1.9.0+cu111. Got it work in both python 3.7.11 and 3.8.10. Hope this helps.

[LumingSun]

It seems like a version issue with pyyaml. I got the same error as you, upgraded pyyaml from 5.3.1 to 5.4.1 and it is working fine. pytorch lightning 1.3.8, pytorch-forecasting 0.9.0, torch 1.9.0+cu111. Got it work in both python 3.7.11 and 3.8.10. Hope this helps.

Thank you, but the default pyyaml using pip install pytorch-forecasting is 5.4.1, and the problem still exists.

[azinflou]

Having the same issue, python3.8 pyyaml 5.4.1

[gdevos010]

I'm also having this issue:
PyTorch-Forecasting version: 0.9.0
PyTorch version: 1.9.0
Python version: 3.7
Operating System: windows 10

[crayonfu]

hmm, that's odd. Maybe I did something else fixed it other than the upgrade of pyyaml. I had a very hacky solution before, modifying pytorch_lightning's code directly. Then I reinstalled a new virtual env, upgraded pyyaml and it works. But you can try this hacky way if needed. Just add another except block to catch the error. Maybe you lose some logging by this, but it gets me pass the block. Looks like it has something to do with pyyaml's use of partial function?

Added a line for your_venv_folder/lib/python38/site_packages/pytorch_lightning/core/savings.py, line 400:

before:

try:
    yaml.dump(v)
except TypeError:
    warn(f"Skipping '{k}' parameter because it is not possible to safely dump to YAML.")
    hparams[k] = type(v).__name__

after

    yaml.dump(v)
except TypeError:
    warn(f"Skipping '{k}' parameter because it is not possible to safely dump to YAML.")
    hparams[k] = type(v).__name__
except AttributeError:
    warn(f"Skipping '{k}' parameter because it is not possible to safely dump to YAML.")
    hparams[k] = type(v).__name__

[LumingSun]

Found the problem (although I don't know why). The problem lies in pandas version. I tested and found that pandas==1.2.x works fine but pandas==1.3.x will raise the error.
Now pip install pytorch-forecasting and pip install --upgrade pandas==1.2.5 is OK to run.

[Nathan-Furnal]

Can we reopen this issue? I'm having the exact same problem and downgrading pandas causes another issue.

[LumingSun]

Can we reopen this issue? I'm having the exact same problem and downgrading pandas causes another issue.

What is the new issue? Is it related to pandas? Maybe it would be better to create a new issue.

[Nathan-Furnal]

I was getting an issue related to memory blocks because of conflicting versions, the downgrade did indeed work after I fixed the file I was using. But shouldn't this stay open anyways as it is not fixed?

[alexkyllo]

I am experiencing this issue with the following versions:

pytorch-lightning==1.4.9
pytorch-forecasting==0.9.1
pandas==1.3.4
pyyaml==6.0