Found multiple nodes named (name).

[antortjim]

Dear SLEAP developers

I am encountering this strange error whenever I try to run inference using an already trained model

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/home/vibflysleep/mambaforge/envs/sleap4/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/home/vibflysleep/mambaforge/envs/sleap4/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/vibflysleep/opt/sleap/sleap/nn/inference.py", line 1564, in _object_builder
    skeleton=skeleton,
  File "/home/vibflysleep/opt/sleap/sleap/instance.py", line 1176, in from_numpy
    points, point_confidences, instance_score, skeleton, track=track
  File "/home/vibflysleep/opt/sleap/sleap/instance.py", line 1116, in from_arrays
    track=track,
  File "<attrs generated init sleap.instance.PredictedInstance>", line 14, in __init__
    self.__attrs_post_init__()
  File "/home/vibflysleep/opt/sleap/sleap/instance.py", line 1011, in __attrs_post_init__
    super(PredictedInstance, self).__attrs_post_init__()
  File "/home/vibflysleep/opt/sleap/sleap/instance.py", line 456, in __attrs_post_init__
    Instance._points_dict_to_array(self._points, parray, self.skeleton)
  File "/home/vibflysleep/opt/sleap/sleap/instance.py", line 491, in _points_dict_to_array
    points = {skeleton.find_node(name): point for name, point in points.items()}
  File "/home/vibflysleep/opt/sleap/sleap/instance.py", line 491, in <dictcomp>
    points = {skeleton.find_node(name): point for name, point in points.items()}
  File "/home/vibflysleep/opt/sleap/sleap/skeleton.py", line 573, in find_node
    raise ValueError("Found multiple nodes named ({}).".format(name))
ValueError: Found multiple nodes named (name).

It seems that the nodes predictor.confmap_config.data.labels.skeletons[0] are all called name. I browsed the code and I could trace back the generation of this config to sleap.nn.config.training_job.TrainingJobConfig.from_json which ultimately, is calling cattr.structure(json_data_dicts, cls). I made a quick MWE to show this

import json
import cattr
from sleap.nn.config.training_job import TrainingJobConfig

config_file = "models/230720_093109.single_instance.n=306/training_config.json"

with open(config_file, "r") as handle:
    json_data_dicts = json.load(handle)

cattr.structure(json_data_dicts, TrainingJobConfig).data.labels.skeletons
# Skeleton(name='Skeleton-3', description='None', nodes=['name', 'name', 'name', 'name', 'name', 'name', 'name', 'name', 'name', 'name', 'name', 'name'], edges=[('name', 'name'), ('name', 'name'), ('name', 'name'), ('name', 'name'), ('name', 'name'), ('name', 'name'), ('name', 'name'), ('name', 'name'), ('name', 'name'), ('name', 'name'), ('name', 'name')], symmetries=[])]

Indeed, I see all nodes get assigned the name name even though they have the expected names in the training_config.zip.

I think this might be an issue with some inner workings of the cattr library, in particular, I think I may have installed a version with a behavior that breaks SLEAP (see #1469 ). Or is this output expected, and the node names get assigned later on in the code? Or maybe the config file is corrupted, but I don't think so because it used to work in the past.
Thank you!

PS On a closer look, could this be explained by idx_to_node being set to None when running make_cattr? Maybe that makes this line https://github.com/talmolab/sleap/blob/develop/sleap/skeleton.py#L257 have the unexpected behavior.

[roomrys]

jsonpickle.decode loses Node names

I am still investigating, but it seems the problem is with jsonpickle.decode

sleap/sleap/skeleton.py

Line 1074 in 0ef52cd

dicts = jsonpickle.decode(json_str)

which is taking in the expected json string json_str

'{"directed": true, "graph": {"name": "Skeleton-3", "num_edges_inserted": 11}, "links": [{"edge_insert_idx": 0, "key": 0, "source": {"py/object": "sleap.skeleton.Node", "py/state": {"name": "proboscis", "weight": 1.0}}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"name": "head", "weight": 1.0}}, "type": {"py/reduce": [{"py/type": "sleap.skeleton.EdgeType"}, {"py/tuple": [1]}]}}, {"edge_insert_idx": 1, "key": 0, "source": {"py/object": "sleap.skeleton.Node", "py/state": {"name": "thorax", "weight": 1.0}}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"name": "foreLeft_Leg", "weight": 1.0}}, "type": {"py/id": 3}}, {"edge_insert_idx": 2, "key": 0, "source": {"py/id": 4}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"name": "midLeftLeg", "weight": 1.0}}, "type": {"py/id": 3}}, {"edge_insert_idx": 3, "key": 0, "source": {"py/id": 4}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"name": "rearLeftLeg", "weight": 1.0}}, "type": {"py/id": 3}}, {"edge_insert_idx": 4, "key": 0, "source": {"py/id": 4}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"name": "foreRightLeg", "weight": 1.0}}, "type": {"py/id": 3}}, {"edge_insert_idx": 5, "key": 0, "source": {"py/id": 4}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"name": "midRightLeg", "weight": 1.0}}, "type": {"py/id": 3}}, {"edge_insert_idx": 6, "key": 0, "source": {"py/id": 4}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"name": "rearRightLeg", "weight": 1.0}}, "type": {"py/id": 3}}, {"edge_insert_idx": 7, "key": 0, "source": {"py/id": 4}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"name": "abdomen", "weight": 1.0}}, "type": {"py/id": 3}}, {"edge_insert_idx": 8, "key": 0, "source": {"py/id": 4}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"name": "leftWing", "weight": 1.0}}, "type": {"py/id": 3}}, {"edge_insert_idx": 9, "key": 0, "source": {"py/id": 4}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"name": "rightWing", "weight": 1.0}}, "type": {"py/id": 3}}, {"edge_insert_idx": 10, "key": 0, "source": {"py/id": 4}, "target": {"py/id": 2}, "type": {"py/id": 3}}], "multigraph": true, "nodes": [{"id": {"py/id": 1}}, {"id": {"py/id": 4}}, {"id": {"py/id": 11}}, {"id": {"py/id": 5}}, {"id": {"py/id": 6}}, {"id": {"py/id": 7}}, {"id": {"py/id": 8}}, {"id": {"py/id": 9}}, {"id": {"py/id": 10}}, {"id": {"py/id": 2}}, {"id": {"py/id": 12}}, {"id": {"py/id": 13}}]}'

and returning a dictionary dict with Node names of "name" (the jsonpickle.encoded string below):

'{"directed": true, "graph": {"name": "Skeleton-3", "num_edges_inserted": 11}, "links": [{"edge_insert_idx": 0, "key": 0, "source": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["name", "weight"]}}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["name", "weight"]}}, "type": {"py/reduce": [{"py/type": "sleap.skeleton.EdgeType"}, {"py/tuple": [1]}]}}, {"edge_insert_idx": 1, "key": 0, "source": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["name", "weight"]}}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["name", "weight"]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 2, "key": 0, "source": {"py/id": 4}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["name", "weight"]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 3, "key": 0, "source": {"py/id": 4}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["name", "weight"]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 4, "key": 0, "source": {"py/id": 4}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["name", "weight"]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 5, "key": 0, "source": {"py/id": 4}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["name", "weight"]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 6, "key": 0, "source": {"py/id": 4}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["name", "weight"]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 7, "key": 0, "source": {"py/id": 4}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["name", "weight"]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 8, "key": 0, "source": {"py/id": 4}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["name", "weight"]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 9, "key": 0, "source": {"py/id": 4}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["name", "weight"]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 10, "key": 0, "source": {"py/id": 4}, "target": {"py/id": 2}, "type": {"py/id": 3}}], "multigraph": true, "nodes": [{"id": {"py/id": 1}}, {"id": {"py/id": 4}}, {"id": {"py/id": 11}}, {"id": {"py/id": 5}}, {"id": {"py/id": 6}}, {"id": {"py/id": 7}}, {"id": {"py/id": 8}}, {"id": {"py/id": 9}}, {"id": {"py/id": 10}}, {"id": {"py/id": 2}}, {"id": {"py/id": 12}}, {"id": {"py/id": 13}}]}'

Is the use of jsonpickle.decode new? No.

The blame shows this line as fairly recent, but futher digging revealed that we have been using jsonpickle.decode for a long time:
https://github.com/talmolab/sleap/blame/57ee08a7490aa950f61b14f80fa8e2d372a2ca5c/sleap/skeleton.py#L1027

        graph = json_graph.node_link_graph(jsonpickle.decode(json_str))

so perhaps it is a version change that triggered this issue?

Could it be a jsonpickle version change? No.

Nope, not a version change as @antortjim and I are both using jonspickle==1.2 (albeit mine is conda-distributed) and so does our previous dependency (pypi-distributed)
https://github.com/talmolab/sleap/blame/1a0404c0ffae7b248eb360562b0bb95a42a287b6/requirements.txt#L4

jsonpickle==1.2

Is it a change in the write to json_str? Yes.

So perhaps a change in the write format? Another json_str last edited 11/23/2022:

'{"directed": true, "graph": {"name": "Skeleton-0", "num_edges_inserted": 46}, "links": [{"edge_insert_idx": 44, "key": 0, "source": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["head", 1.0]}}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["eyeL", 1.0]}}, "type": {"py/reduce": [{"py/type": "sleap.skeleton.EdgeType"}, {"py/tuple": [1]}]}}, {"edge_insert_idx": 45, "key": 0, "source": {"py/id": 1}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["eyeR", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 34, "key": 0, "source": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["thorax", 1.0]}}, "target": {"py/id": 1}, "type": {"py/id": 3}}, {"edge_insert_idx": 35, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["abdomen", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 36, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["wingL", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 37, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["wingR", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 38, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["forelegL4", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 39, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["forelegR4", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 40, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["midlegL4", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 41, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["midlegR4", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 42, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["hindlegL4", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 43, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["hindlegR4", 1.0]}}, "type": {"py/id": 3}}], "multigraph": true, "nodes": [{"id": {"py/id": 1}}, {"id": {"py/id": 5}}, {"id": {"py/id": 6}}, {"id": {"py/id": 7}}, {"id": {"py/id": 8}}, {"id": {"py/id": 9}}, {"id": {"py/id": 10}}, {"id": {"py/id": 11}}, {"id": {"py/id": 12}}, {"id": {"py/id": 13}}, {"id": {"py/id": 14}}, {"id": {"py/id": 2}}, {"id": {"py/id": 4}}]}'

There are some differences here! And the re-encoded string shows the correct Node names:

'{"directed": true, "graph": {"name": "Skeleton-0", "num_edges_inserted": 46}, "links": [{"edge_insert_idx": 44, "key": 0, "source": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["head", 1.0]}}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["eyeL", 1.0]}}, "type": {"py/reduce": [{"py/type": "sleap.skeleton.EdgeType"}, {"py/tuple": [1]}]}}, {"edge_insert_idx": 45, "key": 0, "source": {"py/id": 1}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["eyeR", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 34, "key": 0, "source": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["thorax", 1.0]}}, "target": {"py/id": 1}, "type": {"py/id": 3}}, {"edge_insert_idx": 35, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["abdomen", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 36, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["wingL", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 37, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["wingR", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 38, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["forelegL4", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 39, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["forelegR4", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 40, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["midlegL4", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 41, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["midlegR4", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 42, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["hindlegL4", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 43, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["hindlegR4", 1.0]}}, "type": {"py/id": 3}}], "multigraph": true, "nodes": [{"id": {"py/id": 1}}, {"id": {"py/id": 5}}, {"id": {"py/id": 6}}, {"id": {"py/id": 7}}, {"id": {"py/id": 8}}, {"id": {"py/id": 9}}, {"id": {"py/id": 10}}, {"id": {"py/id": 11}}, {"id": {"py/id": 12}}, {"id": {"py/id": 13}}, {"id": {"py/id": 14}}, {"id": {"py/id": 2}}, {"id": {"py/id": 4}}]}'

Ok... so who done it (probably me 🐑) and where? Some culprits are #1122 and #1222 (which attempted to fix #1122, but may not have done so thoroughly?).

[antortjim]

In case just training again using the same dataset with the current version fixes the issue, I thought of just doing that. However, I am unfortunately getting the same errors I mentioned in #1469

last logs of the training in the terminal

...
INFO:sleap.nn.training:Finished creating training datasets. [7.0s]
INFO:sleap.nn.training:Starting training loop...
Epoch 1/200
2023-08-25 17:59:02.251101: I tensorflow/stream_executor/cuda/cuda_dnn.cc:368] Loaded cuDNN version 8902
2023-08-25 17:59:09.675049: I tensorflow/stream_executor/cuda/cuda_blas.cc:1786] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.

2023-08-25 17:59:10.089052: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 12 } dim { size: 128 } dim { size: 128 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "GPU" vendor: "NVIDIA" model: "NVIDIA RTX A6000" frequency: 1800 num_cores: 84 environment { key: "architecture" value: "8.6" } environment { key: "cuda" value: "11020" } environment { key: "cudnn" value: "8100" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 6291456 shared_memory_size_per_multiprocessor: 102400 memory_size: 47869329408 bandwidth: 768096000 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: -5 } dim { size: -6 } dim { size: 1 } } }
QObject::moveToThread: Current thread (0x560af0299f20) is not the object's thread (0x560af6cbd860).
Cannot move to target thread (0x560af0299f20)

qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "/home/vibflysleep/mambaforge/envs/sleap4/lib/python3.10/site-packages/cv2/qt/plugins" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: xcb, eglfs, minimal, minimalegl, offscreen, vnc, webgl.

[antortjim]

OK no, no problem, I can retrain if I use the environment exactly as specified in #1469 (comment)

[antortjim]

Training again creates a new training config file which does not exhibit this issue anymore

[roomrys]

Hi @antortjim,

Argh still bad for backwards compatibility and should have been depicted by a minor release, but glad it is not a problem for new trainings. Still seeing how we can support older models - do you happen to know which release/when the now-broken training config was created?

Thanks,
Liezl

[antortjim]

I think it was the version in this commit d7c9505 (April 27)

[antortjim]

Thank you for all the help @roomrys :) 🙏

[roomrys]

In the following discussion:

• Error during Inference. ValueError: Found Multiple nodes named (name). #1918

we found that the underlying problem comes from an upgrade to attr>=22.2.0 where the __getstate__ method of attr classes is changed to return a dict instead of a tuple. Then, when encoding the Skeleton using jsonpickle, jsonpickle instead encodes the "py/state" as a dict.

This is not a problem in itself IF staying on the same/compatible version of attr, but when attempting to use this newly encoded skeleton.json in a different SLEAP installation that perhaps resolved to an older attr<22.2.0, the Skeleton is read in incorrectly, taking the dictionary "py/state" keys as the attribute values (instead of the attribute names). Hence the error we see with Found multiple nodes named (name) since the Skeleton quite literally was reconstructed with multiple nodes all named name.