Experiencing issues with potential training #68

ligerzero-ai · 2023-04-02T04:21:49Z

ligerzero-ai
Apr 2, 2023

Hi All,

I've been experiencing some issues while trying to train a toy model on my laptop using my own data:
MnGB.zip

import pandas as pd
df_train = pd.DataFrame.from_pickle("MnGB.pkl")
from sklearn.model_selection import train_test_split
train_test_0, validation_0 = train_test_split(df_train, train_size=0.8)
train_0, test_0 = train_test_split(train_test_0, train_size=0.8)

from m3gnet.models import M3GNet, Potential
from m3gnet.trainers import PotentialTrainer

import tensorflow as tf

m3gnet = M3GNet(is_intensive=False)
potential = Potential(model=m3gnet)

trainer = PotentialTrainer(
    potential=potential, optimizer=tf.keras.optimizers.Adam(1e-3)
)

trainer.train(
    train_0.structures.values,
    train_0.energies.values,
    train_0.forces.values,
    # stresses,
    validation_graphs_or_structures=validation_0.structures.values,
    val_energies=validation_0.energies.values,
    val_forces=validation_0.forces.values,
    # val_stresses=val_stresses,
    epochs=100,
    fit_per_element_offset=True,
    save_checkpoint=False,
    batch_size=2
)

Was wondering if anyone could tell at a glance what is happening that's causing this to throw this error?

I've experimented a little bit and it's happening no matter what subset of my data that I take. (small, large, etc.)

It's happening on the first epoch. Sometimes it happens early (step 10ish) sometimes it happens later (step 50ish).

I suspect that it maybe GPU memory related, but I am not sure.

The structures are of grain boundaries with surfaces at the top and bottom of the cell.

The stacktrace on a NVIDIA 3070 Ti Laptop GPU:

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-04-01 20:59:03.905045: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory
2023-04-01 20:59:03.905282: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory
2023-04-01 20:59:03.905288: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
Epoch 1/10
2023-04-01 21:04:38.577876: I tensorflow/compiler/xla/service/service.cc:173] XLA service 0xa0731150 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2023-04-01 21:04:38.577950: I tensorflow/compiler/xla/service/service.cc:181]   StreamExecutor device (0): NVIDIA GeForce RTX 3070 Ti Laptop GPU, Compute Capability 8.6
2023-04-01 21:04:38.614770: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
2023-04-01 21:04:38.962004: I tensorflow/compiler/jit/xla_compilation_cache.cc:477] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.
WARNING:tensorflow:5 out of the last 5 calls to <function _BaseOptimizer._update_step_xla at 0x7f36bcf9fee0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:5 out of the last 5 calls to <function _BaseOptimizer._update_step_xla at 0x7f36bcf9fee0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:6 out of the last 6 calls to <function _BaseOptimizer._update_step_xla at 0x7f36bcf9fee0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:6 out of the last 6 calls to <function _BaseOptimizer._update_step_xla at 0x7f36bcf9fee0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
     76/Unknown - 57s 368ms/step - loss: 0.0144 - MAE(E): 0.0776 - MAE(F): 0.0060 - MAE(S): 0.0000e+00
2023-04-01 21:05:13.520419: W tensorflow/core/framework/op_kernel.cc:1818] INVALID_ARGUMENT: required broadcastable shapes
2023-04-01 21:05:13.520585: W tensorflow/core/framework/op_kernel.cc:1818] INVALID_ARGUMENT: required broadcastable shapes
---------------------------------------------------------------------------

InvalidArgumentError                      Traceback (most recent call last)
Cell In[15], line 1
----> 1 trainer.train(
      2     train_0.structures.values,
      3     train_0.energies.values,
      4     train_0.forces.values,
      5     # stresses,
      6     validation_graphs_or_structures=validation_0.structures.values,
      7     val_energies=validation_0.energies.values,
      8     val_forces=validation_0.forces.values,
      9     # val_stresses=val_stresses,
     10     epochs=10,
     11     fit_per_element_offset=True,
     12     save_checkpoint=False,
     13     batch_size=2
     14 )

File [~/miniconda3/envs/m3gnet/lib/python3.9/site-packages/m3gnet/trainers/_potential.py:210](https://vscode-remote+wsl-002bubuntu-002d22-002e04.vscode-resource.vscode-cdn.net/root/Fe_Ptable_ML/~/miniconda3/envs/m3gnet/lib/python3.9/site-packages/m3gnet/trainers/_potential.py:210), in PotentialTrainer.train(self, graphs_or_structures, energies, forces, stresses, validation_graphs_or_structures, val_energies, val_forces, val_stresses, loss, force_loss_ratio, stress_loss_ratio, batch_size, epochs, callbacks, save_checkpoint, early_stop_patience, verbose, fit_per_element_offset)
    208 callback_list.on_batch_begin(batch=batch_index)
    209 graph_batch, target_batch = batch
--> 210 lossval, grads, pred_list, emae, fmae, smae = train_one_step(
    211     self.potential, graph_batch.as_tf().as_list(), target_batch
    212 )
    214 self.optimizer.apply_gradients(zip(grads, self.potential.trainable_variables))
    215 epoch_loss_avg.update_state(lossval)

File [~/miniconda3/envs/m3gnet/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py:153](https://vscode-remote+wsl-002bubuntu-002d22-002e04.vscode-resource.vscode-cdn.net/root/Fe_Ptable_ML/~/miniconda3/envs/m3gnet/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py:153), in filter_traceback..error_handler(*args, **kwargs)
    151 except Exception as e:
    152   filtered_tb = _process_traceback_frames(e.__traceback__)
--> 153   raise e.with_traceback(filtered_tb) from None
    154 finally:
    155   del filtered_tb

File [~/miniconda3/envs/m3gnet/lib/python3.9/site-packages/tensorflow/python/eager/execute.py:52](https://vscode-remote+wsl-002bubuntu-002d22-002e04.vscode-resource.vscode-cdn.net/root/Fe_Ptable_ML/~/miniconda3/envs/m3gnet/lib/python3.9/site-packages/tensorflow/python/eager/execute.py:52), in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     50 try:
     51   ctx.ensure_initialized()
---> 52   tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
     53                                       inputs, attrs, num_outputs)
     54 except core._NotOkStatusException as e:
     55   if name is not None:

InvalidArgumentError: Graph execution error:

Detected at node 'sub_1' defined at (most recent call last):
    File "/root/miniconda3/envs/m3gnet/lib/python3.9/runpy.py", line 197, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/root/miniconda3/envs/m3gnet/lib/python3.9/runpy.py", line 87, in _run_code
      exec(code, run_globals)
    File "/root/miniconda3/envs/m3gnet/lib/python3.9/site-packages/ipykernel_launcher.py", line 17, in 
      app.launch_new_instance()
    File "/root/miniconda3/envs/m3gnet/lib/python3.9/site-packages/traitlets/config/application.py", line 1043, in launch_instance
      app.start()
    File "/root/miniconda3/envs/m3gnet/lib/python3.9/site-packages/ipykernel/kernelapp.py", line 712, in start
      self.io_loop.start()
    File "/root/miniconda3/envs/m3gnet/lib/python3.9/site-packages/tornado/platform/asyncio.py", line 199, in start
      self.asyncio_loop.run_forever()
    File "/root/miniconda3/envs/m3gnet/lib/python3.9/asyncio/base_events.py", line 601, in run_forever
      self._run_once()
    File "/root/miniconda3/envs/m3gnet/lib/python3.9/asyncio/base_events.py", line 1905, in _run_once
      handle._run()
    File "/root/miniconda3/envs/m3gnet/lib/python3.9/asyncio/events.py", line 80, in _run
      self._context.run(self._callback, *self._args)
    File "/root/miniconda3/envs/m3gnet/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 510, in dispatch_queue
      await self.process_one()
    File "/root/miniconda3/envs/m3gnet/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 499, in process_one
      await dispatch(*args)
    File "/root/miniconda3/envs/m3gnet/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 406, in dispatch_shell
      await result
    File "/root/miniconda3/envs/m3gnet/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 730, in execute_request
      reply_content = await reply_content
    File "/root/miniconda3/envs/m3gnet/lib/python3.9/site-packages/ipykernel/ipkernel.py", line 383, in do_execute
      res = shell.run_cell(
    File "/root/miniconda3/envs/m3gnet/lib/python3.9/site-packages/ipykernel/zmqshell.py", line 528, in run_cell
      return super().run_cell(*args, **kwargs)
    File "/root/miniconda3/envs/m3gnet/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2961, in run_cell
      result = self._run_cell(
    File "/root/miniconda3/envs/m3gnet/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3016, in _run_cell
      result = runner(coro)
    File "/root/miniconda3/envs/m3gnet/lib/python3.9/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
      coro.send(None)
    File "/root/miniconda3/envs/m3gnet/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3221, in run_cell_async
      has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
    File "/root/miniconda3/envs/m3gnet/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3400, in run_ast_nodes
      if await self.run_code(code, result, async_=asy):
    File "/root/miniconda3/envs/m3gnet/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3460, in run_code
      exec(code_obj, self.user_global_ns, self.user_ns)
    File "/tmp/ipykernel_416/3518045778.py", line 1, in 
      trainer.train(
    File "/root/miniconda3/envs/m3gnet/lib/python3.9/site-packages/m3gnet/trainers/_potential.py", line 210, in train
      lossval, grads, pred_list, emae, fmae, smae = train_one_step(
    File "/root/miniconda3/envs/m3gnet/lib/python3.9/site-packages/m3gnet/trainers/_potential.py", line 192, in train_one_step
      loss_val, emae, fmae, smae = _loss(target_list, pred_list, graph_list[Index.N_ATOMS])
    File "/root/miniconda3/envs/m3gnet/lib/python3.9/site-packages/m3gnet/trainers/_potential.py", line 142, in _loss
      f_metric = _mae(target_batch[1], graph_pred_batch[1])
    File "/root/miniconda3/envs/m3gnet/lib/python3.9/site-packages/m3gnet/trainers/_potential.py", line 132, in _mae
      return tf.reduce_mean(tf.math.abs(x - y))
Node: 'sub_1'
required broadcastable shapes
	 [[{{node sub_1}}]] [Op:__inference_train_one_step_93433]

ligerzero-ai · 2023-04-02T06:44:11Z

ligerzero-ai
Apr 2, 2023
Author

update: I experience the same (required broadcastable shapes) error on a HPC configuration with a V100 GPU as well - so it doesn't seem to be a memory issue.

python3 m3gnet_testdataset.py
2023-04-02 16:58:20.163298: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX512F
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-04-02 16:58:21.189645: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 30955 MB memory:  -> device: 0, name: Tesla V100-SXM2-32GB, pci bus id: 0000:b1:00.0, compute capability: 7.0
Epoch 1/100
/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("PartitionedCall:1", shape=(14448,), dtype=int32), values=Tensor("Neg:0", shape=(14448, 3), dtype=float32), dense_shape=Tensor("PartitionedCall:2", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/gradients/m3g_net/three_d_interaction/UnsortedSegmentSum_grad/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/gradients/m3g_net/three_d_interaction/UnsortedSegmentSum_grad/GatherV2_grad/Reshape:0", shape=(None, 9), dtype=float32), dense_shape=Tensor("gradients/gradients/m3g_net/three_d_interaction/UnsortedSegmentSum_grad/GatherV2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/gradients/m3g_net/three_d_interaction_1/UnsortedSegmentSum_grad/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/gradients/m3g_net/three_d_interaction_1/UnsortedSegmentSum_grad/GatherV2_grad/Reshape:0", shape=(None, 9), dtype=float32), dense_shape=Tensor("gradients/gradients/m3g_net/three_d_interaction_1/UnsortedSegmentSum_grad/GatherV2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/gradients/m3g_net/three_d_interaction_2/UnsortedSegmentSum_grad/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/gradients/m3g_net/three_d_interaction_2/UnsortedSegmentSum_grad/GatherV2_grad/Reshape:0", shape=(None, 9), dtype=float32), dense_shape=Tensor("gradients/gradients/m3g_net/three_d_interaction_2/UnsortedSegmentSum_grad/GatherV2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
      1/Unknown - 22s 22s/step - loss: 0.0011 - MAE(E): 0.0278 - MAE(F): 0.0069 - MAE(S): 0.0000e+00/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("PartitionedCall:1", shape=(14428,), dtype=int32), values=Tensor("Neg:0", shape=(14428, 3), dtype=float32), dense_shape=Tensor("PartitionedCall:2", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
      2/Unknown - 39s 18s/step - loss: 9.9813e-04 - MAE(E): 0.0247 - MAE(F): 0.0074 - MAE(S): 0.0000e+00/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/concat_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/concat:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradients/m3g_net/three_d_interaction_2/GatherV2_2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/GatherV2_5_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/GatherV2_5_grad/Reshape:0", shape=(None,), dtype=float32), dense_shape=Tensor("gradients/m3g_net/GatherV2_5_grad/Cast:0", shape=(1,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/GatherV2_6_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/GatherV2_6_grad/Reshape:0", shape=(None,), dtype=float32), dense_shape=Tensor("gradients/m3g_net/GatherV2_6_grad/Cast:0", shape=(1,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/GatherV2_3_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/GatherV2_3_grad/Reshape:0", shape=(None, 3), dtype=float32), dense_shape=Tensor("gradients/m3g_net/GatherV2_3_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/GatherV2_4_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/GatherV2_4_grad/Reshape:0", shape=(None, 3), dtype=float32), dense_shape=Tensor("gradients/m3g_net/GatherV2_4_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/spherical_bessel_with_harmonics/cond_grad/gradients/m3g_net/spherical_bessel_with_harmonics/cond/Repeat/boolean_mask/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int64), values=Tensor("gradients/m3g_net/spherical_bessel_with_harmonics/cond_grad/gradients/m3g_net/spherical_bessel_with_harmonics/cond/Repeat/boolean_mask/GatherV2_grad/Reshape:0", shape=(None,), dtype=float32), dense_shape=Tensor("gradients/m3g_net/spherical_bessel_with_harmonics/cond_grad/gradients/m3g_net/spherical_bessel_with_harmonics/cond/Repeat/boolean_mask/GatherV2_grad/Cast:0", shape=(1,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/three_d_interaction_2/GatherV2_2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/three_d_interaction_2/GatherV2_2_grad/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradients/m3g_net/three_d_interaction_2/GatherV2_2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/three_d_interaction_1/GatherV2_2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/three_d_interaction_1/GatherV2_2_grad/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradients/m3g_net/three_d_interaction_1/GatherV2_2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/three_d_interaction/GatherV2_2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/three_d_interaction/GatherV2_2_grad/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradients/m3g_net/three_d_interaction/GatherV2_2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("PartitionedCall:1", shape=(None,), dtype=int32), values=Tensor("Neg:0", shape=(None, 3), dtype=float32), dense_shape=Tensor("PartitionedCall:2", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
     15/Unknown - 63s 3s/step - loss: 4.7272e-04 - MAE(E): 0.0137 - MAE(F): 0.0059 - MAE(S): 0.0000e+00/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/three_d_interaction_2/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/three_d_interaction_2/GatherV2_1_grad/Reshape:0", shape=(None, 9), dtype=float32), dense_shape=Tensor("gradients/m3g_net/three_d_interaction_2/GatherV2_1_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_grad/Reshape:0", shape=(None, 64), dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_1_grad/Reshape:0", shape=(None, 64), dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_1_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_2/concat_atoms_2/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_2/concat_atoms_2/GatherV2_grad/Reshape:0", shape=(None, 64), dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_2/concat_atoms_2/GatherV2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/three_d_interaction_1/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/three_d_interaction_1/GatherV2_1_grad/Reshape:0", shape=(None, 9), dtype=float32), dense_shape=Tensor("gradients/m3g_net/three_d_interaction_1/GatherV2_1_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_grad/Reshape:0", shape=(None, 64), dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_1_grad/Reshape:0", shape=(None, 64), dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_1_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_1/concat_atoms_1/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_1/concat_atoms_1/GatherV2_grad/Reshape:0", shape=(None, 64), dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_1/concat_atoms_1/GatherV2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/three_d_interaction/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/three_d_interaction/GatherV2_1_grad/Reshape:0", shape=(None, 9), dtype=float32), dense_shape=Tensor("gradients/m3g_net/three_d_interaction/GatherV2_1_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer/gated_atom_update/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer/gated_atom_update/GatherV2_grad/Reshape:0", shape=(None, 64), dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer/gated_atom_update/GatherV2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer/gated_atom_update/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer/gated_atom_update/GatherV2_1_grad/Reshape:0", shape=(None, 64), dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer/gated_atom_update/GatherV2_1_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer/concat_atoms/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer/concat_atoms/GatherV2_grad/Reshape:0", shape=(None, 64), dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer/concat_atoms/GatherV2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/GatherV2_2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/GatherV2_2_grad/Reshape:0", shape=(None, 3), dtype=float32), dense_shape=Tensor("gradients/m3g_net/GatherV2_2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
  warnings.warn(
/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/GatherV2_1_grad/Reshape:0", shape=(None, 3), dtype=float32), dense_shape=Tensor("gradients/m3g_net/GatherV2_1_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
     23/Unknown - 89s 3s/step - loss: 4.0682e-04 - MAE(E): 0.0121 - MAE(F): 0.0057 - MAE(S): 0.0000e+002023-04-02 16:59:54.570450: W tensorflow/core/framework/op_kernel.cc:1733] INVALID_ARGUMENT: required broadcastable shapes
2023-04-02 16:59:54.570735: W tensorflow/core/framework/op_kernel.cc:1733] INVALID_ARGUMENT: required broadcastable shapes
Traceback (most recent call last):
  File "/g/data/v43/Han/Data/FeGBProject_m3gnet/m3gnet_testdataset.py", line 54, in <module>
    trainer.train(
  File "/g/data/v43/Han/python/lib/python3.10/site-packages/m3gnet/trainers/_potential.py", line 210, in train
    lossval, grads, pred_list, emae, fmae, smae = train_one_step(
  File "/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/apps/tensorflow/2.8.0/lib/python3.10/site-packages/tensorflow/python/eager/execute.py", line 54, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError: Graph execution error:

Detected at node 'sub_1' defined at (most recent call last):
    File "/g/data/v43/Han/Data/FeGBProject_m3gnet/m3gnet_testdataset.py", line 54, in <module>
      trainer.train(
    File "/g/data/v43/Han/python/lib/python3.10/site-packages/m3gnet/trainers/_potential.py", line 210, in train
      lossval, grads, pred_list, emae, fmae, smae = train_one_step(
    File "/g/data/v43/Han/python/lib/python3.10/site-packages/m3gnet/trainers/_potential.py", line 192, in train_one_step
      loss_val, emae, fmae, smae = _loss(target_list, pred_list, graph_list[Index.N_ATOMS])
    File "/g/data/v43/Han/python/lib/python3.10/site-packages/m3gnet/trainers/_potential.py", line 142, in _loss
      f_metric = _mae(target_batch[1], graph_pred_batch[1])
    File "/g/data/v43/Han/python/lib/python3.10/site-packages/m3gnet/trainers/_potential.py", line 132, in _mae
      return tf.reduce_mean(tf.math.abs(x - y))
Node: 'sub_1'
required broadcastable shapes

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Experiencing issues with potential training #68

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Experiencing issues with potential training #68

ligerzero-ai Apr 2, 2023

Replies: 1 comment

ligerzero-ai Apr 2, 2023 Author

ligerzero-ai
Apr 2, 2023

ligerzero-ai
Apr 2, 2023
Author