Need help understanding the error trace

[ParikhKadam]

The error trace shown is quite complex and I am unable to understand it. Also, I wished to try the other way by printing shapes of tensors in every layer at the time of running the model but I found no function in keras which helps me do that.

Can anyone help me understand the error trace so that I can find a solution to remove the errors? Also, can anyone provide a trick to print shapes of tensors at model runtime?

Here's my model - https://github.com/ParikhKadam/bidaf-keras

Here's the error trace when calling model.fit_generator(train_generator, steps_per_epoch=1, epochs=1, validation_data=validate_generator, validation_steps=1, shuffle=False):

Epoch 1/1
---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
<ipython-input-24-ffa5706fca38> in <module>()
----> 1 model.fit_generator(train_generator, steps_per_epoch=1, epochs=1, validation_data=validate_generator, validation_steps=1, shuffle=False)

C:\Python36\lib\site-packages\keras\legacy\interfaces.py in wrapper(*args, **kwargs)
     89                 warnings.warn('Update your `' + object_name + '` call to the ' +
     90                               'Keras 2 API: ' + signature, stacklevel=2)
---> 91             return func(*args, **kwargs)
     92         wrapper._original_function = func
     93         return wrapper

C:\Python36\lib\site-packages\keras\engine\training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
   1416             use_multiprocessing=use_multiprocessing,
   1417             shuffle=shuffle,
-> 1418             initial_epoch=initial_epoch)
   1419 
   1420     @interfaces.legacy_generator_methods_support

C:\Python36\lib\site-packages\keras\engine\training_generator.py in fit_generator(model, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
    215                 outs = model.train_on_batch(x, y,
    216                                             sample_weight=sample_weight,
--> 217                                             class_weight=class_weight)
    218 
    219                 outs = to_list(outs)

C:\Python36\lib\site-packages\keras\engine\training.py in train_on_batch(self, x, y, sample_weight, class_weight)
   1215             ins = x + y + sample_weights
   1216         self._make_train_function()
-> 1217         outputs = self.train_function(ins)
   1218         return unpack_singleton(outputs)
   1219 

C:\Python36\lib\site-packages\keras\backend\tensorflow_backend.py in __call__(self, inputs)
   2713                 return self._legacy_call(inputs)
   2714 
-> 2715             return self._call(inputs)
   2716         else:
   2717             if py_any(is_tensor(x) for x in inputs):

C:\Python36\lib\site-packages\keras\backend\tensorflow_backend.py in _call(self, inputs)
   2673             fetched = self._callable_fn(*array_vals, run_metadata=self.run_metadata)
   2674         else:
-> 2675             fetched = self._callable_fn(*array_vals)
   2676         return fetched[:len(self.outputs)]
   2677 

C:\Python36\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs)
   1380           ret = tf_session.TF_SessionRunCallable(
   1381               self._session._session, self._handle, args, status,
-> 1382               run_metadata_ptr)
   1383         if run_metadata:
   1384           proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

C:\Python36\lib\site-packages\tensorflow\python\framework\errors_impl.py in __exit__(self, type_arg, value_arg, traceback_arg)
    517             None, None,
    518             compat.as_text(c_api.TF_Message(self.status.status)),
--> 519             c_api.TF_GetCode(self.status.status))
    520     # Delete the underlying status object from memory otherwise it stays alive
    521     # as there is a reference to status from this from the traceback due to

InvalidArgumentError: slice index 49 of dimension 0 out of bounds.
	 [[Node: metrics/negative_avg_log_error/map/while/strided_slice_1 = StridedSlice[Index=DT_INT32, T=DT_FLOAT, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1, _device="/job:localhost/replica:0/task:0/device:GPU:0"](metrics/negative_avg_log_error/map/while/TensorArrayReadV3_1, metrics/negative_avg_log_error/map/while/strided_slice_1/stack, metrics/negative_avg_log_error/map/while/strided_slice_1/stack_1, metrics/negative_avg_log_error/map/while/strided_slice/stack_1)]]
	 [[Node: training/Adadelta/gradients/query_to_context_attention/ExpandDims_grad/Reshape/_243 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_9091_...ad/Reshape", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Also, I don't know why but the error printed in command prompt is different than what I copied above (it's from jupyter notebook). The error printed on command prompt is as below:

2019-01-04 19:26:17.634769: W T:\src\github\tensorflow\tensorflow\core\framework\op_kernel.cc:1275] OP_REQUIRES failed at strided_slice_op.cc:105 : Invalid argument: slice index 49 of dimension 0 out of bounds.
2019-01-04 19:26:17.646436: W T:\src\github\tensorflow\tensorflow\core\framework\op_kernel.cc:1275] OP_REQUIRES failed at strided_slice_op.cc:105 : Invalid argument: slice index 49 of dimension 0 out of bounds.
2019-01-04 19:26:17.657683: W T:\src\github\tensorflow\tensorflow\core\framework\op_kernel.cc:1275] OP_REQUIRES failed at strided_slice_op.cc:105 : Invalid argument: slice index 54 of dimension 0 out of bounds.
2019-01-04 19:26:17.671311: W T:\src\github\tensorflow\tensorflow\core\framework\op_kernel.cc:1275] OP_REQUIRES failed at strided_slice_op.cc:105 : Invalid argument: slice index 54 of dimension 0 out of bounds.

The first time it was 49 but at the end it produced 54. I need help in understanding these parts of error trace:

InvalidArgumentError: slice index 49 of dimension 0 out of bounds.
	 [[Node: metrics/negative_avg_log_error/map/while/strided_slice_1 = StridedSlice[Index=DT_INT32, T=DT_FLOAT, begin_mask=0, ellipsis_mask=0, end_mask=0, new_axis_mask=0, shrink_axis_mask=1, _device="/job:localhost/replica:0/task:0/device:GPU:0"](metrics/negative_avg_log_error/map/while/TensorArrayReadV3_1, metrics/negative_avg_log_error/map/while/strided_slice_1/stack, metrics/negative_avg_log_error/map/while/strided_slice_1/stack_1, metrics/negative_avg_log_error/map/while/strided_slice/stack_1)]]

and

[[Node: training/Adadelta/gradients/query_to_context_attention/ExpandDims_grad/Reshape/_243 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_9091_...ad/Reshape", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Also, I don't know if these two errors are related to each other..

[gabrieldemarmiesse]

The easiest way to avoid shape errors is to catch them early. If you specify the input shape completely in the input tensor, the error message should be a bit easier to understand. Also, tf.eager will be supported in keras in the future, and will help with debugging. You can try tf.keras in the meantime which already supports tf.eager.

[ParikhKadam]

@gabrieldemarmiesse Ok.. I will look into it. I am totally new to Keras and so I don't even know what 'eager execution' is. Thank you for the suggestion..

Also, as you said, we can use tf.eager via tf.keras. My question is, can we also use other functionalities of tensorflow via tf.keras? Like tf.print? If yes, can you provide some links or articles for this understanding. I will also try searching google but keras documentation lacks such explanation.

And do you know anything about how to print the shape of tensors while training?

[ParikhKadam]

@gabrieldemarmiesse Read about eager execution on tensorflow's doc and also read about tf.keras. The above doubts are clear to me. But I still need a way to print dynamic shapes of tensors at model runtime. It is actually of great help..

BTW, the error I was having was already solved yesterday so I am closing this issue. Should I open a feature request at keras for printing dynamic shape of tensors?