-
Notifications
You must be signed in to change notification settings - Fork 163
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Problem with Informer while using ProbAttention #47
Comments
Hi @Nodon447 , You can try the latest version now. Thanks |
Hi @LongxingTan train_length = 49 x_train = ( x_valid = ( model = AutoModel("Informer", predict_length=predict_length,custom_model_params=custom_params) InvalidArgumentError Traceback (most recent call last) File /anaconda/envs/azureml_py38_PT_TF/lib/python3.8/site-packages/tfts/trainer.py:310, in KerasTrainer.train(self, train_dataset, valid_dataset, n_epochs, batch_size, steps_per_epoch, callback_metrics, early_stopping, checkpoint, verbose, **kwargs) File /anaconda/envs/azureml_py38_PT_TF/lib/python3.8/site-packages/keras/utils/traceback_utils.py:70, in filter_traceback..error_handler(*args, **kwargs) File /anaconda/envs/azureml_py38_PT_TF/lib/python3.8/site-packages/tfts/models/informer.py:151, in Encoder.call(self, x, mask) File /anaconda/envs/azureml_py38_PT_TF/lib/python3.8/site-packages/tfts/models/informer.py:181, in EncoderLayer.call(self, x, mask) File /anaconda/envs/azureml_py38_PT_TF/lib/python3.8/site-packages/tfts/layers/attention_layer.py:218, in ProbAttention.call(self, q, k, v, mask) File /anaconda/envs/azureml_py38_PT_TF/lib/python3.8/site-packages/tfts/layers/attention_layer.py:166, in ProbAttention._prob_qk(self, q, k, sample_k, top_n) InvalidArgumentError: Exception encountered when calling layer 'prob_attention' (type ProbAttention). {{function_node _wrapped__Pack_N_3_device/job:localhost/replica:0/task:0/device:CPU:0}} Shapes of all inputs must match: values[0].shape = [2,1,20] != values[2].shape = [2,20] [Op:Pack] name: stack Call arguments received by layer 'prob_attention' (type ProbAttention): |
I have fixed it in the latest version, try 0.0.10 version The squeeze should assign the specific dimension, as below. If not, the batch dimension will also be removed
Thanks |
So i wanted to use the Informer model to predict time series.
but even the Example doesnt work for me when setting prob_attention True
so this is my code:
params: Dict[str, Any] = {
"n_encoder_layers": 1,
"n_decoder_layers": 1,
"attention_hidden_sizes": 32 * 1,
"num_heads": 1,
"attention_dropout": 0.0,
"ffn_hidden_sizes": 32 * 1,
"ffn_filter_sizes": 32 * 1,
"ffn_dropout": 0.0,
"skip_connect_circle": False,
"skip_connect_mean": False,
"prob_attention": False,
"distil_conv": False,
}
custom_params = params.copy()
custom_params["prob_attention"] = True
option1: np.ndarray
train_length = 49
predict_length = 10
n_encoder_feature = 2
n_decoder_feature = 3
x_train = (
np.random.rand(1, train_length, 1), # inputs: (batch, train_length, 1)
np.random.rand(1, train_length, n_encoder_feature), # encoder_feature: (batch, train_length, encoder_features)
np.random.rand(1, predict_length, n_decoder_feature), # decoder_feature: (batch, predict_length, decoder_features)
)
y_train = np.random.rand(1, predict_length, 1) # target: (batch, predict_length, 1)
x_valid = (
np.random.rand(1, train_length, 1),
np.random.rand(1, train_length, n_encoder_feature),
np.random.rand(1, predict_length, n_decoder_feature),
)
y_valid = np.random.rand(1, predict_length, 1)
model = AutoModel("Informer", predict_length=predict_length,custom_model_params=custom_params)
trainer = KerasTrainer(model)
trainer.train((x_train, y_train), (x_valid, y_valid), n_epochs=1)
and this is the error:
TypeError Traceback (most recent call last)
Cell In[9], line 45
43 model = AutoModel("Informer", predict_length=predict_length,custom_model_params=custom_params)
44 trainer = KerasTrainer(model)
---> 45 trainer.train((x_train, y_train), (x_valid, y_valid), n_epochs=1)
File /anaconda/envs/azureml_py38_PT_TF/lib/python3.8/site-packages/tfts/trainer.py:289, in KerasTrainer.train(self, train_dataset, valid_dataset, n_epochs, batch_size, steps_per_epoch, callback_eval_metrics, early_stopping, checkpoint, verbose, **kwargs)
286 else:
287 raise ValueError("tfts inputs should be either tf.data instance or 3d array list/tuple")
--> 289 self.model = self.model.build_model(inputs=inputs)
291 # print(self.model.summary())
292 self.model.compile(loss=self.loss_fn, optimizer=self.optimizer, metrics=callback_eval_metrics, run_eagerly=True)
File /anaconda/envs/azureml_py38_PT_TF/lib/python3.8/site-packages/tfts/models/auto_model.py:81, in AutoModel.build_model(self, inputs)
80 def build_model(self, inputs):
---> 81 outputs = self.model(inputs)
82 return tf.keras.Model([inputs], [outputs])
File /anaconda/envs/azureml_py38_PT_TF/lib/python3.8/site-packages/tfts/models/informer.py:120, in Informer.call(self, inputs, teacher)
115 decoder_feature = tf.cast(
116 tf.reshape(tf.range(self.predict_sequence_length), (-1, self.predict_sequence_length, 1)), tf.float32
117 )
119 encoder_feature = self.encoder_embedding(encoder_feature) # batch * seq * embedding_size
--> 120 memory = self.encoder(encoder_feature, mask=None)
122 B, L, _ = tf.shape(decoder_feature)
123 casual_mask = CausalMask(B * self.params["num_heads"], L).mask
File /anaconda/envs/azureml_py38_PT_TF/lib/python3.8/site-packages/keras/utils/traceback_utils.py:70, in filter_traceback..error_handler(*args, **kwargs)
67 filtered_tb = _process_traceback_frames(e.traceback)
68 # To get the full stack trace, call:
69 #
tf.debugging.disable_traceback_filtering()
---> 70 raise e.with_traceback(filtered_tb) from None
71 finally:
72 del filtered_tb
File /tmp/autograph_generated_filewsa2jpfz.py:56, in outer_factory..inner_factory..tf__call(self, x, mask)
54 conv_layer = ag.Undefined('conv_layer')
55 attn_layer = ag__.Undefined('attn_layer')
---> 56 ag__.if_stmt((ag__.ld(self).conv_layers is not None), if_body, else_body, get_state_2, set_state_2, ('x',), 1)
58 def get_state_3():
59 return (x,)
File /tmp/autograph_generated_filewsa2jpfz.py:36, in outer_factory..inner_factory..tf__call..if_body()
34 attn_layer = ag.Undefined('attn_layer')
35 ag__.for_stmt(ag__.converted_call(ag__.ld(zip), (ag__.ld(self).layers, ag__.ld(self).conv_layers), None, fscope), None, loop_body, get_state, set_state, ('x',), {'iterate_names': '(attn_layer, conv_layer)'})
---> 36 x = ag__.converted_call(ag__.ld(self).layers[(- 1)], (ag__.ld(x), ag__.ld(mask)), None, fscope)
File /tmp/autograph_generated_file32nu44x0.py:12, in outer_factory..inner_factory..tf__call(self, x, mask)
10 retval = ag_.UndefinedReturnValue()
11 input = ag__.ld(x)
---> 12 x = ag__.converted_call(ag__.ld(self).attn_layer, (ag__.ld(x), ag__.ld(x), ag__.ld(x), ag__.ld(mask)), None, fscope)
13 x = ag__.converted_call(ag__.ld(self).drop, (ag__.ld(x),), None, fscope)
14 x = (ag__.ld(x) + ag__.ld(input))
File /tmp/autograph_generated_file6otuhk1u.py:16, in outer_factory..inner_factory..tf__call(self, q, k, v, mask)
14 (B, L, D) = ag.ld(q).shape
15 (, S, ) = ag_.ld(k).shape
---> 16 q = ag__.converted_call(ag__.ld(tf).reshape, (ag__.ld(q), (ag__.ld(B), ag__.ld(self).num_heads, ag__.ld(L), (- 1))), None, fscope)
17 k_ = ag__.converted_call(ag__.ld(tf).reshape, (ag__.ld(k), (ag__.ld(B), ag__.ld(self).num_heads, ag__.ld(S), (- 1))), None, fscope)
18 v_ = ag__.converted_call(ag__.ld(tf).reshape, (ag__.ld(v), (ag__.ld(B), ag__.ld(self).num_heads, ag__.ld(S), (- 1))), None, fscope)
TypeError: Exception encountered when calling layer "encoder_4" (type Encoder).
in user code:
Call arguments received by layer "encoder_4" (type Encoder):
• x=tf.Tensor(shape=(None, 49, 32), dtype=float32)
• mask=None
I have no clue how to fix this. So it would be really nice if anyone could help.
The text was updated successfully, but these errors were encountered: