text match model , an illegal memory access was encountered.

[shikeno]

报错信息：
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1492: UserWarning: Skip loading for concat_fc.weight. concat_fc.w is not found in the provided dict.
warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1492: UserWarning: Skip loading for concat_fc.bias. concat.b is not found in the provided dict.
warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1492: UserWarning: Skip loading for output_layer.weight. linear_74.w_0 is not found in the provided dict.
warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1492: UserWarning: Skip loading for output_layer.bias. linear_74.b_0 is not found in the provided dict.
warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
Traceback (most recent call last):
File "run_trainer.py", line 124, in
run_trainer(_params)
File "run_trainer.py", line 101, in run_trainer
trainer.do_train()
File "/home/aistudio/work/ERNIE/applications/tasks/text_matching/trainer/custom_dynamic_trainer.py", line 55, in do_train
forward_out = self.model_class(example, phase=InstanceName.TRAINING)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 930, in call
return self._dygraph_call_func(*inputs, **kwargs)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
outputs = self.forward(*inputs, **kwargs)
File "/home/aistudio/work/ERNIE/applications/tasks/text_matching/model/ernie_matching_siamese_pointwise.py", line 68, in forward
task_ids=text_task_ids)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 930, in call
return self._dygraph_call_func(*inputs, **kwargs)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
outputs = self.forward(*inputs, **kwargs)
File "../../../erniekit/modules/ernie.py", line 169, in forward
sent_embedded = self.sent_emb(sent_ids)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 930, in call
return self._dygraph_call_func(*inputs, **kwargs)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
outputs = self.forward(*inputs, **kwargs)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/nn/layer/common.py", line 1469, in forward
name=self._name)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/nn/functional/input.py", line 206, in embedding
'remote_prefetch', False, 'padding_idx', padding_idx)
OSError: (External) CUDA error(700), an illegal memory access was encountered.
[Hint: 'cudaErrorIllegalAddress'. The device encountered a load or store instruction on an invalid memory address. This leaves the process in an inconsistentstate and any further CUDA work will return the same error. To continue using CUDA, the process must be terminated and relaunched. ] (at /paddle/paddle/phi/backends/gpu/cuda/cuda_info.cc:258)
[operator < lookup_table_v2 > error]

我参考了#466 ，不过这里比较久远了。
使用的环境是ai studio 上的V100 环境，paddlepaddle-gpu==2.3.2, paddlenlp==2.3.2

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reopen it. Thank you for your contributions.