PaddlePaddle Hackathon 57 提交 #1128

renmada · 2021-10-09T14:36:21Z

权重文件链接: https://pan.baidu.com/s/1-FJDmtfO8MuPQgq0EEbUhw 提取码: gst6
添加XLNetLMHeadModel、XLNetForMultipleChoice、XLNetForQuestionAnswering。
新增单元测试代码。XLNetLMHeadModel、XLNetForMultipleChoice、XLNetForQuestionAnswering。

CLAassistant · 2021-10-09T14:36:25Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ yingyibiao
❌ deeplaying
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

yingyibiao · 2021-10-13T11:05:46Z

新增权重请参考https://paddlenlp.readthedocs.io/zh/latest/community/contribute_models/contribute_awesome_pretrained_models.html

renmada · 2021-10-14T02:54:13Z

新增权重请参考https://paddlenlp.readthedocs.io/zh/latest/community/contribute_models/contribute_awesome_pretrained_models.html

transformers下没有community文件夹，需要自己新建吗？

yingyibiao · 2021-10-14T12:50:36Z

新增权重请参考https://paddlenlp.readthedocs.io/zh/latest/community/contribute_models/contribute_awesome_pretrained_models.html

transformers下没有community文件夹，需要自己新建吗？

这里写错了，是PaddleNLP/community文件夹

renmada · 2021-10-15T01:31:18Z

新增权重请参考https://paddlenlp.readthedocs.io/zh/latest/community/contribute_models/contribute_awesome_pretrained_models.html

transformers下没有community文件夹，需要自己新建吗？

这里写错了，是PaddleNLP/community文件夹

tokenizer_config_file不是必需的吧？

yingyibiao · 2021-10-18T05:06:16Z

新增权重请参考https://paddlenlp.readthedocs.io/zh/latest/community/contribute_models/contribute_awesome_pretrained_models.html

transformers下没有community文件夹，需要自己新建吗？

这里写错了，是PaddleNLP/community文件夹

tokenizer_config_file不是必需的吧？

tokenizer_config_file这个文件也是需要的

yingyibiao · 2021-10-18T05:07:36Z

community/renmada/distilbert-base-multilingual-cased/files.json

+{
+  "model_config_file": "https://paddlenlp.bj.bcebos.com/models/transformers/community/renmada/distilbert-base-multilingual-cased/model_config.json",
+  "model_state": "https://paddlenlp.bj.bcebos.com/models/transformers/community/renmada/distilbert-base-multilingual-cased/model_state.pdparams",
+  "tokenizer_config_file": "https://paddlenlp.bj.bcebos.com/models/transformers/community/renmada/bert-base-uncased-sst-2-finetuned/tokenizer_config.json",


这里需要使用模型相对应的tokenizer_config_file

yingyibiao · 2021-10-18T05:09:09Z

请在百度网盘中添加对应的tokenizer_config_file文件

paddlenlp/transformers/xlnet/modeling.py

renmada · 2021-10-18T14:37:22Z

新增权重请参考https://paddlenlp.readthedocs.io/zh/latest/community/contribute_models/contribute_awesome_pretrained_models.html

transformers下没有community文件夹，需要自己新建吗？

这里写错了，是PaddleNLP/community文件夹

tokenizer_config_file不是必需的吧？

tokenizer_config_file这个文件也是需要的

已经上传了，不过默认都是空文件啊

yingyibiao · 2021-10-19T03:32:39Z

https://github.com/PaddlePaddle/PaddleNLP/blob/develop/docs/model_zoo/transformers.rst
这个文件也需要同步修改

renmada · 2021-10-19T07:07:40Z

上面的问题改好了

yingyibiao · 2021-10-24T12:06:14Z

所有提交commit的人员都需要签署CLA.

yingyibiao · 2021-10-24T12:25:57Z

新增权重请参考https://paddlenlp.readthedocs.io/zh/latest/community/contribute_models/contribute_awesome_pretrained_models.html

transformers下没有community文件夹，需要自己新建吗？

这里写错了，是PaddleNLP/community文件夹

tokenizer_config_file不是必需的吧？

tokenizer_config_file这个文件也是需要的

已经上传了，不过默认都是空文件啊

tokenizer_config.json 和 model_config.json 两者都不应该为空。
具体格式可以参考对应class的save_pretrained接口保存后的文件格式。
例如"sshleifer/tiny-distilbert-base-uncased-finetuned-sst-2-english"对应的model_config.json 文件可以参考DistilBertForMaskedLM.save_pretrained接口保存后的model_config.json文件，tokenizer_config.json文件同理参考DistilBertTokenizer.save_pretrained接口保存后的tokenizer_config.json文件。

yingyibiao · 2021-10-24T12:42:14Z

需要添加DistilBert模型的权重转换代码

yingyibiao · 2021-10-24T12:50:31Z

另外，提交PR时请使用 pre-commit 钩子
https://www.paddlepaddle.org.cn/documentation/docs/zh/guides/10_contribution/local_dev_guide_cn.html#pre-commit

yingyibiao · 2021-10-24T12:27:25Z

community/renmada/distilbert-base-multilingual-cased/README.md

+model = DistilBertForMaskedLM.from_pretrained('distilbert-base-multilingual-cased')
+tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-multilingual-cased')


这里的调用方式为：
model = DistilBertForMaskedLM.from_pretrained('renmada/distilbert-base-multilingual-cased')
tokenizer = DistilBertTokenizer.from_pretrained('renmada/distilbert-base-multilingual-cased')

yingyibiao · 2021-10-24T12:30:00Z

community/renmada/sshleifer-tiny-distilbert-base-uncased-finetuned-sst-2-english/REAMME.md

+model = DistilBertModel.from_pretrained('distilbert-base-multilingual-cased')
+tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-multilingual-cased')


这个权重对应的class是DistilBertForSequenceClassification，权重名称修改同上。

yingyibiao · 2021-10-24T12:34:18Z

community/renmada/sshleifer-tiny-distilbert-base-uncased-finetuned-sst-2-english/REAMME.md

+# 模型来源
+https://huggingface.co/sshleifer/tiny-distilbert-base-uncased-finetuned-sst-2-english
+# 模型使用
+这个模型的命名方式用的是bert的前缀，转化成paddle时手动改成了distilbert。由于他的权重里有pooler而paddlenlp的distilbert没有pooler实现，因此例子只显示如何用DistilBertModel加载权重。


这个权重对应的class是DistilBertForSequenceClassification

个人感觉这里好像没意义啊，
原权重在paddlenlp中的DistilBertForSequenceClassification是加载不全的，原因是原权重pooler而paddlenlp的distilbert没有pooler实现

yingyibiao · 2021-10-24T12:38:13Z

paddlenlp/transformers/xlnet/modeling.py

+    XLNet Model with a language modeling head on top (linear layer with weights tied to the input embeddings).
+    Args:


Args:前面添加空行

yingyibiao · 2021-10-24T12:43:28Z

community/renmada/sshleifer-tiny-distilbert-base-uncased-finetuned-sst-2-english/REAMME.md

+# 模型来源
+https://huggingface.co/sshleifer/tiny-distilbert-base-uncased-finetuned-sst-2-english
+# 模型使用
+这个模型的命名方式用的是bert的前缀，转化成paddle时手动改成了distilbert。由于他的权重里有pooler而paddlenlp的distilbert没有pooler实现，因此例子只显示如何用DistilBertModel加载权重。


不应该存在上述无法对应的情况。

paddlenlp的distilbert实现没有pooler啊

原权重的命名方式更接近bertmodel而不是distilbert
我在转换的时候，前面的transformer layers可以转成distilbert的命名方式，但是它的pooler没有在paddlenlp的distilbert中实现

红框内和Pooler是同样的结构，你需要转换一下参数的key进行映射。

pooler和pre_classifier的激活函数不一样，分别是tanh和relu，会导致最后forward的结果不一样

看了一下代码，两者是一致的，都是ReLU

paddlenlp/transformers/xlnet/modeling.py

yingyibiao · 2021-10-24T13:23:13Z

paddlenlp/transformers/xlnet/modeling.py

+    """
+      XLNet Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear
+      layers on top of the hidden-states output to compute `span start logits` and `span end logits`).
+    """


添加__init__函数的docstring

yingyibiao · 2021-10-24T13:23:25Z

paddlenlp/transformers/xlnet/modeling.py

+    """
+    XLNet Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a
+    softmax) e.g. for RACE/SWAG tasks.
+    """


添加__init__函数的docstring

renmada · 2021-10-25T02:35:22Z

需要添加DistilBert模型的权重转换代码
两个问题

这个代码放在哪里
两个模型的命名方式不一样，所以转换代码不是完全通用

yingyibiao · 2021-10-25T04:19:51Z

需要添加DistilBert模型的权重转换代码
两个问题

这个代码放在哪里

两个模型的命名方式不一样，所以转换代码不是完全通用

代码放置在community/renmada目录下
转换代码需要你对模型参数的key进行映射，模型代码（Pytorch版本和Paddle版本）确定后，转换代码就是确定的了，需要实现的就是该代码。

renmada · 2021-10-26T02:15:17Z

之前的问题都已修复提交
model_config 和 tokenizer_config更新为save_pretrained的结果

yingyibiao · 2021-11-26T03:05:42Z

麻烦签署一下CLA.

yingyibiao · 2021-11-26T06:48:07Z

paddlenlp/transformers/xlnet/modeling.py

+            return_dict=return_dict, )
+        output = transformer_outputs if not return_dict \
+            else transformer_outputs["last_hidden_state"]
+        logits = self.classifier(output)


self.classifier没有定义

yingyibiao · 2021-11-26T06:51:58Z

paddlenlp/transformers/xlnet/modeling.py

+
+        self.init_weights()
+
+    def forward(
+            self,
+            input_ids,
+            token_type_ids=None,
+            attention_mask=None,
+            mems=None,
+            perm_mask=None,
+            target_mapping=None,
+            input_mask=None,
+            head_mask=None,
+            inputs_embeds=None,
+            use_mems_train=False,
+            use_mems_eval=False,
+            return_dict=False, ):
+        r"""
+        The XLNetForQuestionAnswering forward method, overrides the `__call__()` special method.
+
+        Args:
+            input_ids (Tensor):
+                See :class:`XLNetModel`.
+            token_type_ids (Tensor, optional):
+                See :class:`XLNetModel`.
+            attention_mask (Tensor, optional):
+                See :class:`XLNetModel`.
+            mems (Tensor, optional):
+                See :class:`XLNetModel`.
+            perm_mask (Tensor, optional):
+                See :class:`XLNetModel`.
+            target_mapping (Tensor, optional):
+                See :class:`XLNetModel`.
+            input_mask (Tensor, optional):
+                See :class:`XLNetModel`.
+            head_mask (Tensor, optional):
+                See :class:`XLNetModel`.
+            inputs_embeds (Tensor, optional):
+                See :class:`XLNetModel`.
+            use_mems_train (bool, optional):
+                See :class:`XLNetModel`.
+            use_mems_eval (bool, optional):
+                See :class:`XLNetModel`.
+            return_dict (bool, optional):
+                See :class:`XLNetModel`.
+
+        Returns:
+            tuple or dict: Returns tensor (`start_logits`, `end_logits`) or a dict with key-value pairs:
+             {"start_logits": `start_logits`, "end_logits": `end_logits`, "mems": `mems`,
+            "hidden_states": `hidden_states`, "attentions": `attentions`}
+
+            With the corresponding fields:
+            - `start_logits` (Tensor):
+                A tensor of the input token classification logits, indicates the start position of the labelled span.
+                Its data type should be float32 and its shape is [batch_size, sequence_length].
+            - `end_logits` (Tensor):
+                A tensor of the input token classification logits, indicates the end position of the labelled span.
+                Its data type should be float32 and its shape is [batch_size, sequence_length].
+            - `mems` (List[Tensor]):
+                See :class:`XLNetModel`.
+            - `hidden_states` (List[Tensor], optional):
+                See :class:`XLNetModel`.
+            - `attentions` (List[Tensor], optional):
+                See :class:`XLNetModel`.
+
+        Example:
+            .. code-block::
+
+                import paddle
+                from paddlenlp.transformers.xlnet.modeling import XLNetForQuestionAnswering
+                from paddlenlp.transformers.xlnet.tokenizer import XLNetTokenizer
+
+                tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased')
+                model = XLNetForQuestionAnswering.from_pretrained('xlnet-base-cased')
+
+                inputs = tokenizer("Welcome to use PaddlePaddle and PaddleNLP!")
+                inputs = {k:paddle.to_tensor([v]) for (k, v) in inputs.items()}
+                outputs = model(**inputs)
+                start_logits = outputs[0]
+                end_logits = outputs[1]
+        """
+        transformer_outputs = self.transformer(
+            input_ids,
+            token_type_ids=token_type_ids,
+            attention_mask=attention_mask,
+            mems=mems,
+            perm_mask=perm_mask,
+            target_mapping=target_mapping,
+            input_mask=input_mask,
+            head_mask=head_mask,
+            inputs_embeds=inputs_embeds,
+            use_mems_train=use_mems_train,
+            use_mems_eval=use_mems_eval,
+            return_dict=return_dict, )
+        output = transformer_outputs if not return_dict \
+            else transformer_outputs["last_hidden_state"]
+        logits = self.classifier(output)
+        logits = paddle.transpose(logits, perm=[2, 0, 1])
+        start_logits, end_logits = paddle.unstack(x=logits, axis=0)
+        return start_logits, end_logits


XLNetForQuestionAnswering 这个任务的逻辑和HuggingFace参考代码不一致？

这里实现的是HuggingFace的XLNetForQuestionAnsweringSimple，整体逻辑与paddlenlp的其他模型比较一致

HuggingFace的XLNetForQuestionAnswering比较复杂，是否需要实现？

yingyibiao · 2021-11-26T06:56:20Z

麻烦尽快按照review意见修改，解决conflicts～

renmada · 2021-11-28T11:28:10Z

麻烦签署一下CLA.
CLA签了老是不更新，还是未签状态，不知道怎么回事

yingyibiao

LGTM

xlnet downstream

e5c739b

renmada mentioned this pull request Oct 9, 2021

【PaddlePaddle Hackathon】任务总览 PaddlePaddle/Paddle#35940

Closed

renmada changed the title ~~PaddlePaddle Hackathon 52 提交~~ PaddlePaddle Hackathon 57 提交 Oct 9, 2021

yingyibiao self-assigned this Oct 10, 2021

return_dict

cee1f8b

deeplaying added 2 commits October 15, 2021 10:48

更新权重说明

8548e07

更新权重说明

4ca81f2

yingyibiao reviewed Oct 18, 2021

View reviewed changes

Merge branch 'develop' into hackathon57

db0188b

yingyibiao reviewed Oct 18, 2021

View reviewed changes

paddlenlp/transformers/xlnet/modeling.py Outdated Show resolved Hide resolved

deeplaying added 2 commits October 18, 2021 23:06

fix error

771f35c

Merge remote-tracking branch 'origin/hackathon57' into hackathon57

bc67ce6

deeplaying added 4 commits October 19, 2021 14:11

Update doc

09853ea

Remove unused

a60506f

Update doc

67f45e6

Update doc

02dda50

yingyibiao added the Hackathon label Oct 21, 2021

yingyibiao reviewed Oct 24, 2021

View reviewed changes

deeplaying added 2 commits October 25, 2021 15:39

update

2c58fc4

Remove num_classes

baed51e

yingyibiao reviewed Nov 26, 2021

View reviewed changes

deeplaying and others added 4 commits November 28, 2021 19:40

model_zoo

b9dcc6c

Update

2341c3d

Update

7d55377

Merge branch 'develop' into hackathon57

add19d1

yingyibiao approved these changes Nov 29, 2021

View reviewed changes

ZeyuChen merged commit 2afd760 into PaddlePaddle:develop Nov 29, 2021

		model = DistilBertForMaskedLM.from_pretrained('distilbert-base-multilingual-cased')
		tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-multilingual-cased')

		model = DistilBertModel.from_pretrained('distilbert-base-multilingual-cased')
		tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-multilingual-cased')

		XLNet Model with a language modeling head on top (linear layer with weights tied to the input embeddings).
		Args:

PaddlePaddle Hackathon 57 提交 #1128

PaddlePaddle Hackathon 57 提交 #1128

Conversation

renmada commented Oct 9, 2021

CLAassistant commented Oct 9, 2021 • edited Loading

yingyibiao commented Oct 13, 2021

renmada commented Oct 14, 2021 • edited Loading

yingyibiao commented Oct 14, 2021

renmada commented Oct 15, 2021

yingyibiao commented Oct 18, 2021

Choose a reason for hiding this comment

yingyibiao commented Oct 18, 2021

renmada commented Oct 18, 2021

yingyibiao commented Oct 19, 2021

renmada commented Oct 19, 2021 • edited Loading

yingyibiao commented Oct 24, 2021

yingyibiao commented Oct 24, 2021

yingyibiao commented Oct 24, 2021

yingyibiao commented Oct 24, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

renmada Oct 25, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

renmada commented Oct 25, 2021

yingyibiao commented Oct 25, 2021

renmada commented Oct 26, 2021

yingyibiao commented Nov 26, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yingyibiao commented Nov 26, 2021

renmada commented Nov 28, 2021

yingyibiao left a comment

Choose a reason for hiding this comment

CLAassistant commented Oct 9, 2021 •

edited

Loading

renmada commented Oct 14, 2021 •

edited

Loading

renmada commented Oct 19, 2021 •

edited

Loading

renmada Oct 25, 2021 •

edited

Loading