ACL2020_PretrainLanguageModel.md

预训练语言模型及部分应用

QuASE: Question-Answer Driven Sentence Encoding

• 论文链接：https://www.aclweb.org/anthology/2020.acl-main.772.pdf
• 代码链接：https://github.com/CogComp/QuASE

TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data

• 论文链接：https://www.aclweb.org/anthology/2020.acl-main.745.pdf
• 代码链接：https://github.com/facebookresearch/TaBERT

Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks

• 论文链接：https://www.aclweb.org/anthology/2020.acl-main.740.pdf
• 代码链接：https://github.com/allenai/dont-stop-pretraining

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

• 论文链接：https://www.aclweb.org/anthology/2020.acl-main.703.pdf
• 代码链接：https://github.com/pytorch/fairseq

Toward Better Storylines with Sentence-Level Language Models

• 论文链接：https://www.aclweb.org/anthology/2020.acl-main.666.pdf
• 代码链接：https://github.com/google-research/google-research/tree/master/better_storylines

tBERT: Topic Models and BERT Joining Forces for Semantic Similarity Detection

• 论文链接：https://www.aclweb.org/anthology/2020.acl-main.630.pdf
• 代码链接：https://github.com/wuningxi/tBERT

FastBERT: a Self-distilling BERT with Adaptive Inference Time

• 论文链接：https://www.aclweb.org/anthology/2020.acl-main.537.pdf
• 代码链接：https://github.com/autoliuweijie/FastBERT

Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models

• 论文链接：https://www.aclweb.org/anthology/2020.acl-main.439.pdf
• 代码链接：https://github.com/google-research/language/tree/master/language/conpono

DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering

• 论文链接：https://www.aclweb.org/anthology/2020.acl-main.411.pdf
• 代码链接：https://github.com/StonyBrookNLP/deformer

Enhancing Pre-trained Chinese Character Representation with Word-aligned Attention

• 论文链接：https://www.aclweb.org/anthology/2020.acl-main.315.pdf
• 代码链接：https://github.com/lsvih/MWA

Span Selection Pre-training for Question Answering

• 论文链接：https://www.aclweb.org/anthology/2020.acl-main.247.pdf
• 代码链接：https://github.com/IBM/span-selection-pretraining

DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference

• 论文链接：https://www.aclweb.org/anthology/2020.acl-main.204.pdf
• 代码链接：https://github.com/castorini/DeeBERT

MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices

• 论文链接：https://www.aclweb.org/anthology/2020.acl-main.195.pdf
• 代码链接：https://github.com/google-research/google-research/tree/master/mobilebert

Fast and Accurate Deep Bidirectional Language Representations for Unsupervised Learning

• 论文链接：https://www.aclweb.org/anthology/2020.acl-main.76.pdf
• 代码链接：https://github.com/joongbo/tta

Few-Shot NLG with Pre-Trained Language Model

• 论文链接：https://www.aclweb.org/anthology/2020.acl-main.18.pdf
• 代码链接：https://github.com/czyssrs/Few-Shot-NLG