BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding - NAACL 2019)
ERNIE 2.0: A Continual Pre-training Framework for Language Understanding - arXiv 2019)
StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding - arXiv 2019)
RoBERTa: A Robustly Optimized BERT Pretraining Approach - arXiv 2019)
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations - arXiv 2019)
Multi-Task Deep Neural Networks for Natural Language Understanding - arXiv 2019)
What does BERT learn about the structure of language? (ACL2019)
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned (ACL2019) [github]
Open Sesame: Getting Inside BERT's Linguistic Knowledge (ACL2019 WS)
Analyzing the Structure of Attention in a Transformer Language Model (ACL2019 WS)
What Does BERT Look At? An Analysis of BERT's Attention (ACL2019 WS)
Do Attention Heads in BERT Track Syntactic Dependencies?
Blackbox meets blackbox: Representational Similarity and Stability Analysis of Neural Language Models and Brains (ACL2019 WS)
Inducing Syntactic Trees from BERT Representations (ACL2019 WS)
A Multiscale Visualization of Attention in the Transformer Model (ACL2019 Demo)
Visualizing and Measuring the Geometry of BERT
How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings (EMNLP2019)
Are Sixteen Heads Really Better than One? (NeurIPS2019)
On the Validity of Self-Attention as Explanation in Transformer Models
Visualizing and Understanding the Effectiveness of BERT (EMNLP2019)
Attention Interpretability Across NLP Tasks
Revealing the Dark Secrets of BERT (EMNLP2019)
Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs (EMNLP2019)
The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives (EMNLP2019)
A Primer in BERTology: What we know about how BERT works
Do NLP Models Know Numbers? Probing Numeracy in Embeddings (EMNLP2019)
How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representations (CIKM2019)
Whatcha lookin' at? DeepLIFTing BERT's Attention in Question Answering
What does BERT Learn from Multiple-Choice Reading Comprehension Datasets?
Calibration of Pre-trained Transformers
exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformers Models [github]
Transformer Series
Attention Is All You Need - arXiv 2017)
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context - arXiv 2019)
Universal Transformers - ICLR 2019)
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer - arXiv 2019)
Reformer: The Efficient Transformer - ICLR 2020)
Adaptive Attention Span in Transformers (ACL2019)
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context (ACL2019) [github]
Generating Long Sequences with Sparse Transformers
Adaptively Sparse Transformers (EMNLP2019)
Compressive Transformers for Long-Range Sequence Modelling
The Evolved Transformer (ICML2019)
Reformer: The Efficient Transformer (ICLR2020) [github]
GRET: Global Representation Enhanced Transformer (AAAI2020)
Transformer on a Diet [github]
Efficient Content-Based Sparse Attention with Routing Transformers
BP-Transformer: Modelling Long-Range Context via Binary Partitioning
Recipes for building an open-domain chatbot
Longformer: The Long-Document Transformer
Transfer Learning
Deep contextualized word representations - NAACL 2018)
Universal Language Model Fine-tuning for Text Classification - ACL 2018)
Improving Language Understanding by Generative Pre-Training - Alec Radford)
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding - NAACL 2019)
Cloze-driven Pretraining of Self-attention Networks - arXiv 2019)
Unified Language Model Pre-training for Natural Language Understanding and Generation - arXiv 2019)
MASS: Masked Sequence to Sequence Pre-training for Language Generation - ICML 2019)
Text Summarization
Positional Encoding to Control Output Sequence Length - Sho Takase(2019)
Fine-tune BERT for Extractive Summarization - Yang Liu(2019)
Language Models are Unsupervised Multitask Learners - Alec Radford(2019)
A Unified Model for Extractive and Abstractive Summarization using Inconsistency Loss - Wan-Ting Hsu(2018)
A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents - Arman Cohan(2018)
GENERATING WIKIPEDIA BY SUMMARIZING LONG SEQUENCES - Peter J. Liu(2018)
Get To The Point: Summarization with Pointer-Generator Networks - Abigail See(2017) * A Neural Attention Model for Sentence Summarization - Alexander M. Rush(2015)
Sentiment Analysis
Multi-Task Deep Neural Networks for Natural Language Understanding - Xiaodong Liu(2019)
Aspect-level Sentiment Analysis using AS-Capsules - Yequan Wang(2019)
On the Role of Text Preprocessing in Neural Network Architectures: An Evaluation Study on Text Categorization and Sentiment Analysis - Jose Camacho-Collados(2018)
Learned in Translation: Contextualized Word Vectors - Bryan McCann(2018)
Universal Language Model Fine-tuning for Text Classification - Jeremy Howard(2018)
Convolutional Neural Networks with Recurrent Neural Filters - Yi Yang(2018)
Information Aggregation via Dynamic Routing for Sequence Encoding - Jingjing Gong(2018)
Learning to Generate Reviews and Discovering Sentiment - Alec Radford(2017)
A Structured Self-attentive Sentence Embedding - Zhouhan Lin(2017)
Question Answering
Language Models are Unsupervised Multitask Learners - Alec Radford(2019)
Improving Language Understanding by Generative Pre-Training - Alec Radford(2018)
Bidirectional Attention Flow for Machine Comprehension - Minjoon Seo(2018)
Reinforced Mnemonic Reader for Machine Reading Comprehension - Minghao Hu(2017)
Neural Variational Inference for Text Processing - Yishu Miao(2015)
Machine Translation
The Evolved Transformer - David R. So(2019)
Surver paper
Evolution of transfer learning in natural language processing
Pre-trained Models for Natural Language Processing: A Survey
A Survey on Contextual Embeddings
Downstream task
QA MC Dialogue
A BERT Baseline for the Natural Questions
MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension (ACL2019)
Unsupervised Domain Adaptation on Reading Comprehension
BERTQA -- Attention on Steroids
A Multi-Type Multi-Span Network for Reading Comprehension that Requires Discrete Reasoning (EMNLP2019)
SDNet: Contextualized Attention-based Deep Network for Conversational Question Answering
Multi-hop Question Answering via Reasoning Chains
Select, Answer and Explain: Interpretable Multi-hop Reading Comprehension over Multiple Documents
Multi-step Entity-centric Information Retrieval for Multi-Hop Question Answering (EMNLP2019 WS)
End-to-End Open-Domain Question Answering with BERTserini (NAALC2019)
Latent Retrieval for Weakly Supervised Open Domain Question Answering (ACL2019)
Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering (EMNLP2019)
Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering (ICLR2020)
Learning to Ask Unanswerable Questions for Machine Reading Comprehension (ACL2019)
Unsupervised Question Answering by Cloze Translation (ACL2019)
Reinforcement Learning Based Graph-to-Sequence Model for Natural Question Generation
A Recurrent BERT-based Model for Question Generation (EMNLP2019 WS)
Learning to Answer by Learning to Ask: Getting the Best of GPT-2 and BERT Worlds
Enhancing Pre-Trained Language Representations with Rich Knowledge for Machine Reading Comprehension (ACL2019)
Incorporating Relation Knowledge into Commonsense Reading Comprehension with Multi-task Learning (CIKM2019)
