QAGAN: Adversarial Approach To Learning Domain Invariant Language Features
Shubham Shrivastava, Kaiyue Wang
Jun 24, 2022
Citations
1
Citations
Journal
ArXiv
Full text
Semantic Scholar
Key Takeaway Key Takeaway: Adversarial training with domain-invariant features improves question-answering language models' generalization to out-of-domain datasets, achieving 15.2% improvement in EM score and 5.6% boost in F1 score.
Abstract
Training models that are robust to data domain shift has gained an increasing interest both in academia and industry [1]. Question-Answering language models, being one of the typical problem in NLP research, has received much success with the advent of large transformer models [2]. However, existing approaches mostly works under the assumption that data is drawn from same distribution which is unrealistic. In this paper, we explore adversarial training approach towards learning domain-invariant features so that language models can generalize well to out-of-domain datasets. We also inspect various other ways to boost our model performance including data augmentation by paraphrasing sentences, conditioning end of answer span prediction on the start word, and carefully designed annealing function. Our initial results shows that in combination with these methods, we are able to achieve 15 . 2% improvement in EM score and 5 . 6% boost in F1 score on out-of-domain validation dataset over the baseline. We also dissect our model outputs and visualize the model hidden-states by projecting them onto a lower-dimensional space, and discover that our specific adversarial training approach indeed encourages the model to learn domain invariant embedding and bring them closer in the multi-dimensional space. [SEP] impressive results using only a single classification layer on top of pre-trained BERT. While these models result in great performance on downstream tasks, they fail to generalize well across datasets. This is attributed by the inherent domain gap between various datasets which causes a large performance drop when tested on an unseen dataset. We attempt to minimize this domain gap and work towards building a model that is robust to domain shifts in dataset by learning domain-invariant features through adversarial training. We show through systematic evaluations that our approach actually helps the network close this gap and generalize better to the out-of-domain datasets.