Symbolic Semantic Memory in Transformer Language Models
February 14, 2022
Wednesday, March 2nd at 3-4:30pm, Summit Room 3346 TMCB
Advisor: Dan Ventura
Robert Morain MS Thesis Defense/PhD Qualification
This paper demonstrates how transformer language models can be improved by giving them access to relevant structured data extracted from a knowledge base. The knowledge base preparation process and modifications to transformer models are explained. We evaluate these methods on language modeling and question answering tasks. These results show that even simple additional knowledge augmentation leads to a reduction in validation loss by 73%. These methods also significantly outperform common ways of improving language models such as increasing the model size or adding more data.