Symbolic Semantic Memory in Transformer Language Models

February 14, 2022

Wednesday, March 2nd at 3-4:30pm, Summit Room 3346 TMCB

Advisor: Dan Ventura

Robert Morain MS Thesis Defense/PhD Qualification


This paper demonstrates how transformer language models can be improved by giving them access to relevant structured data extracted from a knowledge base. The knowledge base preparation process and modifications to transformer models are explained. We evaluate these methods on language modeling and question answering tasks. These results show that even simple additional knowledge augmentation leads to a reduction in validation loss by 73%. These methods also significantly outperform common ways of improving language models such as increasing the model size or adding more data.