Interpreting Word-Level Hidden State Behaviour of Character-Level LSTM Language Models
Full Text: W18-5428.pdf
While Long Short-Term Memory networks (LSTMs) and other forms of recurrent neural network have been successfully applied to language modeling on a character level, the hidden state dynamics of these models can be difficult to interpret. We investigate the hidden states of such a model by using the HDBSCAN clustering algorithm to identify points in the text at which the hidden state is similar. Focusing on whitespace characters prior to the beginning of a word reveals interpretable clusters that offer insight into how the LSTM may combine contextual and character-level information to identify parts of speech. We also introduce a method for deriving word vectors from the hidden state representation in order to investigate the word-level knowledge of the model. These word vectors encode meaningful semantic information even for words that appear only once in the training text.
Citation
A. Hiebert, C. Peterson, A. Fyshe, N. Mehta. "Interpreting Word-Level Hidden State Behaviour of Character-Level LSTM Language Models". BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp 258–266, November 2018.| Keywords: | |
| Category: | In Workshop |
| Web Links: | ACL |
| DOI |
BibTeX
@misc{Hiebert+al:18,
author = {Avery Hiebert and Cole Peterson and Alona Fyshe and Nishant Mehta},
title = {Interpreting Word-Level Hidden State Behaviour of Character-Level
LSTM Language Models},
Booktitle = "Proceedings",
Pages = {258–266},
booktitle = {BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP},
year = 2018,
}Last Updated: June 22, 2020Submitted by Sabina P
