Compressing Neural Language Models by Sparse Word Representations
Neural networks are among the state-ofthe-art techniques for language modeling. Existing neural language models typically map discrete words to distributed, dense vector representations. After information processing of the preceding context words by hidden layers, an output layer estimates the probability of the next word. Such approaches are time- and memory-intensive because of the large numbers of parameters for word embeddings and the output layer. In this paper, we propose to compress neural language models by sparse word representations. In the experiments, the number of parameters in our model increases very slowly with the growth of the vocabulary size, which is almost imperceptible. Moreover, our approach not only reduces the parameter space to a large extent, but also improves the performance in terms of the perplexity measure.Citation
Y. Chen, L. Mou, Y. Xu, G. Li, Z. Jin. "Compressing Neural Language Models by Sparse Word Representations". International Conference on Computational Linguistics and the Association for Computational Linguist, pp 226–235, August 2016.| Keywords: | |
| Category: | In Conference |
| Web Links: | ACL |
BibTeX
@incollection{Chen+al:ACL16,
author = {Yunchuan Chen and Lili Mou and Yan Xu and Ge Li and Zhi Jin},
title = {Compressing Neural Language Models by Sparse Word Representations},
Pages = {226–235},
booktitle = {International Conference on Computational Linguistics and the
Association for Computational Linguist},
year = 2016,
}Last Updated: February 04, 2021Submitted by Sabina P
