Supervised autoencoders: Improving generalization performance with unsupervised regularizers
- Lei Le, Indiana University, Bloomington
- Andrew Patterson
- Martha White, University of Alberta
Generalization performance is a central goal in machine learning, particularly when learning representations with large neural networks. A common strategy to improve generalization has been through the use of regularizers, typically as a norm constraining the parameters. Regularizing hidden layers in a neural network architecture, however, is not straightforward. There have been a few effective layer-wise suggestions, but without theoretical guarantees for improved performance. In this work, we theoretically and empirically analyze one such model, called a supervised auto-encoder: a neural network that predicts both inputs (reconstruction error) and targets jointly. We provide a novel generalization result for linear auto-encoders, proving uniform stability based on the inclusion of the reconstruction error---particularly as an improvement on simplistic regularization such as norms or even on more advanced regularizations such as the use of auxiliary tasks. Empirically, we then demonstrate that, across an array of architectures with a different number of hidden units and activation functions, the supervised auto-encoder compared to the corresponding standard neural network never harms performance and can significantly improve generalization.
Citation
L. Le, A. Patterson, M. White. "Supervised autoencoders: Improving generalization performance with unsupervised regularizers". Neural Information Processing Systems (NIPS), (ed: Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, NicoloCesa-Bianchi, Roman Garnett), pp 107--117, December 2018.| Keywords: | |
| Category: | In Conference |
| Web Links: | NeurIPS |
BibTeX
@incollection{Le+al:NIPS18,
author = {Lei Le and Andrew Patterson and Martha White},
title = {Supervised autoencoders: Improving generalization performance with
unsupervised regularizers},
Editor = {Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman,
NicoloCesa-Bianchi, Roman Garnett},
Pages = {107--117},
booktitle = {Neural Information Processing Systems (NIPS)},
year = 2018,
}Last Updated: February 25, 2020Submitted by Sabina P
