Improving Regression Performance with Distributional Losses
- Ehsan Imani
- Martha White, University of Alberta
There is growing evidence that converting targets to soft targets in supervised learning can provide considerable gains in performance. Much of this work has considered classification, converting hard zero-one values to soft labels—such as by adding label noise, incorporating label ambiguity or using distillation. In parallel, there is some evidence from a regression setting in reinforcement learning that learning distributions can improve performance. In this work, we investigate the reasons for this improvement, in a regression setting. We introduce a novel distributional regression loss, and similarly find it significantly improves prediction accuracy. We investigate several common hypotheses, around reducing overfitting and improved representations. We instead find evidence for an alternative hypothesis: this loss is easier to optimize, with better behaved gradients, resulting in improved generalization. We provide theoretical support for this alternative hypothesis, by characterizing the norm of the gradients of this loss.
Citation
E. Imani, M. White. "Improving Regression Performance with Distributional Losses". International Conference on Machine Learning (ICML), (ed: Jennifer G. Dy, Andreas Krause), pp 2162-2171, July 2018.| Keywords: | |
| Category: | In Conference |
| Web Links: | PMLR |
BibTeX
@incollection{Imani+White:ICML18,
author = {Ehsan Imani and Martha White},
title = {Improving Regression Performance with Distributional Losses},
Editor = {Jennifer G. Dy, Andreas Krause},
Pages = {2162-2171},
booktitle = {International Conference on Machine Learning (ICML)},
year = 2018,
}Last Updated: February 25, 2020Submitted by Sabina P
