Intra-Option Learning About Temporally Abstract Actions
- Richard S. Sutton, Department of Computing Science, University of Alberta
- Doina Precup, McGill University, Montreal
- Satinder Singh, University of Michigan, Ann Arbor, MI
Several researchers have proposed modeling temporally abstract actions in reinforcement learning by the combination of a policy and a termination condition, which we refer to as an option. Value functions over options and models of options can be learned using methods designed for semi-Markov decision processes (SMDPs). However, these methods all require an option to be executed to termination. In this paper we explore methods that learn about an option from small fragments of experience consistent with that option, even if the option itself is not executed. We call these methods intra-option learning methods because they learn from experience within an option. Intra-option methods are sometimes much more efficient than SMDP methods because they can use off-policy temporal-difference mechanisms to learn simultaneously about all the options consistent with an experience, not just the few that were actually executed. In this paper we present intra-option learning methods for learning value functions over options and for learning multi-step models of the consequences of options. We present computational examples in which these new methods learn much faster than SMDP methods and learn effectively when SMDP methods cannot learn at all. We also sketch a convergence proof for intra-option value learning.
Citation
R. Sutton, D. Precup, S. Singh. "Intra-Option Learning About Temporally Abstract Actions". International Conference on Machine Learning (ICML), Madison, Wisconsin USA, pp 556-564, January 1998.| Keywords: | mechanisms, SMDP, convergence, machine learning |
| Category: | In Conference |
BibTeX
@incollection{Sutton+al:ICML98,
author = {Richard S. Sutton and Doina Precup and Satinder Singh},
title = {Intra-Option Learning About Temporally Abstract Actions},
Pages = {556-564},
booktitle = {International Conference on Machine Learning (ICML)},
year = 1998,
}Last Updated: May 31, 2007Submitted by Staurt H. Johnson
