Not Logged In

Generalization in Reinforcement Learning: Successful Examples Using Sparse Coarse Coding

Full Text: sutton96generalization.pdf PDF

On large problems, reinforcement learning systems must use parameterized function approximators such as neural networks in order to generalize between similar situations and actions. In these cases, there are no strong theoretical results on the accuracy of convergence, and computational results have been mixed. In particular, Boyan and Moore reported at last year's meeting a series of negative results in attempting to apply dynamic programming together function approximation to simple control problems with continuous state spaces. In this paper, we present positive results for all the control tasks they attempted, and for one that is significantly larger. The most important differences are that we used sparse-coarse-coded function approximators (CMACs) whereas they used mostly global function approximators, and that we learned online whereas they used learned offline. Boyan and Moore and others have suggested that the problems they encountered could be solved by using actual outcomes (rollouts), as in classical Monte Carlo methods, and as in the TD(lambda) algorithm when lambda=1. However, in our experiments this always resulted in substantially poorer performance. We conclude that reinforcement learning can work robustly in conjunction with function approximators, and that there is little justification at present for avoiding the case of general lambda.


R. Sutton. "Generalization in Reinforcement Learning: Successful Examples Using Sparse Coarse Coding". Neural Information Processing Systems (NIPS), pp 1038-1044, January 1996.

Keywords: sparse-coarse-coded, global, machine learning
Category: In Conference


  author = {Richard S. Sutton},
  title = {Generalization in Reinforcement Learning: Successful Examples Using
    Sparse Coarse Coding},
  Pages = {1038-1044},
  booktitle = {Neural Information Processing Systems (NIPS)},
  year = 1996,

Last Updated: May 31, 2007
Submitted by Staurt H. Johnson

University of Alberta Logo AICML Logo