Learning to Predict by the Methods of Temporal Diierences

Abstract

This article introduces a class of incremental learning procedures specialized for prediction|that is, for using past experience with an incompletely known system to predict its future behavior. Whereas conventional prediction-learning methods assign credit by means of the diierence between predicted and actual outcomes, the new methods assign credit by means of the diierence between temporally successive predictions. Although such temporal-diierence methods have been used in Samuel's checker player, Holland's bucket brigade, and the author's Adaptive Heuristic Critic, they have remained poorly understood. Here we p r o ve their convergence and optimality for special cases and relate them to supervised-learning methods. For most real-world prediction problems, temporal-diierence methods require less memory and less peak computation than conventional methodss and they produce more accurate predictions. We argue that most problems to which supervised learning is currently applied are really prediction problems of the sort to which temporal-diierence methods can be applied to advantage.

Extracted Key Phrases

Showing 1-10 of 28 references

Temporal credit assignment in reinforcement learning. D o c toral dissertation

  • R S Sutton
  • 1984
Highly Influential
7 Excerpts

A neuronal model of classical conditioning (Air Force Wright Aeronautical Laboratories Technical Report 87-1139)

  • A H Klopf
  • 1987
1 Excerpt

A temporal-diierence model of classical conditioning

  • R S Sutton, A G Barto
  • 1987

Temporal primacy overrides prior training in serial compound conditioning of the rabbit's nictitating membrane response

  • E J Kehoe, B G Schreurs, P Graham
  • 1987

Escaping brittleness: The possibilities of general-purpose learning algorithms applied to parallel rule-based systems

  • J H Holland
  • 1986

Learning static evaluation functions by linear regression Machine learning: A guide to current research

  • J Christensen
  • 1986
Showing 1-10 of 801 extracted citations
050100150'89'92'95'98'01'04'07'10'13'16
Citations per Year

1,746 Citations

Semantic Scholar estimates that this publication has received between 1,520 and 2,002 citations based on the available data.

See our FAQ for additional information.