By Topic

A model for multi-time scaled sequential decision making processes

Sign In

Cookies must be enabled to login.After enabling cookies , please use refresh or reload or ctrl+f5 on the browser for the login options.

Formats Non-Member Member
$33 $13
Learn how you can qualify for the best price for this item!
Become an IEEE Member or Subscribe to
IEEE Xplore for exclusive pricing!
close button

puzzle piece

IEEE membership options for an individual and IEEE Xplore subscriptions for an organization offer the most affordable access to essential journal articles, conference papers, standards, eBooks, and eLearning courses.

Learn more about:

IEEE membership

IEEE Xplore subscriptions

4 Author(s)
Hyeong Soo Chang ; Dept. of Electr. & Comput. Eng., Maryland Univ., College Park, MD, USA ; P. Fard ; S. I. Marcus ; M. A. Shayman

We propose a simple analytical model, called M time-scale Markov Decision Process (MMDP), for hierarchically structured sequential decision making processes, where decisions in each level in the M-level hierarchy are made in M different time-scales. In this model, the state space and action space of each level in the hierarchy are non-overlapping with those of the other levels, respectively, and the hierarchy is structured in a "pyramid" sense such that a decision made at level m(slower time-scale) state and/or the state will affect the evolutionary decision making process of the lower level m+1 (faster time-scale) until a new decision is made at the higher level but the lower level decisions themselves do not affect the higher level transition dynamics. The performance produced by the lower level decisions will affect the higher level decisions. A hierarchical objective function is defined such that the finite-horizon value of following a (nonstationary) policy at the level m+1 over a decision epoch of the level m plus an immediate reward at the level m is the single step reward for the level m decision making process. From this we define "multi-level optimal value function" and derive "multi-level optimality equation". We then give some example control problems that can be modeled as MMDPs.

Published in:

Decision and Control, 2002, Proceedings of the 41st IEEE Conference on  (Volume:4 )

Date of Conference:

10-13 Dec. 2002