Everand Logo

Welcome to Everand!

  • Discover millions of ebooks, audiobooks, and so much more with a free trial

    Only $11.99/month after trial. Cancel anytime.

    Dynamic Probabilistic Systems, Volume II: Semi-Markov and Decision Processes
    Dynamic Probabilistic Systems, Volume II: Semi-Markov and Decision Processes
    Dynamic Probabilistic Systems, Volume II: Semi-Markov and Decision Processes
    Ebook1,051 pages8 hours

    Dynamic Probabilistic Systems, Volume II: Semi-Markov and Decision Processes

    By Ronald A. Howard

    ()

    Read preview

    About this ebook

    This book is an integrated work published in two volumes. The first volume treats the basic Markov process and its variants; the second, semi-Markov and decision processes. Its intent is to equip readers to formulate, analyze, and evaluate simple and advanced Markov models of systems, ranging from genetics and space engineering to marketing. More than a collection of techniques, it constitutes a guide to the consistent application of the fundamental principles of probability and linear system theory.
    Author Ronald A. Howard, Professor of Management Science and Engineering at Stanford University, continues his treatment from Volume I with surveys of the discrete- and continuous-time semi-Markov processes, continuous-time Markov processes, and the optimization procedure of dynamic programming. The final chapter reviews the preceding material, focusing on the decision processes with discussions of decision structure, value and policy iteration, and examples of infinite duration and transient processes. Volume II concludes with an appendix listing the properties of congruent matrix multiplication.
    LanguageEnglish
    PublisherDover Publications
    Release dateJan 18, 2013
    ISBN9780486152004
    Dynamic Probabilistic Systems, Volume II: Semi-Markov and Decision Processes

    Related to Dynamic Probabilistic Systems, Volume II

    Titles in the series (100)

    View More
    View More

    Related ebooks

    Mathematics For You

    View More
    View More

    Related podcast episodes

    Related articles

    • Article

      Math After COVID-19

      Apr 28, 2020

      Math After COVID-19
      4 min read

    • Article

      Are Neural Networks About to Reinvent Physics?

      Nov 21, 2019

      Can AI teach itself the laws of physics? Will classical computers soon be replaced by deep neural networks? Sure looks like it, if you’ve been following the news, which lately has been filled with headlines like, “A neural net solves the three-body p

      9 min read

    • Article

      Why Physics Can’t Tell Us What Life Is

      Oct 21, 2020

      There is just something obviously reasonable about the following notion: If all life is built from atoms that obey precise equations we know—which seems to be true—then the existence of life might just be some downstream consequence of these laws tha

      11 min read

    • Article

      The Strange Inevitability of Evolution: Good solutions to biology’s problems are astonishingly plentiful.

      Oct 27, 2016

      Is the natural world creative? Just take a look around it. Look at the brilliant plumage of tropical birds, the diverse pattern and shape of leaves, the cunning stratagems of microbes, the dazzling profusion of climbing, crawling, flying, swimming th

      15 min read

    • Article

      ‘New Force Of Nature’: Scientists ‘Cautiously Excited’ As Experiment Points Towards New Era In Our Understanding Of The Universe

      Mar 23, 2021

      Scientists studying the fundamental forces which govern the universe and everything in it are “cautiously excited” recent experiments could lead to a “new era” in our understanding of physics. While much scientific experimentation confirms existing h

      4 min read

    • Article

      The Strange Similarity of Neuron and Galaxy Networks

      Jul 20, 2017

      Christof Koch, a leading researcher on consciousness and the human brain, has famously called the brain “the most complex object in the known universe.” It’s not hard to see why this might be true. With a hundred billion neurons and a hundred trillio

      6 min read

    • Article

      The Strange Similarity of Neuron and Galaxy Networks

      Jul 25, 2019

      Christof Koch, a leading researcher on consciousness and the human brain, has famously called the brain “the most complex object in the known universe.” It’s not hard to see why this might be true. With a hundred billion neurons and a hundred trillio

      6 min read

    • Article

      Quantum Mechanics 101

      Dec 5, 2019

      Quantum Mechanics 101
      7 min read

    • Article

      Why Physics Is Not a Discipline: Physics is not just what happens in the Department of Physics.

      Apr 21, 2016

      Have you heard the one about the biologist, the physicist, and the mathematician? They’re all sitting in a cafe watching people come and go from a house across the street. Two people enter, and then some time later, three emerge. The physicist says,

      12 min read

    • Article

      Schrödinger’s Cat When Nobody Is Looking

      Aug 26, 2020

      Some of the most perplexing topics in physics revolve around quantum theory. The quandary is seen most famously in the Schrödinger’s cat question and the issue of information loss in black hole evaporation. Richard Feynman said, “I think that I can s

      13 min read

    • Article

      Can We Change The Past?

      Feb 21, 2018

      Can We Change The Past?
      4 min read

    • Article

      A Theory of Consciousness Can Help Build a Theory of Everything

      May 4, 2017

      For an empirical science, physics can be remarkably dismissive of some of our most basic observations. We see objects existing in definite locations, but the wave nature of matter washes that away. We perceive time to flow, but how could it, really?

      15 min read

    • Article

      Heavyweight Particle

      Jun 12, 2022

      Heavyweight Particle
      13 min read

    • Article

      What Quantum Gravity Needs Is More Experiments

      Feb 2, 2017

      In the mid-1990s, I studied mathematics. I wasn’t really sure just what I wanted to do with my life, but I was awed by the power of mathematics to describe the natural world. After classes on differential geometry and Lie algebras, I attended a semin

      8 min read

    • Article

      What Quantum Gravity Needs Is More Experiments

      Dec 12, 2019

      In the mid-1990s, I studied mathematics. I wasn’t really sure just what I wanted to do with my life, but I was awed by the power of mathematics to describe the natural world. After classes on differential geometry and Lie algebras, I attended a semin

      8 min read

    • Article

      Neuroscience Weighs in on Physics’ Biggest Questions

      Oct 13, 2021

      For an empirical science, physics can be remarkably dismissive of some of our most basic observations. We see objects existing in definite locations, but the wave nature of matter washes that away. We perceive time to flow, but how could it, really?

      15 min read

    • Article

      Books

      Oct 19, 2023

      Shohini Ghose MIT Press £27 • HB How many women scientists can you name? A 2017 US study showed that less than one per cent of people surveyed could name a living woman scientist. Have you sat in a lecture and been greeted with ‘good morning, gentlem

      5 min read

    • Article

      How to Build a Search Engine for Mathematics: The Surprising Power of Neil Sloane’s Encyclopedia of Integer Sequences.: The surprising power of Neil Sloane’s Encyclopedia of Integer Sequences.

      Oct 22, 2015

      On the average summer Saturday, the mathematician Neil Sloane woke up to a crisis. “There are always crises,” he said— albeit crises of the teapot tempest variety. One Saturday over breakfast, he faced an inbox message titled “edits from outer space.

      10 min read

    • Article

      Teaching Me Softly: Machine learning is teaching us the secret to teaching.

      Sep 8, 2016

      When Pyotr Stolyarsky died in 1944, he was considered Russia’ s greatest violin teacher. He counted among his pupils a coterie of stars, including David Oistrakh and Nathan Milstein, and a school for gifted musicians in his native Odessa was named af

      8 min read

    Related categories

    Reviews for Dynamic Probabilistic Systems, Volume II

    0 ratings

    0 ratings0 reviews

    What did you think?

    Tap to rate

    Review must be at least 10 words

      Book preview

      Dynamic Probabilistic Systems, Volume II - Ronald A. Howard

      DOVER BOOKS ON MATHEMATICS

      HANDBOOK OF MATHEMATICAL FUNCTIONS, Milton Abramowitz and Irene A. Stegun. (0-486-61272-4)

      TENSOR ANALYSIS ON MANIFOLDS, Richard L. Bishop and Samuel 1. Goldberg. (0-486-64039-6)

      VECTOR AND TENSOR ANALYSIS WITH APPLICATIONS, A. I. Borisenko and I. E. Tarapov. (0-486-63833-2)

      THE HISTORY OF THE CALCULUS AND ITS CONCEPTUAL DEVELOPMENT, Carl B. Boyer. (0-486-60509-4)

      THE QUALITATIVE THEORY OF ORDINARY DIFFERENTIAL EQUATIONS: AN INTRODUCTION, Fred Brauer and John A. Nohel. (0-486-65846-5)

      PRINCIPLES OF STATISTICS, M. G. Bulmer. (0-486-63760-3)

      THE THEORY OF SPINORS, Élie Cartan. (0-486-64070-1)

      ADVANCED NUMBER THEORY, Harvey Cohn. (0-486-64023-X)

      STATISTICS MANUAL, Edwin L. Crow, Francis Davis, and Margaret Maxfield. (0-486-60599-X)

      FOURIER SERIES AND ORTHOGONAL FUNCTIONS, Harry F. Davis. (0-486-65973-9)

      COMPUTABILITY AND UNSOLVABILITY, Martin Davis. (0-486-61471-9)

      ASYMPTOTIC METHODS IN ANALYSIS, N. G. de Bruijn. (0-486-64221-6)

      THE MATHEMATICS OF GAMES OF STRATEGY, Melvin Dresher. (0-486-64216-X)

      APPLIED PARTIAL DIFFERENTIAL EQUATIONS, Paul DuChateau and David Zachmann. (0-486-41976-2)

      ASYMPTOTIC EXPANSIONS, A. Erdélyi. (0-486-60318-0)

      COMPLEX VARIABLES: HARMONIC AND ANALYTIC FUNCTIONS, Francis J. Flanigan. (0-486-61388-7)

      DIFFERENTIAL TOPOLOGY, David B. Gauld. (0-486-45021-X)

      ON FORMALLY UNDECIDABLE PROPOSITIONS OF PRINCIPIA MATHEMATICA AND RELATED SYSTEMS, Kurt Gödel. (0-486-66980-7)

      A HISTORY OF GREEK MATHEMATICS, Sir Thomas Heath. (0-486-24073-8, 0-486-24074-6) Two-volume set

      PROBABILITY: ELEMENTS OF THE MATHEMATICAL THEORY, C. R. Heathcote. (0-486-41149-4)

      INTRODUCTION TO NUMERICAL ANALYSIS, Francis B. Hildebrand. (0-486-65363-3)

      METHODS OF APPLIED MATHEMATICS, Francis B. Hildebrand. (0-486-67002-3)

      TOPOLOGY, John G. Hocking and Gail S. Young. (0-486-65676-4)

      MATHEMATICS AND LOGIC, Mark Kac and Stanislaw M. Ulam. (0-486-67085-6)

      MATHEMATICAL FOUNDATIONS OF INFORMATION THEORY, A. I. Khinchin. (0-486-60434-9)

      ARITHMETIC REFRESHER, A. Albert Klaf. (0-486-21241-6)

      CALCULUS REFRESHER, A. Albert Klaf. (0-486-20370-0)

      PROBLEM BOOK IN THE THEORY OF FUNCTIONS, Konrad Knopp. (0-486-41451-5)

      INTRODUCTORY REAL ANALYSIS, A. N. Kolmogorov and S. V. Fomin. (0-486-61226-0)

      SPECIAL FUNCTIONS AND THEIR APPLICATIONS, N. N. Lebedev. (0-486-60624-4)

      CHANCE, LUCK AND STATISTICS, Horace C. Levinson. (0-486-41997-5)

      TENSORS, DIFFERENTIAL FORMS, AND VARIATIONAL PRINCIPLES, David Lovelock and Hanno Rund. (0-486-65840-6)

      SURVEY OF MATRIX THEORY AND MATRIX INEQUALITIES, Marvin Marcus and Henryk Mine. (0-486-67102-X)

      ABSTRACT ALGEBRA AND SOLUTION BY RADICALS, John E. and Margaret W. Maxfield. (0-486-67121-6)

      FUNDAMENTAL CONCEPTS OF ALGEBRA, Bruce E. Meserve. (0-486-61470-0)

      FUNDAMENTAL CONCEPTS OF GEOMETRY, Bruce E. Meserve. (0-486-63415-9)

      FIFTY CHALLENGING PROBLEMS IN PROBABILITY WITH SOLUTIONS, Frederick Mosteller. (0-486-65355-2)

      NUMBER THEORY AND ITS HISTORY, Oystein Ore. (0-486-65620-9)

      MATRICES AND TRANSFORMATIONS, Anthony J. Pettofrezzo. (0-486-63634-8)

      THE UMBRAL CALCULUS, Steven Roman. (0-486-44139-3)

      PROBABILITY THEORY: A CONCISE COURSE, Y. A. Rozanov. (0-486-63544-9)

      LINEAR ALGEBRA, Georgi E. Shilov. (0-486-63518-X)

      ESSENTIAL CALCULUS WITH APPLICATIONS, Richard A. Silverman. (0-486-66097-4)

      INTERPOLATION, J.F. Steffensen. (0-486-45009-0)

      A CONCISE HISTORY OF MATHEMATICS, Dirk J. Struik. (0-486-60255-9)

      PROBLEMS IN PROBABILITY THEORY, MATHEMATICAL STATISTICS AND THEORY OF RANDOM FUNCTIONS, A. A. Sveshnikov. (0-486-63717-4)

      TENSOR CALCULUS, J. L. Synge and A. Schild. (0-486-63612-7)

      MODERN ALGEBRA: Two VOLUMES BOUND As ONE, B.L. Van der Waerden. (0-486-44281-0)

      CALCULUS OF VARIATIONS WITH APPLICATIONS TO PHYSICS AND ENGINEERING, Robert Weinstock. (0-486-63069-2)

      INTRODUCTION TO VECTOR AND TENSOR ANALYSIS, Robert C. Wrede. (0-486-61879-X)

      DISTRIBUTION THEORY AND TRANSFORM ANALYSIS, A. H. Zemanian. (0-486-65479-6)

      Copyright

      Copyright © 1971, 1999, 2007 by Ronald A. Howard

      All rights reserved.

      Bibliographical Note

      This Dover edition, first published in 2007, is an unabridged republication of the work originally published by John Wiley & Sons, Inc., New York, in 1971. The author has written a new Preface to the Dover Edition.

      International Standard Book Number

      9780486152004

      Manufactured in the United States of America

      Dover Publications, Inc., 31 East 2nd Street, Mineola, N.Y 11501

      To Polly,

      my trapping state.

      PREFACE TO THE DOVER EDITION

      When these volumes were published over three decades ago, a major difficulty in their application was assigning the necessary transition probabilities and holding-time distributions. The original motivation 50 years ago was a rare business application where the relevant information was available. The story of this application appears in:

      Comments on the Origin and Application of Markov Decision Processes, Operations Research, Vol. 50, No. 1, January-February 2002, pp. 100-102

      With the development of modem data acquisition and processing systems in many current enterprises, data paucity has been changed to data abundance, if not data overload. Decision methods that could only rarely be implemented are now widely economically feasible. The republication of this work will bring these methods to those who can use them.

      RONALD A. HOWARD

      Stanford University

      February, 2007

      PREFACE

      Uncertainty, complexity, and dynamism have been continuing challenges to man’s understanding and control of his physical environment. In the development of logical structures to describe these phenomena, the model originated by A. A. Markov stands as a major accomplishment. Where previous contributors had modeled uncertainty as a sequence of independent trials, Markov saw the advantage of introducing dependence of each trial on the result of its predecessor. While it is tempting to consider even more complex dependency of the present trial on the results of past trials, such temptation usually leads to results that are both analytically and computationally intractable.

      Consequently, Markov models represent the first outpost in the domain of dependent models that is powerful both in capturing the essence of many dependent systems observed in practice and in producing the analytical and computational results necessary to gain insight into their behavior. The purpose of this book is to equip the reader to formulate, analyze, and evaluate simple and advanced Markov models of systems that range from genetics to space engineering to marketing. This book should be viewed not as a collection of techniques (although many will value it as such), but rather as a consistent application of the fundamental principles of probability and linear system theory.

      It is often said that good ideas are simple; the Markov process is no exception. In fact, there is no problem considered in this book that cannot be made clear to a child. The device we use to make such expositions simple is a pond covered with lily pads among which a frog may jump. Although his jumps may be random, the frog never falls into the water. Furthermore, he may occupy the same pad after successive jumps. We can use the lily pond analogy to clarify any discussion in the text.

      The book is an integrated work published in two volumes. The first volume, Chapters 1—9, treats the basic Markov process and its variants; the second, Chapters 10—15, semi-Markov and decision processes. Although the reader who is already familiar with Markov process terminology will understand the design of the book from the Contents, it should be helpful to all readers to discuss each chapter using the lily pond analogy.

      Chapter 1 introduces the basic Markov model. The number of pads in the lily pond is assumed to be finite. The Markov assumption states that the probability of the frog’s making his next jump to any pad depends only on the pad he currently occupies and not on how he reached his present pad. The discussion then considers the question, given the starting pad and the fact that n jumps have been made without our observing the pond, what probability must we assign to the frog’s occupying each pad in the pond. Graphical and transform methods are used to find these probabilities and to illustrate the typical behavior of a variety of processes.

      Chapter 2 is a diversion to present the theory of linear systems and graphical methods for their analysis. Chapter 3 shows how to apply these powerful techniques to answering the problems of Chapter 1 with respect to the lily pond.

      Chapter 4 considers the case where certain pads, once occupied, cannot be left and investigates the statistics for the number of jumps required before such trapping occurs. Chapter 5 treats two types of statistics: First, how many jumps will be made onto each pad in a given number of total jumps and, second, how many jumps will be required to reach one pad from another pad for the first time.

      Chapter 6 finds the statistics of jumps from one set of pads to another set of pads. It allows identifying certain jumps as being of special interest and then finding the statistics of these jumps. Furthermore, it considers the implications of the information provided by these jumps for probability assignments on the frog’s location.

      Chapter 7 focuses on the case where we observe only one pad and where the frog’s returns to that pad can be governed by an arbitrary probability distribution. We explore the statistics of these returns from several points of view. This chapter also discusses for the first time the possibility that the pond may have an infinite number of pads and illustrates the analytic modifications necessary to treat this situation.

      Chapter 8 investigates the possibility of having several frogs jumping in the pond at the same time. It expands the discussion to the case where at certain times frogs may be added to each pad in the pond in varying numbers. Finally, it treats the case where the frogs may breed to produce new generations of jumpers.

      Chapter 9 considers the case where the probability of a jump from one pad to another may be different on successive jumps. For example, the frog may be getting tired.

      Chapter 10 introduces another dimension to the discussion. We explicitly consider the time between jumps as a multiple of some time unit and let this be a random variable that depends on the jump made by the frog. We can now consider the probability of the frog’s occupying each pad given both the number of jumps he has made and the total time units he has been jumping. All previous questions about the frog’s behavior are re-examined in this generalized form.

      Chapter 11 allows the time between jumps to be a continuous random variable and extends all previous discussions to this case. Chapter 12 considers the continuous-time Markov process, the case in which the time that the frog has occupied the same pad has no bearing on the amount of time he will continue to occupy it. It also discusses the infinite-pad lily pond once again.

      Chapter 13 concerns the payment of rewards on the basis of the frog’s jumping. He may earn a reward for making a particular jump or from occupying a pad. We develop and illustrate expressions for the expected total reward the frog will earn in a finite and infinite time with and without discounting.

      Chapter 14 presents another diversion, a discussion of the optimization procedure of dynamic programming, the natural way to formulate optimization problems for Markov processes.

      Chapter 15 combines the material of the last several chapters to develop procedures for optimizing the expected reward generated by the frog. Control is effected by altering the probabilistic structure and reward structure of his jumping.

      Since the level of interest in frog-jumping is relatively low, the examples throughout are drawn from a variety of fields: consumer purchasing, taxicab operation, inventory control, rabbit reproduction, coin-tossing, gambling, family-name extinction, search, car rental, machine repair, depreciation, production, action-timing, reliability, reservation policy, machine maintenance and replacement, network traversal, project scheduling, space exploration, and success in business. The applicability of the models to still other areas is also made evident.

      The background suggested for the reader if he is to achieve greatest benefit from the book is a foundation in calculus, probability theory, and matrix theory. The emphasis is on the fundamental rather than the esoteric in these areas, so preparation should not be a major stumbling block. The book may be read with ease by anyone with the basic background whether his field be engineering, management, psychology, or whatever. Specialized material from any field used in examples is presented as required. Consequently, the professional reading independently to expand his knowledge of Markov models should be able to proceed rapidly and effectively.

      This book developed from courses taught at M.I.T. and Stanford University over a decade. Although the advanced undergraduate can master the subject with no difficulty, the course has always been offered at the graduate level. The material is developed naturally for course presentation. To cover the whole book typically requires the entire academic year. The two volumes serve well as texts for successive one-semester courses. If a two-quarter version is desired, then Chapters 8 and 9 can be placed in the optional category.

      One feature of the development that is becoming increasingly advantageous to exploit is the suitability of examples and problems for solution by computer, particularly the time-shared variety. Virtually every chapter contains material appropriate for computer demonstration, such as simulation of Markov process behavior, or solution for statistics via difference equations. While such integration with computer methods is not necessary, it does provide an opportunity for curriculum unification.

      Some of the material presented here has appeared in more primitive form elsewhere. For example, Chapter 2 is a highly revised form of System Analysis of Linear Models, Multistage Inventory Models and Techniques, eds., Scarf, Shelley, Guilford, 143—184, Stanford University Press, 1963; Chapter 13 is an expanded version of Dynamic Programming, Management Science, Vol. 12, No. 5, 317—348, January 1966. Much of the development in the second volume is based on System Analysis of Semi-Markov Processes, Trans. IEEE Prof. Group on Mil. Elec., Vol. MIL—8, No. 2, 114—124, April 1964, and on Semi-Markovian Decision Processes, Proceedings of the 34th Session International Statistical Institute, 625—652, Ottawa, Canada, August 1963.

      Finally, some credit where credit is due. Much of the original work on this book was performed during a visiting year at Stanford University under the sponsorship of what has become the Department of Engineering-Economic Systems. Most of the final touches were supplied during a visiting year as Ford Research Professor at the Stanford Graduate School of Business. The book contains research results that grew from supported research. Most of the early research was sponsored by the Office of Naval Research; more recently the research was supported by the National Science Foundation. To all these organizations I express my appreciation.

      The question of personal acknowledgments is a difficult one for any author. Only he knows the debt he owes to so many over a period of several years. By selecting a few for special mention, he accepts the risk that the important contributions of others will not receive sufficient notice. Accepting that risk, let me reveal how this work is the result of many contributors.

      Several individuals have provided me with advice, criticism, and suggestions on the development of the manuscript. In the early versions, I benefited greatly from my contact with Edward S. Silver and Jerome D. Herniter. Richard D. Smallwood has provided a continuing stream of suggestions that have materially improved the manuscript. Herbert F. Ayres and W. Howard Cook have each made several suggestions that clarified the text. Finally, I must credit the Herculean task of my friend James E. Matheson who read through the completed manuscript in detail with no hope of reward but the promise of a feast in a fancy New York restaurant. Having fulfilled that promise, I am happy to report that he has found and corrected all mistakes in the manuscript, thus relieving me of the necessity of making that boring statement that any idiocies in the book are the full responsibility of the author.

      I believe that there is no more important part of a textbook than its problems, and so I asked Richard D. Smallwood to collate and augment the problem sets we have used in courses over the years. He has done a thorough and imaginative job and has demonstrated that the challenging need not be humorless. Many of the problems were originally composed by teaching assistants; age and evolutionary change have made their authorship obscure. However, in many I discern the deft touch of Edward A. Silver who has a special knack for making a student smile and sweat at the same time.

      Any book that would illustrate complex Markov models will require extensive electronic computation. I have been fortunate in having the services of individuals who have unusual flair and ability in this area, namely, Claude Dieudonné, Richard D. Smallwood, Paul Schweitzer, Leif Tronstad, and Alain Viguier. Their contribution ranged from solution of numerical examples to the development of programming systems that treated large classes of models: The reader is the beneficiary of their ingenuity.

      A challenging part of the manuscript was the translation of Markov’s paper. Here I needed someone who could translate classic Russian into English—I found him in George Petelin. Mr. Petelin’s English translation provided the basis for the final version that appears in the book; for its accuracy I personally bear full responsibility.

      Obviously a project of this magnitude is a secretarial nightmare. Fortunately, I have found myself over the years in the hands of a succession of ladies whose secretarial skills were exceeded only by their intelligence and charm. The first was Mrs. Nobuko McNeill who personally supervised the typing of over one thousand pages of Gregg shorthand directly into finished manuscript. The second was Mrs. Edna Tweet who prepared the final chapter of Volume II. The third was Mrs. Louise Goodrich who suffered through the endless revision, galley reading, and proof checking necessary to bring this project to a successful conclusion.

      Therefore, let me express to all these fine people my appreciation and thanks for helping me tell my story about the Markov process and what became of it.

      Palo Alto, California

      January, 1971

      RONALD A. HOWARD

      Table of Contents

      DOVER BOOKS ON MATHEMATICS

      Title Page

      Copyright Page

      Dedication

      PREFACE TO THE DOVER EDITION

      PREFACE

      10 - THE DISCRETE-TIME SEMI-MARKOV PROCESS

      11 - THE CONTINUOUS-TIME SEMI-MARKOV PROCESS

      12 - CONTINUOUS-TIME MARKOV PROCESSES

      13 - REWARDS

      14 - DYNAMIC PROGRAMMING

      15 - SEMI-MARKOV DECISION PROCESSES

      NOTATION

      APPENDIX - A PROPERTIES OF CONGRUENT MATRIX MULTIPLICATION

      REFERENCES

      INDEX

      A CATALOG OF SELECTED DOVER BOOKS IN SCIENCE AND MATHEMATICS

      10

      THE DISCRETE-TIME SEMI-MARKOV PROCESS

      All of the Markov models we discussed in Volume I have the property that a transition is made at every time instant. The transition may return the process to the state it previously occupied, but a transition occurs nevertheless. Now we want to turn our attention to a more general class of processes where the time between transitions may be several of the unit time-intervals, and where this transition time can depend on the transition that is made. As we shall see, this process is no longer strictly Markovian. However, it retains enough of the Markovian properties to deserve the name of a semi-Markov process. In this chapter we shall define and investigate the semi-Markov process and reveal the additional flexibility it brings to the problem of modeling dynamic probabilistic systems.

      10.1 THE FORMAL MODEL

      We can think of the semi-Markov process as a process whose successive state occupancies are governed by the transition probabilities of a Markov process, but whose stay in any state is described by an integer-valued random variable that depends on the state presently occupied and on the state to which the next transition will be made. Thus at transition instants the semi-Markov process behaves just like a Markov process. We call this process the imbedded Markov process. However, the times at which transitions occur are governed by a different probabilistic mechanism.

      To make these notions precise, let pij be the probability that a semi-Markov process that entered state i on its last transition will enter state j on its next transition. The transition probabilities pij must satisfy the same equations as the transition probabilities for a Markov process,

      (10.1.1)

      and

      (10.1.2)

      where, as usual, N is the total number of states in the system. Whenever a process enters a state i, we imagine that it determines the next state j to which it will move according to state i’s transition probabilities pi1, pi2, . . . , piN. However, after j has been selected, but before making this transition from state i to state j, the process holds for a time τij in state i. The holding times τij are positive, integer-valued random variables each governed by a probability mass function hij(·) called the holding time mass function for a transition from state i to staté j. Thus,

      (10.1.3)

      of all holding time distributions are finite and that all holding times are at least one time unit in length,

      (10.1.4)

      We must specify N² holding time mass functions, in addition to the transition probabilities, to describe a discrete-time semi-Markov process completely.

      After holding in state i for the holding time τij, the process makes the transition to state j, and then immediately selects a new destination state k using the transition probabilities pj1, pj2, . . . , pjN. It next chooses a holding time τjk in state j according to the mass function hjk(·), and makes its next transition at time τjk after entering state j. The process continues developing its trajectory in this way indefinitely.

      Figure 10.1.1 shows a portion of a possible trajectory for a discrete-time semi-Markov process. The large black dots indicate the state of the process immediately after a transition. Thus the figure shows that the process entered state 2 at time 0, held in state 2 for three time units, then made a transition to state 4. It held in state 4 for one time unit and then moved to state 3 at time 4. The process remained in state 3 for two time units before making a transition to state 1. When last seen, the process had stayed at least 2 time units in state 1.

      We see that we have removed the basic connection between the time scale and the time of transition. We can thus talk about the probability of 3 transitions in a total time 7, etc. We also observe that we can consider the discrete-time Markov process we have been discussing in earlier chapters to be a discrete-time semi-Markov process for which

      (10,1.5)

      that is, all holding times are exactly one time unit in length.

      Figure 10.1.1 A possible semi-Markov process trajectory.

      Real and Virtual Transitions

      In Section 4.3 (Volume I) we discussed a subject that is closely related to the topic of this chapter. We made a distinction between real transitions, which required an actual change of state indices as the result of a transition, and virtual transitions, where the state indices could be the same after the transition. We saw that we could consider the discrete-time Markov process as a process that could make only real transitions and had geometric holding times, or as a process that could make virtual transitions and had all holding times equal to one unit. From either point of view, the discrete-time Markov process is a discrete-time semi-Markov process.

      The distinction between real and virtual transitions is useful in the theory of the semi-Markov process itself. When modeling real systems by semi-Markov processes, we must often decide whether we want to call a movement from a state to itself a transition of the process. The decision depends on the circumstances: Some physical processes require that only real transitions be allowed, in other processes the virtual transitions are most important. The impact of the decision on the possible trajectories of the process is indicated in Figure 10.1.2. We observe that when virtual transitions are allowed, transitions can occur with no change of state. We have already noted that when the state of the process represents the last brand purchased by the customer in a marketing model, a virtual transition represents a repeat purchase of a brand, an event of frequent importance to the analyst. Since the semi-Markov process is an even more general model for marketing and other processes than is the Markov process, we must preserve our ability to speak of virtual transitions. Therefore, in our future developments we shall always allow the possibility of virtual transitions—the transition probabilities pii may or may not be zero.

      Figure 10.1.2 Real and virtual transitions. (a) Real transitions only. (b) Virtual transitions allowed.

      Holding Times and Waiting Times

      We shall find it useful to develop additional notation for the holding time behavior. We use ≤hij(·) for the cumulative probability distribution of τij,

      (10.1.6)

      and >hij(·) for the complementary cumulative probability distribution of τij,

      (10.1.7)

      Suppose now that the process enters state i and chooses a successor state j, but we as observers do not know the successor chosen. The probability mass function we would assign to the time τi spent in i, we shall then call wi( ), where

      (10.1.8)

      that is, the probability that the system will spend m time units in state i if we do not know its successor state is the probability that it will spend m time units in state i if its successor state is j multiplied by the probability its successor state is j and summed over all possible successor states. We shall call τi the waiting time in state i, and w) the waiting time probability mass function. Thus, a waiting time is merely a holding time that is unconditional on the destination state.

      by

      (10.1.9)

      using

      (10.1.10)

      , as

      (10.1.11)

      In view of Equation 10.1.7, the cumulative and complementary cumulative probability distributions for the waiting times are

      (10.1.12)

      and

      (10.1.13)

      A Car Rental Example

      Before proceeding to more theoretical developments, let us show a very simple example of a discrete-time semi-Markov process. An automobile rental agency rents cars at two locations, town 1 and town 2. Consider one car and let its state be the number of the town at which it was last rented. Thus we have a two-state system.

      Transition probabilities

      The experience of the company shows that when a car is rented in town 1 there is a 0.8 probability that it will be returned to town 1 and a 0.2 probability that it will be returned to town 2. When the car is rented in town 2, there is a 0.7 probability that it will be returned to town 2 and a 0.3 probability that it will be returned to town 1. We assume that there are always many customers available at both towns and that cars are always rented at the towns to which they are last returned. Then we can represent the transition probabilities of this imbedded Markov process by the transition probability matrix

      (10.1.14)

      Note that this is the same transition probability matrix we used in our two-state Markov process marketing example.

      Holding times

      But now we give the example a semi-Markovian flavor. Because of the nature of the trips involved, the length of time a car will be rented depends on both where it is rented and where it is returned. The holding time τij is thus the length of time a car will be rented if it was rented at town i and returned to town j. The four possible holding time probability mass functions are found from the company’s records to be:

      (10.1.15)

      Note that these holding time distributions are all geometric distributions with different parameters. However, they could have been distributions of arbitrary complexity. As we found in Section 4.1, the geometric distribution (1 — a)an—1, n = 1, 2, 3, has a mean 1/(1 — a), second moment (1 + a)/(1 — a)² , and variance a/(1 — a)². Therefore the moments of our four holding times are:

      (10.1.16)

      These numbers indicate that people renting cars at town 2 and returning them to town 2 often have long rental periods; perhaps town 2 is in the center of a scenic region.

      ) with elements h). For the example,

      (10.1.17)

      ), we have created a complete description of the semi-Markov process. Figure 10.1.3 shows how this description can be represented in a transition diagram. Each branch in the diagram must be labeled not only with the transition probability that corresponds to that branch, but also with the holding time mass function for the branch.

      Figure 10.1.3 A discrete semi-Markov example—car rental problem.

      If hij(m) is the geometric distribution (1 – a)am – 1 m = 1, 2, 3, . . . , then the cumulative and complementary cumulative distributions ≤h¡j(n) and >hij(n) are

      (10.1.18)

      and

      (10.1.19)

      Therefore, the matrix forms of these distributions for the example are

      (10.1.20)

      and

      (10.1.21)

      These results show, for example, that the chance that a car rented in town 1 and returned to town 2 will be rented for n or fewer time periods is 1—(5/6)n. A car rented in 2 and returned to 1 has a chance (3/4)n of being rented for more than n periods.

      Waiting times

      We can use Equations 10.1.8 through 10.1.13 to find the waiting time statistics for the example. We start with the means,

      (10.1.22)

      The mean time that a car rented in town 1 will be rented, destination unknown, is 3.6 periods. If the car is rented in town 2, the mean is 9.6 periods.

      , from Equation 10.1.10,

      (10.1.23)

      then the variance of the waiting times from Equation 10.1.11,

      (10.1.24)

      For the distributions of waiting time we find

      (10.1.25)

      These expressions give the probability that a car rented in each town will be rented for m periods, destination unknown. We use Equations 10.1.12 and 10.1.13 to compute from Equations 10.1.20 and 10.1.21 the cumulative and complementary cumulative distributions of waiting time,

      (10.1.26)

      and

      (10.1.27)

      The expression for >w2(n), for example, shows the probability that a car rented in town 2 will be rented for more than n periods if its destination is unknown.

      10.2 THE INTERVAL TRANSITION PROBABILITIES

      The central statistics in semi-Markov processes are the interval transition probabilities, quantities that correspond to the multistep transition probabilities for the Markov process. For mnemonic reasons we shall use the same notation we used in Markov processes. We define Φij(n) as the probability that a discrete-time semi-Markov process will be in state j at time n given that it entered state i at time zero. We call this probability the interval transition probability from state i to state j in the interval (0, n). Note that an essential part of the definition is that the system entered state i at time zero as opposed to its simply being in state i at time zero. We must be precise on this point because, as we shall see, the length of time a process has occupied a state can often affect the probabilities we assign to the destination of its next transition.

      We now turn to the question of developing an expression for the interval transition probabilities. How can a process that started by entering state i at time zero be in state j at time n? One way this can happen is for i and j to be the same state and for the process never to have left state i throughout the period (0, n). This requires that the process make its first transition after time n. Every other way to get from state i to state j in the interval (0, n) requires that the process make at least one transition during that interval. For example, the process could have made its first transition from state i to some state k at a time m, 0 < m ≤ n, and then by some succession of transitions have made its way to state j at time n. These considerations lead us to the equation,

      (10.2.1)

      The quantity δij ensures that the term in which it appears occurs only when i = j. The complementary cumulative waiting time probability >w¡(n) is the probability that the process will leave its starting state i at a time greater than n. The second term in the equation represents the probability of the sequence of events where the process makes its first transition from the starting state i to some state k (maybe i itself) at some time m and then proceeds somehow from state k to state j in the remaining time n — m. This probability is summed over all states k to which the initial transition could have been made and over all times of first transition m between 1 and n. Extending the summation to include m = 0 is innocuous because hij(0) = 0.

      Matrix Formulation

      Let us now place Equation 10.2.1 in matrix form. In addition to the usual matrices denoted by the corresponding upper case letters, we define a set of diagonal waiting time matrices by giving their elements,

      (10.2.2)

      ), we write Equation 10.2.1 as

      (10.2.3)

      We call Φ(n) the interval transition probability matrix for the semi-Markov process in the interval (0, n); note that

      (10.2.4)

      Equation 10.2.3 provides a convenient recursive basis for computing the interval transition probability matrix for any semi-Markov process. The quantities P and H(m) come directly from the definition of the process. From Equation 10.1.13 we see that we can compute >wi(n), the ith diagonal element of >W(n), as the sum from m = n + 1 to infinity of the sum of the elements in the ith row of P H(m),

      (10.2.5)

      The core matrix

      We observe that computation of the interval transition probabilities rests only on the matrix P H(m). This matrix we shall call the core matrix of the semi-Markov process, and give it the symbol C(m),

      (10.2.6)

      The ijth element of C(m) is the probability of the joint event that a system that entered state i at time zero makes its next transition to state j and makes that transition at time m. If we sum the elements of C(m) across the ith row, we obtain the waiting time mass function for the ith state,

      (10.2.7)

      a result that leads directly to Equation 10.2.5. If we sum the ijth element of C(m) for all values of m, we obtain the transition probability pij,

      (10.2.8)

      We shall usually write the core matrix C(m) H(m) to emphasize the relationship between semi-Markov and Markov processes.

      Specialization to a Markov process

      We can check Equation 10.2.3 by applying it to the discrete-time Markov process. For this process, as we observed in Equation 10.1.5, the holding time mass function matrix is

      (10.2.9)

      We calculate the waiting time mass functions from

      (10.2.10)

      Therefore the complementary cumulative waiting time distribution for each state is

      (10.2.11)

      that is, there is probability one that the waiting time will exceed 0, and probability zero that it will exceed any number equal to or greater than one. This makes sense because the waiting time is always exactly equal to one, just like the holding time for all states. We could, of course, have made these observations without mathematical guidance. However, we have now established that the complementary cumulative waiting time matrix for a discrete-time Markov process is just the identity matrix at time zero and a zero matrix for all other times,

      (10.2.12)

      We now substitute the results of Equations 10.2.9 and 10.2.12 into the recursive Equation 10.2.3 for the interval transition probabilities,

      (10.2.13)

      This is, of course, just the equation we wrote in Chapter 1 for the multistep transition probability matrix for the discrete-time Markov process. We know that its solution is

      (10.2.14)

      Unfortunately, it will seldom be the case that we can solve Equation 10.2.3 in such a simple form. Generally we shall require a computing machine to assist us.

      The car rental example

      Let us apply this equation to the car rental example to show how the calculation proceeds and what type of results are generated. First we construct the core matrix C(m) = P H(m) for the example from Equations 10.1.14 and 10.1.17,

      (10.2.15)

      The row sums of P H(m) (or the results of Equation 10.1.25) allow us to construct the diagonal waiting time mass function matrix,

      (10.2.16)

      Finally, by summing this matrix from m = n + 1 to infinity (or by using Equation 10.1.27), we write the complementary cumulative waiting time distribution matrix

      (10.2.17)

      Table 10.2.1 shows the result of substituting Equations 10.2.15 and 10.2.17 into Equation 10.2.3 for a few values of n. The interval transition probabilities φ11(n) and φ22(n) are plotted in Figure 10.2.1. We need only plot these two quantities because they jointly determine the entire Φ(n) matrix. The quantity φ11,(n) is the probability that a car rented in town 1 at time zero will have town 1 as its last rental point at time n. Note that both φ11(n) and φ22(n) appear to approach limiting values when n is large. However, the tendency is much more evident in the case of φ22(n). We shall soon devote considerable attention to this limiting behavior.

      Table 10.2.1 Interval Transition Probability Matrices for Car Rental Problem

      Figure 10.2.1 Plot of interval transition probabilities φ11(n) and φ22(n) for the car rental problem.

      10.3 TRANSFORM ANALYSIS

      The recursive Equation 10.2.3 is the most practical method for finding the interval transition probabilities of discrete-time semi-Markov processes. However, for completeness and for theoretical purposes let us develop the transform analysis of such processes. We begin by writing the geometric transform of Equation 10.2.1 in our usual transform notation. Since the convolution becomes a multiplication of transforms, we write

      (10.3.1)

      We obtain the transform hikg(z) by direct geometric transformation of the holding time mass function for the i → k transition. The transform of the ith complementary cumulative waiting time distribution is directly related to the transform of the waiting time mass function. Since >wi(n) = 1—≤wi(n), Equation 10.1.13 and relations 3 and 25 of Table 1.5.1 (Vol. I) show that

      (10.3.2)

      Matrix Formulation

      We place Equation 10.3.1 in matrix form either directly or by transforming Equation 10.2.3,

      (10.3.3)

      Here we have used the symbol Cg(z) for the transformed core matrix. Equation 10.3.2 becomes

      (10.3.4)

      We can rearrange Equation 10.3.3 to produce

      (10.3.5)

      or

      (10.3.6)

      Equation 10.3.6 shows that the transform of the interval transition probability matrix for a discrete-time semi-Markov process is given by the inverse of I minus the transformed core matrix for the process postmultiplied by the transform of the complementary cumulative waiting time distribution matrix for the process. The inverse matrix will always exist for the type of process we are considering.

      The inverse matrix [I — Cg(z)]—1 has properties that parallel those of [I — Pz]-¹. In view of the series expansion,

      (10.3.7)

      we note that

      (10.3.8)

      Furthermore, for future reference,

      (10.3.9)

      or, alternately,

      (10.3.10)

      Equations 10.3.9 and 10.3.10 jointly show

      (10.3.11)

      a series of relationships that also follow directly from the series expansion of Equation 10.3.7.

      Specialization to a Markov process

      We can check the result of Equation 10.3.6 by applying it to the discrete-time Markov process. From Equation 10.2.9, we have

      (10.3.12)

      Therefore,

      (10.3.13)

      From Equation 10.2.10, we obtain

      (10.3.14)

      By inserting Equation 10.3.14 into Equation 10.3.4 or directly from Equation 10.2.12, we find

      (10.3.15)

      Now we substitute the results of Equations 10.3.13 and 10.3.15 into Equation 10.3.6 to produce

      (10.3.16)

      and we have obtained our usual expression for the transform of the multistep transition probability matrix for the discrete-time Markov process.

      The car rental example

      The car rental example illustrates the difficulty of applying transform methods to all but the smallest problems. First we compute the transformed core matrix Cg(z) Hg(z) from Equation 10.2.15,

      (10.3.17)

      Then we compute the transform of the complementary cumulative waiting time matrix from Equation 10.2.17 or from Equations 10.2.16 and 10.3.4,

      (10.3.18)

      Mean waiting time matrix. We can develop a useful check on >Wg(z) by attempting to evaluate >Wg(z) as expressed by Equation 10.3.4 at the point z = 1,

      (10.3.19)

      Since

      (10.3.20)

      the limit is indeterminate. We therefore use L’Hôpital’s rule,

      (10.3.21)

      where M , the mean waiting time in state i. This relation recalls the general result of Equation 7.2.59 that the geometric transform of the complementary cumulative probability distribution of any random variable evaluated at z = 1 is equal to the mean of the variable.

      For the car rental example, we obtain from Equation 10.3.18

      (10.3.22)

      The mean waiting times in the matrix M are just those found in Equation 10.1.22.

      Transform representation. Hg(z)

      (10.3.23)

      We obtain the determinant of this matrix after some algebraic labor as

      (10.3.24)

      then

      (10.3.25)

      Finally, we postmultiply this matrix by >Wg(z) from Equation 10.3.18 to produce the transformed interval transition probability matrix,

      (10.3.26)

      The next step is partial fraction expansion, again a laborious process,

      (10.3.27)

      Note that the rows of Φg(z) sum to 1/(1 — z) since the process must be in some state at time n,

      (10.3.28)

      Solution. Upon taking the inverse geometric transform, we obtain the interval transition probability matrix,

      (10.3.29)

      We check that Φ(0) = 1, and note that when n is very large Φ(n) approaches a limiting matrix,

      (10.3.30)

      Thus the interval transition probabilities approach a limit that is independent of starting state just as did the state probabilities of a monodesmic Markov process. We shall have more to say of this limiting behavior in a moment.

      We must not read too much into the form of Equation 10.3.29. Although the transient components of Φ(n) are geometrically weighted differential matrices, this property will not hold in general for all discrete-time semi-Markov processes. In the car rental example, it is a consequence of the geometric holding time distributions. Note that Φ(n) has two geometrically decaying components—in a two-state discrete-time Markov process Φ(n) would have only one component. While we appreciate the insight provided by such closed-form solutions, the problem of calculating Φ(n) by transform analysis is generally so difficult that we prefer to use the recursive method of Equation 10.2.3.

      The diagonal elements of Φ(n) are

      (10.3.31)

      and

      (10.3.32)

      These equations for φ11(n) and φ22(n) produce the values for these quantities already plotted in Figure 10.2.1 using the recursive method.

      Limiting Behavior of Monodesmic Processes

      Let us now turn to the question of finding the limiting behavior of interval transition probabilities over long intervals. We first note that the chain structure of a semi-Markov process is the same as that of its imbedded Markov process. Therefore the interval transition probabilities of a semi-Markov process can exhibit a unique limiting behavior only within the same chain of the imbedded process. We shall assume that we are dealing with a monodesmic imbedded Markov process and, therefore, with a monodesmic semi-Markov process. The extension to the polydesmic case is then straightforward.

      We begin by defining a limiting interval transition probability matrix Φ for the process by

      (10.3.33)

      From the final value theorem of geometric transforms, Φ is also given by

      (10.3.34)

      Since the limit of a product is the product of the limits if they exist, we can find Φ by taking limits in Equation 10.3.6,

      (10.3.35)

      We now consider separately each limit on the right side of this equation. We have already evaluated the second limit in Equation 10.3.21 where we found that it represented a diagonal matrix M of mean waiting times. If we define T(z) by

      (10.3.36)

      Enjoying the preview?
      Page 1 of 1