Partially Observed Markov Decision Processes
Filtering, Learning and Controlled Sensing

This new edition includes inverse reinforcement learning, non-parametric Bayesian inference, variational Bayes and conformal prediction.

Vikram Krishnamurthy (Author)

9781009449434, Cambridge University Press

Hardback, published 5 June 2025

652 pages
25.9 x 18.5 x 4 cm, 1.4 kg

'An outstanding advanced graduate-level introduction to the increasingly important topic of partially observed Markov decision processes. The book is a delight to read - comprehensive, clear, up-to-date and insightful while preserving rigor. An essential resource for both researchers seeking to further advance the field and practitioners wishing to implement stochastic control in real engineering systems.' Rob Evans, University of Melbourne

Covering formulation, algorithms and structural results and linking theory to real-world applications in controlled sensing (including social learning, adaptive radars and sequential detection), this book focuses on the conceptual foundations of partially observed Markov decision processes (POMDPs). It emphasizes structural results in stochastic dynamic programming, enabling graduate students and researchers in engineering, operations research, and economics to understand the underlying unifying themes without getting weighed down by mathematical technicalities. In light of major advances in machine learning over the past decade, this edition includes a new Part V on inverse reinforcement learning as well as a new chapter on non-parametric Bayesian inference (for Dirichlet processes and Gaussian processes), variational Bayes and conformal prediction.

Preface to revised edition
Notation
1. Introduction
I. Stochastic Models and Bayesian Filtering: 2. Stochastic state space model
3. Optimal filtering
4. Algorithms for maximum likelihood parameter estimation
5. Multi-agent sensing: social learning and data incest
6. Nonparametric Bayesian inference
II. POMDPs: Models and Applications: 7. Fully observed Markov decision processes
8. Partially observed Markov decision processes
9. POMDPs in controlled sensing and sensor scheduling
III. POMDP Structural Results: 10. Structural results for Markov decision processes
11. Structural results for optimal filters
12. Monotonicity of value function for POMDPs
13. Structural results for stopping-time POMDPs
14. Stopping-Time POMDPs for quickest detection
15. Myopic policy bounds for POMDPs and sensitivity to model parameters
IV. Stochastic Gradient Algorithms and Reinforcement Learning: 16. Stochastic optimization and gradient estimation
17. Reinforcement learning
18. Stochastic gradient algorithms: convergence analysis
19. Discrete stochastic optimization
V. Inverse Reinforcement Learning: 20. Revealed preferences for inverse reinforcement learning
21. Bayesian inverse reinforcement learning
Appendix A. Short primer on stochastic stimulation
Appendix B. Continuous-time HMM filters
Appendix C. Discrete-time Martingales
Appendix D. Markov processes
Appendix E. Some limit theorems in statistics
Appendix F. Summary of POMDP algorithms
Bibliography
Index.

Subject Areas: Probability & statistics [PBT]