Skip to content. | Skip to navigation

Personal tools
Document Actions

Nicholas Roy, Geoffrey Gordon, and Sebastian Thrun (2005)

Finding approximate POMDP solutions through belief compression

Journal of Artificial Intelligence Research, 23:1-40.

Standard value function approaches to fi nding policies for Partially Observable Markov Decision Processes (POMDPs) are generally considered to be intractable for large models. The intractability of these algorithms is to a large extent a consequence of computing an exact, optimal policy over the entire belief space. However, in real-world POMDP problems, computing the optimal policy for the full belief space is often unnecessary for good control even for problems with complicated policy classes. The beliefs experienced by the controller often lie near a structured, low-dimensional subspace embedded in the high-dimensional belief space. Finding a good approximation to the optimal value function for only this subspace can be much easier than computing the full value function. We introduce a new method for solving large-scale POMDPs by reducing the dimensionality of the belief space. We use Exponential family Principal Components Analysis (Collins, Dasgupta, & Schapire, 2002) to represent sparse, high-dimensional belief spaces using small sets of learned features of the belief state. We then plan only in terms of the low-dimensional belief features. By planning in this low-dimensional space, we can find policies for POMDP models that are orders of magnitude larger than models that can be handled by conventional techniques. We demonstrate the use of this algorithm on a synthetic problem and on mobile robot navigation tasks.
Relevant for: WP4 abstraction. WP6 hierarchical architectures. Several researchers have looked at finding structure not only in feature spaces, but also in the space of value functions, or, as this paper, in the belief space. This may also be a valuable direction for thinking about abstraction.