4.2 Deliberative control

Deliberative control overview

Deliberative control is a decision-making approach where a robot builds an internal model of its environment, reasons about that model, and plans a sequence of actions to achieve its goals. Where reactive control maps sensor inputs directly to motor outputs (fast but short-sighted), deliberative control asks: given what I know about the world, what's the best sequence of actions to reach my goal?

This makes deliberative systems more flexible and capable of handling complex tasks, but at a cost: they need more computation and more time to produce decisions.

Comparison to reactive control

• Reactive control uses direct sensor-to-action mappings with no internal world model. It's fast and computationally cheap, but it can only respond to what's happening right now.
• Deliberative control maintains an explicit representation of the world, enabling the robot to reason about future states, weigh consequences, and pursue long-term objectives.
• The tradeoff is straightforward: reactive systems are quicker to respond but can't plan ahead; deliberative systems can plan ahead but take longer to compute a response.

Advantages of deliberative control

• Robots can consider long-term goals and the consequences of their actions, producing more purposeful behavior
• Plans can be updated as the robot gains new information, making it possible to operate in complex, changing environments
• Multiple robots can share world models and coordinate plans, enabling collaboration
• Deliberative frameworks support higher-level cognitive functions like reasoning, learning, and problem-solving

World representation

Before a deliberative robot can plan, it needs an internal model of its environment. This world representation captures the relevant features of the surroundings (geometry, objects, spatial relationships) and serves as the foundation for all reasoning and planning.

Modeling the environment

Different model types capture different kinds of information:

• Geometric models represent the physical layout. An occupancy grid divides the environment into cells and marks each as free or occupied. 3D point clouds extend this to three dimensions.
• Semantic models capture higher-level meaning: object categories, properties (e.g., "this is a door; it can be opened"), and relationships between objects.
• Topological models represent the environment as a graph of connected regions or landmarks. Instead of encoding every spatial detail, they focus on how places connect, which makes path planning more efficient.

Knowledge representation techniques

• Ontologies provide a formal vocabulary of concepts, relations, and rules for describing the environment (e.g., "a cup is a container that can hold liquid").
• Probabilistic graphical models (Bayesian networks, Markov random fields) capture uncertainty and dependencies among variables, which is critical when sensor data is noisy.
• Logic-based representations (first-order logic, description logics) support symbolic reasoning and inference, letting the robot derive new facts from known ones.
• Spatial databases store and query spatial information like object locations, distances, and topological relationships.

Planning algorithms

Planning algorithms take the robot's world model and goals and produce a sequence of actions (or a policy) to achieve those goals. Different planning approaches suit different problem structures.

Search-based planning

Search-based planners treat planning as finding a path through a state space from an initial state to a goal state.

1. Define the state space (all possible configurations of the robot and environment).
2. Apply a search algorithm to explore states and find a path to the goal.
3. Return the sequence of actions along that path.

Common algorithms:

• A* and Dijkstra's algorithm find optimal (shortest/cheapest) paths. A* uses a heuristic to focus the search toward the goal, making it faster than Dijkstra's in most cases.
• Weighted A* trades some optimality for speed by inflating the heuristic, exploring fewer states.
• Game-theoretic approaches like Minimax and Monte Carlo Tree Search handle adversarial or uncertain scenarios where the environment may "push back."

Sampling-based planning

When the state space is too large to search exhaustively (common in high-dimensional problems like robot arm motion), sampling-based methods build an approximate map of feasible configurations.

• Probabilistic Roadmaps (PRM): Randomly sample collision-free configurations, then connect nearby samples with local paths to build a graph. Query the graph to find a route from start to goal.
• Rapidly-exploring Random Trees (RRT): Grow a tree from the start state by repeatedly sampling a random point and extending the nearest tree node toward it. The tree rapidly spreads through free space and eventually reaches the goal region.

These methods don't guarantee optimal solutions, but they work well in complex, high-dimensional environments where grid-based search is infeasible.

Optimization-based planning

These methods frame planning as minimizing a cost function (or maximizing a reward).

• Trajectory optimization (CHOMP, TrajOpt) produces smooth, collision-free trajectories by iteratively refining an initial path to reduce cost.
• Optimal control methods like LQR (Linear Quadratic Regulator) and MPC (Model Predictive Control) compute control inputs that minimize cost over a time horizon. MPC is especially useful because it replans at each time step using updated state information.
• Reinforcement learning (Q-learning, Policy Gradients) learns policies through trial and error. The robot interacts with the environment, receives rewards, and gradually improves its behavior.

Plan execution

Generating a plan is only half the job. The robot must carry it out in the real world, where things rarely go exactly as expected.

Plan monitoring

The robot continuously compares its expected state with its actual state during execution. Techniques like state estimation (e.g., Kalman filters) and sensor fusion (combining data from multiple sensors) help the robot maintain an accurate picture of what's happening. When the actual state diverges from the expected state, the system flags a problem.

Plan repair

For minor deviations, the robot can patch the existing plan rather than starting over. Plan repair techniques include:

• Local re-optimization of a small portion of the trajectory
• Relaxing constraints that are no longer satisfiable
• Applying predefined repair strategies for common failure modes

This keeps the overall plan structure intact and avoids the cost of full replanning.

Replanning

When the current plan becomes infeasible or significantly suboptimal (e.g., a major obstacle appears, or the goal changes), the robot generates a completely new plan. Replanning uses the same algorithms as initial planning but incorporates updated world knowledge and constraints. The trigger is typically a large deviation, a critical failure, or new information that invalidates the original plan's assumptions.

Reasoning under uncertainty

Real-world environments are noisy and unpredictable. Sensor readings are imperfect, action outcomes aren't guaranteed, and the robot often can't observe everything. Deliberative control needs principled ways to handle this uncertainty.

Probabilistic reasoning

Instead of treating the world as fully known, the robot represents uncertain quantities as probability distributions and updates those distributions as new observations arrive.

• Bayesian inference is the core mechanism: given a prior belief and new evidence, compute an updated (posterior) belief.
• Kalman filters do this efficiently for linear systems with Gaussian noise. Particle filters handle nonlinear, non-Gaussian problems by representing the belief as a set of weighted samples.
• Probabilistic graphical models (Bayesian networks, Markov random fields) encode dependencies among variables, enabling the robot to reason about how uncertain quantities relate to each other.

Markov decision processes

An MDP models sequential decision-making under uncertainty as a tuple $(S, A, T, R)$:

• $S$: the set of possible states
• $A$: the set of possible actions
• $T(s, a, s')$: the probability of transitioning from state $s$ to state $s'$ after taking action $a$
• $R(s, a)$: the reward received for taking action $a$ in state $s$

The goal is to find an optimal policy $\pi^*$ that maximizes expected cumulative reward. Standard solution methods include value iteration and policy iteration (dynamic programming) or reinforcement learning methods like Q-learning and SARSA.

Partially observable Markov decision processes

A POMDP extends the MDP to situations where the robot can't directly observe the true state. It adds:

• An observation space $O$ (the set of possible observations)
• An observation function $Z(o \mid s, a)$ giving the probability of observation $o$ given state $s$ and action $a$

Because the true state is hidden, the robot maintains a belief state: a probability distribution over all possible states. Decisions are made based on this belief rather than a known state. POMDPs are harder to solve than MDPs, but techniques like point-based value iteration and Monte Carlo tree search make them tractable for moderately sized problems.

Applications of deliberative control

Autonomous navigation

Deliberative control lets robots plan paths through complex environments while accounting for obstacles, uncertainty, and multiple objectives. This applies to autonomous driving (planning routes through traffic), aerial navigation (UAV mission planning), and planetary exploration (Mars rovers planning traversals over uncertain terrain). Algorithms like PRM, RRT, and MPC are commonly used here.

Manipulation tasks

For tasks like grasping, placing, and assembling objects, the robot must reason about object poses, grasp configurations, and task constraints. Task and motion planning (TAMP) combines symbolic task planning ("pick up block A, then place it on block B") with geometric motion planning to produce executable action sequences.

Multi-robot coordination

When multiple robots work together, deliberative control helps them share information, negotiate plans, and allocate tasks. Techniques include distributed planning algorithms, consensus protocols (for agreeing on shared variables), and game-theoretic methods for handling competing objectives.

Challenges in deliberative control

Computational complexity

Deliberative planning often involves searching or optimizing over large state and action spaces. The curse of dimensionality means that as the number of state variables grows, the computation required grows exponentially. Practical mitigations include approximate algorithms, hierarchical problem decomposition (breaking a big problem into smaller subproblems), and domain-specific heuristics.

Real-time performance

A robot operating in the physical world can't pause and think indefinitely. Planning algorithms must produce usable solutions within tight time budgets. Anytime algorithms help here: they quickly find an initial solution and then keep improving it as time allows. Incremental planning (reusing parts of previous plans) and parallel processing also improve responsiveness.

Handling dynamic environments

The world changes while the robot is executing its plan. New obstacles appear, goals shift, and sensor data reveals surprises. The deliberative loop must incorporate continuous sensing and state estimation to keep the world model current, and use plan monitoring, repair, and replanning to adapt.

Integration with other control paradigms

Pure deliberative control is rarely used alone. In practice, it's combined with other approaches to balance planning depth with responsiveness.

Hybrid deliberative-reactive control

The deliberative layer generates high-level plans and goals. The reactive layer handles low-level execution and immediate responses (like stopping when an unexpected obstacle appears). This combination gives the robot both foresight and reflexes.

Hierarchical control architectures

The control system is organized into layers of increasing abstraction:

• Lower layers (reactive) handle fast sensorimotor loops
• Middle layers manage behaviors and sequencing
• Upper layers (deliberative) handle long-term planning and reasoning

Each layer operates at a different time scale and can be developed independently, making the system modular and scalable.

Combining deliberative and learning-based approaches

Machine learning (deep learning, reinforcement learning) can be integrated into the deliberative framework. Learning-based methods extract features from raw data, learn environment models, and optimize policies. The deliberative structure provides prior knowledge and safety constraints to guide the learning process. This combination enables data-efficient learning, knowledge transfer across tasks, and adaptation to new environments.