Continuous Rewards

A continuous reward function varies continuously with changes in the environment observations and actions. In general, continuous reward signals improve convergence during training and can lead to simpler network structures.

An example of a continuous reward is the quadratic regulator (QR) cost function, where the cumulative long-term reward can be expressed as:

Here, Q_τ, Q, R, and N are the weight matrices. Q_τ is the terminal weight matrix, applied only at the end of the episode. Also, s is the observation vector, a is the action vector, and τ is the terminal iteration of the episode. The (instantaneous) reward for this cost function is

This QR reward structure encourages an agent to drive s to zero with minimal action effort. A QR-based reward structure is a good reward to choose for regulation or stationary point problems, such as pendulum swing-up or regulating the position of the double integrator. For training examples that use a QR reward, see Train DQN Agent to Swing Up and Balance Pendulum and Compare DDPG Agent to LQR Controller.

Smooth continuous rewards, such as the QR regulator, are good for fine-tuning parameters and can provide policies similar to optimal controllers (LQR/MPC).

Discrete Rewards

A discrete reward function varies discontinuously with changes in the environment observations or actions. These types of reward signals can make convergence slower and can require more complex network structures. Discrete rewards are usually implemented as events that occur in the environment—for example, when an agent receives a positive reward if it exceeds some target value or a penalty when it violates some performance constraint.

While discrete rewards can slow down convergence, they can also guide the agent toward better reward regions in the state space of the environment. For example, a region-based reward, such as a fixed reward when the agent is near a target location, can emulate final-state constraints. Also, a region-based penalty can encourage an agent to avoid certain areas of the state space.

Mixed Rewards

In many cases, providing a mixed reward signal that has a combination of continuous and discrete reward components is beneficial. The discrete reward signal can be used to drive the system away from bad states, and the continuous reward signal can improve convergence by providing a smooth reward near target states. For example, in Train DDPG Agent to Control Sliding Robot, the reward function has three components: r₁, r₂, and r₃.

Here:

Reward Generation from Control Specifications

For applications where a working control system already exists, specifications such as cost functions or constraints might already be available. In these cases, you can use generateRewardFunction to automatically generate a reward function, coded in MATLAB^®, that can be used as a starting point for reward design. This function allows you to generate reward functions from:

In both cases, when constraints are violated, a negative reward is calculated using penalty functions such as exteriorPenalty (default), hyperbolicPenalty or barrierPenalty functions.

Starting from the generated reward function, you can tune the cost and penalty weights, use a different penalty function, and then use the resulting reward function within an environment to train an agent.

Observation Signals

When you create a custom environment, the signals you select as actions and observations depend on your application. For example, for control system applications, the integrals (and sometimes derivatives) of error signals are often useful observations. Also, for reference-tracking applications, having a time-varying reference signal as an observation is helpful.

When you define your observation signals, it is best practice to include all the available environment states in the observation vector.

Failure to do so can lead to situations in which different environment states result in the same observation. For such states, the agent policy (assuming it is a static function of the observation) returns the same action. Such a policy is typically unsuccessful, because it is normally the case that a successful policy needs to react to different environment states by returning different actions.

For example, an image observation of a swinging pendulum has position information but does not have enough information, by itself, to determine the pendulum velocity. In this case, a static policy that cannot sense the velocity would not be able to stabilize the pendulum. But if the velocity can be measured or estimated, adding it as an additional entry in the observation vector will provide a static policy with enough information to stabilize the pendulum.

When not all states are available as observation signals (for example because it would be unrealistic to measure them), a possible workaround is to use an estimator (as a part of the environment) that estimates the values of the unmeasured states, and makes such estimates available to the agent as observations. Alternatively, you can use recurrent networks such as an LSTM in your policy. Doing so results in a policy that has states, and that might therefore be able to use its state as an internal representation of the environment state. Such a policy can consequently return different actions (based on different values of its internal state) even when there is not enough information to reconstruct the correct environment state from the current observation.