Action-Conditional Video Prediction using Deep Networks in Atari Games

Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis, Satinder Singh

Introduction

Over the years, deep learning approaches (see for survey) have shown great success in many visual perception problems (e.g., ). However, modeling videos (building a generative model) is still a very challenging problem because it often involves high-dimensional natural-scene data with complex temporal dynamics. Thus, recent studies have mostly focused on modeling simple video data, such as bouncing balls or small patches, where the next frame is highly-predictable given the previous frames . In many applications, however, future frames depend not only on previous frames but also on control or action variables. For example, the first-person-view in a vehicle is affected by wheel-steering and acceleration. The camera observation of a robot is similarly dependent on its movement and changes of its camera angle. More generally, in vision-based reinforcement learning (RL) problems, learning to predict future images conditioned on actions amounts to learning a model of the dynamics of the agent-environment interaction, an essential component of model-based approaches to RL. In this paper, we focus on Atari games from the Arcade Learning Environment (ALE) as a source of challenging action-conditional video modeling problems. While not composed of natural scenes, frames in Atari games are high-dimensional, can involve tens of objects with one or more objects being controlled by the actions directly and many other objects being influenced indirectly, can involve entry and departure of objects, and can involve deep partial observability. To the best of our knowledge, this paper is the first to make and evaluate long-term predictions on high-dimensional images conditioned by control inputs.

This paper proposes, evaluates, and contrasts two spatio-temporal prediction architectures based on deep networks that incorporate action variables (See Figure 1). Our experimental results show that our architectures are able to generate realistic frames over 100-step action-conditional future frames without diverging in some Atari games. We show that the representations learned by our architectures 1) approximately capture natural similarity among actions, and 2) discover which objects are directly controlled by the agent’s actions and which are only indirectly influenced or not controlled. We evaluated the usefulness of our architectures for control in two ways: 1) by replacing emulator frames with predicted frames in a previously-learned model-free controller (DQN; DeepMind’s state of the art Deep-Q-Network for Atari Games ), and 2) by using the predicted frames to drive a more informed than random exploration strategy to improve a model-free controller (also DQN).

Related Work

The problem of video prediction has led to a variety of architectures in deep learning. A recurrent temporal restricted Boltzmann machine (RTRBM) was proposed to learn temporal correlations from sequential data by introducing recurrent connections in RBM. A structured RTRBM (sRTRBM) scaled up RTRBM by learning dependency structures between observations and hidden variables from data. More recently, Michalski et al. proposed a higher-order gated autoencoder that defines multiplicative interactions between consecutive frames and mapping units, and showed that temporal prediction problem can be viewed as learning and inferring higher-order interactions between consecutive images. Srivastava et al. applied a sequence-to-sequence learning framework to a video domain, and showed that long short-term memory (LSTM) networks are capable of generating video of bouncing handwritten digits. In contrast to these previous studies, this paper tackles problems where control variables affect temporal dynamics, and in addition scales up spatio-temporal prediction to larger-size images.

ALE: Combining Deep Learning and RL.

Atari 2600 games provide challenging environments for RL because of high-dimensional visual observations, partial observability, and delayed rewards. Approaches that combine deep learning and RL have made significant advances . Specifically, DQN combined Q-learning with a convolutional neural network (CNN) and achieved state-of-the-art performance on many Atari games. Guo et al. used the ALE-emulator for making action-conditional predictions with slow UCT , a Monte-Carlo tree search method, to generate training data for a fast-acting CNN, which outperformed DQN on several domains. Throughout this paper we will use DQN to refer to the architecture used in (a more recent work used a deeper CNN with more data to produce the currently best-performing Atari game players).

Action-Conditional Predictive Model for RL.

The idea of building a predictive model for vision-based RL problems was introduced by Schmidhuber and Huber . They proposed a neural network that predicts the attention region given the previous frame and an attention-guiding action. More recently, Lenz et al. proposed a recurrent neural network with multiplicative interactions that predicts the physical coordinate of a robot. Compared to this previous work, our work is evaluated on much higher-dimensional data with complex dependencies among observations. There have been a few attempts to learn from ALE data a transition-model that makes predictions of future frames. One line of work divides game images into patches and applies a Bayesian framework to predict patch-based observations. However, this approach assumes that neighboring patches are enough to predict the center patch, which is not true in Atari games because of many complex interactions. The evaluation in this prior work is 1-step prediction loss; in contrast, here we make and evaluate long-term predictions both for quality of pixels generated and for usefulness to control.

Proposed Architectures and Training Method

The goal of our architectures is to learn a function $f:\textbf{x}_{1:t},\textbf{a}_{t}\rightarrow\textbf{x}_{t+1}$ , where $\textbf{x}_{t}$ and $\textbf{a}_{t}$ are the frame and action variables at time $t$ , and $\textbf{x}_{1:t}$ are the frames from time $1$ to time $t$ . Figure 1 shows our two architectures that are each composed of encoding layers that extract spatio-temporal features from the input frames (§3.1), action-conditional transformation layers that transform the encoded features into a prediction of the next frame in high-level feature space by introducing action variables as additional input (§3.2) and finally decoding layers that map the predicted high-level features into pixels (§3.3). Our contributions are in the novel action-conditional deep convolutional architectures for high-dimensional, long-term prediction as well as in the novel use of the architectures in vision-based RL domains.

Recurrent encoding takes one frame as an input for each time-step and extracts spatio-temporal features using an RNN in which the temporal dynamics is modeled by the recurrent layer on top of the high-level feature vector extracted by convolution layers (Figure 1(b)). In this paper, LSTM without peephole connection is used for the recurrent layer as follows:

2 Multiplicative Action-Conditional Transformation

We use multiplicative interactions between the encoded feature vector and the control variables:

3 Convolutional Decoding

It has been recently shown that a CNN is capable of generating an image effectively using upsampling followed by convolution with stride of 1 . Similarly, we use the “inverse” operation of convolution, called deconvolution, which maps $1\times 1$ spatial region of the input to $d\times d$ using deconvolution kernels. The effect of $s\times s$ upsampling can be achieved without explicitly upsampling the feature map by using stride of $s$ . We found that this operation is more efficient than upsampling followed by convolution because of the smaller number of convolutions with larger stride.

In the proposed architecture, the transformed feature vector $\mathbf{h}^{dec}$ is decoded into pixels as follows:

where Reshape is a fully-connected layer where hidden units form a 3D feature map, and Deconv consists of multiple deconvolution layers, each of which is followed by a non-linearity except for the last deconvolution layer.

4 Curriculum Learning with Multi-Step Prediction

It is almost inevitable for a predictive model to make noisy predictions of high-dimensional images. When the model is trained on a 1-step prediction objective, small prediction errors can compound through time. To alleviate this effect, we use a multi-step prediction objective. More specifically, given the training data $D=\left\{\left(\left(\mathbf{x}_{1}^{(i)},\mathbf{a}_{1}^{(i)}\right),...,\left(\mathbf{x}_{T_{i}}^{(i)},\mathbf{a}_{T_{i}}^{(i)}\right)\right)\right\}_{i=1}^{N}$ , the model is trained to minimize the average squared error over $K$ -step predictions as follows:

where $\hat{\mathbf{x}}_{t+k}^{(i)}$ is a $k$ -step future prediction. Intuitively, the network is repeatedly unrolled through $K$ time steps by using its prediction as an input for the next time-step.

The model is trained in multiple phases based on increasing $K$ as suggested by Michalski et al. . In other words, the model is trained to predict short-term future frames and fine-tuned to predict longer-term future frames after the previous phase converges. We found that this curriculum learning approach is necessary to stabilize the training. A stochastic gradient descent with backpropagation through time (BPTT) is used to optimize the parameters of the network.

Experiments

In the experiments that follow, we have the following goals for our two architectures. 1) To evaluate the predicted frames in two ways: qualitatively evaluating the generated video, and quantitatively evaluating the pixel-based squared error, 2) To evaluate the usefulness of predicted frames for control in two ways: by replacing the emulator’s frames with predicted frames for use by DQN, and by using the predictions to improve exploration in DQN, and 3) To analyze the representations learned by our architectures. We begin by describing the details of the data, and model architecture, and baselines.

We used our replication of DQN to generate game-play video datasets using an $\epsilon$ -greedy policy with $\epsilon=0.3$ , i.e. DQN is forced to choose a random action with 30 $\%$ probability. For each game, the dataset consists of about $500,000$ training frames and $50,000$ test frames with actions chosen by DQN. Following DQN, actions are chosen once every $4$ frames which reduces the video from 60fps to 15fps. The number of actions available in games varies from $3$ to $18$ , and they are represented as one-hot vectors. We used full-resolution RGB images ( $210\times 160$ ) and preprocessed the images by subtracting mean pixel values and dividing each pixel value by $255$ .

Network Architecture.

Across all game domains, we use the same network architecture as follows. The encoding layers consist of $4$ convolution layers and one fully-connected layer with $2048$ hidden units. The convolution layers use $64$ $(8\times 8)$ , $128$ $(6\times 6)$ , $128$ $(6\times 6)$ , and $128$ $(4\times 4)$ filters with stride of 2. Every layer is followed by a rectified linear function . In the recurrent encoding network, an LSTM layer with $2048$ hidden units is added on top of the fully-connected layer. The number of factors in the transformation layer is $2048$ . The decoding layers consists of one fully-connected layer with $11264$ $(=128\times 11\times 8)$ hidden units followed by $4$ deconvolution layers. The deconvolution layers use $128$ $(4\times 4)$ , $128$ $(6\times 6)$ , $128$ $(6\times 6)$ , and $3$ $(8\times 8)$ filters with stride of 2. For the feedforward encoding network, the last $4$ frames are given as an input for each time-step. The recurrent encoding network takes one frame for each time-step, but it is unrolled through the last $11$ frames to initialize the LSTM hidden units before making a prediction. Our implementation is based on Caffe toolbox .

Details of Training.

We use the curriculum learning scheme above with three phases of increasing prediction step objectives of $1$ , $3$ and $5$ steps, and learning rates of $10^{-4}$ , $10^{-5}$ , and $10^{-5}$ , respectively. RMSProp is used with momentum of $0.9$ , (squared) gradient momentum of $0.95$ , and min squared gradient of $0.01$ . The batch size for each training phase is $32$ , $8$ , and $8$ for the feedforward encoding network and $4$ , $4$ , and $4$ for the recurrent encoding network, respectively. When the recurrent encoding network is trained on 1-step prediction objective, the network is unrolled through $20$ steps and predicts the last $10$ frames by taking ground-truth images as input. Gradients are clipped at $[-0.1,0.1]$ before non-linearity of each gate of LSTM as suggested by .

Two Baselines for Comparison.

The first baseline is a multi-layer perceptron (MLP) that takes the last frame as input and has 4 hidden layers with 400, 2048, 2048, and 400 units. The action input is concatenated to the second hidden layer. This baseline uses approximately the same number of parameters as the recurrent encoding model. The second baseline, no-action feedforward (or naFf), is the same as the feedforward encoding model (Figure 1(a)) except that the transformation layer consists of one fully-connected layer that does not get the action as input.

1 Evaluation of Predicted Frames

The prediction videos of our models and baselines are available in the supplementary material and at the following website: https://sites.google.com/a/umich.edu/junhyuk-oh/action-conditional-video-prediction. As seen in the videos, the proposed models make qualitatively reasonable predictions over $30$ – $500$ steps depending on the game. In all games, the MLP baseline quickly diverges, and the naFf baseline fails to predict the controlled object. An example of long-term predictions is illustrated in Figure 2. We observed that both of our models predict complex local translations well such as the movement of vehicles and the controlled object. They can predict interactions between objects such as collision of two objects. Since our architectures effectively extract hierarchical features using CNN, they are able to make a prediction that requires a global context. For example, in Figure 2, the model predicts the sudden change of the location of the controlled object (from the top to the bottom) at 257-step.

However, both of our models have difficulty in accurately predicting small objects, such as bullets in Space Invaders. The reason is that the squared error signal is small when the model fails to predict small objects during training. Another difficulty is in handling stochasticity. In Seaquest, e.g., new objects appear from the left side or right side randomly, and so are hard to predict. Although our models do generate new objects with reasonable shapes and movements (e.g., after appearing they move as in the true frames), the generated frames do not necessarily match the ground-truth.

Quantitative Evaluation: Squared Prediction Error.

Mean squared error over 100-step predictions is reported in Figure 3. Our predictive models outperform the two baselines for all domains. However, the gap between our predictive models and naFf baseline is not large except for Seaquest. This is due to the fact that the object controlled by the action occupies only a small part of the image.

Qualitative Analysis of Relative Strengths and Weaknesses of Feedforward and Recurrent Encoding.

We hypothesize that feedforward encoding can model more precise spatial transformations because its convolutional filters can learn temporal correlations directly from pixels in the concatenated frames. In contrast, convolutional filters in recurrent encoding can learn only spatial features from the one-frame input, and the temporal context has to be captured by the recurrent layer on top of the high-level CNN features without localized information. On the other hand, recurrent encoding is potentially better for modeling arbitrarily long-term dependencies, whereas feedforward encoding is not suitable for long-term dependencies because it requires more memory and parameters as more frames are concatenated into the input.

As evidence, in Figure 4(a) we show a case where feedforward encoding is better at predicting the precise movement of the controlled object, while recurrent encoding makes a 1-2 pixel translation error. This small error leads to entirely different predicted frames after a few steps. Since the feedforward and recurrent architectures are identical except for the encoding part, we conjecture that this result is due to the failure of precise spatio-temporal encoding in recurrent encoding. On the other hand, recurrent encoding is better at predicting when the enemies move in Space Invaders (Figure 4(b)). This is due to the fact that the enemies move after 9 steps, which is hard for feedforward encoding to predict because it takes only the last four frames as input. We observed similar results showing that feedforward encoding cannot handle long-term dependencies in other games.

2 Evaluating the Usefulness of Predictions for Control

To evaluate how useful the predictions are for playing the games, we implement an evaluation method that uses the predictive model to replace the game emulator. More specifically, a DQN controller that takes the last four frames is first pre-trained using real frames and then used to play the games based on $\epsilon=0.05$ -greedy policy where the input frames are generated by our predictive model instead of the game emulator. To evaluate how the depth of predictions influence the quality of control, we re-initialize the predictions using the true last frames after every n-steps of prediction for $1\leq n\leq 100$ . Note that the DQN controller never takes a true frame, just the outputs of our predictive models.

The results are shown in Figure 5. Unsurprisingly, replacing real frames with predicted frames reduces the score. However, in all the games using the model to repeatedly predict only a few time steps yields a score very close to that of using real frames. Our two architectures produce much better scores than the two baselines for deep predictions than would be suggested based on the much smaller differences in squared error. The likely cause of this is that our models are better able to predict the movement of the controlled object relative to the baselines even though such an ability may not always lead to better squared error. In three out of the five games the score remains much better than the score of random play even when using 100 steps of prediction.

Improving DQN via Informed Exploration.

To learn control in an RL domain, exploration of actions and states is necessary because without it the agent can get stuck in a bad sub-optimal policy. In DQN, the CNN-based agent was trained using an $\epsilon$ -greedy policy in which the agent chooses either a greedy action or a random action by flipping a coin with probability of $\epsilon$ . Such random exploration is a basic strategy that produces sufficient exploration, but can be slower than more informed exploration strategies. Thus, we propose an informed exploration strategy that follows the $\epsilon$ -greedy policy, but chooses exploratory actions that lead to a frame that has been visited least often (in the last $d$ time steps), rather than random actions. Implementing this strategy requires a predictive model because the next frame for each possible action has to be considered.

The method works as follows. The most recent $d$ frames are stored in a trajectory memory, denoted $D=\left\{\mathbf{x}^{(i)}\right\}_{i=1}^{d}$ . The predictive model is used to get the next frame ${\textbf{x}}^{(a)}$ for every action $a$ . We estimate the visit-frequency for every predicted frame by summing the similarity between the predicted frame and the most $d$ recent frames stored in the trajectory memory using a Gaussian kernel as follows:

where $\delta$ is a threshold, and $\sigma$ is a kernel bandwidth. The trajectory memory size is 200 for QBert and 20 for the other games, $\delta=0$ for Freeway and 50 for the others, and $\sigma=100$ for all games. For computational efficiency, we trained a new feedforward encoding network on $84\times 84$ gray-scaled images as they are used as input for DQN. The details of the network architecture are provided in the supplementary material. Table 1 summarizes the results. The informed exploration improves DQN’s performance using our predictive model in three of five games, with the most significant improvement in QBert. Figure 7 shows how the informed exploration strategy improves the initial experience of DQN.

3 Analysis of Learned Representations

In the factored multiplicative interactions, every action is linearly transformed to $f$ factors ( $\mathbf{W}^{a}\mathbf{a}$ in Equation 4). In Figure 7 we present the cosine similarity between every pair of action-factors after training in Seaquest. ‘N’ and ‘F’ corresponds to ‘no-operation’ and ‘fire’. Arrows correspond to movements with (black) or without (white) ‘fire’. There are positive correlations between actions that have the same movement directions (e.g., ‘up’ and ‘up+fire’), and negative correlations between actions that have opposing directions. These results are reasonable and discovered automatically in learning good predictions.

Distinguishing Controlled and Uncontrolled Objects is itself a hard and interesting problem. Bellemare et al. proposed a framework to learn contingent regions of an image affected by agent action, suggesting that contingency awareness is useful for model-free agents. We show that our architectures implicitly learn contingent regions as they learn to predict the entire image.

Conclusion

This paper introduced two different novel deep architectures that predict future frames that are dependent on actions and showed qualitatively and quantitatively that they are able to predict visually-realistic and useful-for-control frames over 100-step futures on several Atari game domains. To our knowledge, this is the first paper to show good deep predictions in Atari games. Since our architectures were domain independent we expect that they will generalize to many vision-based RL problems. In future work we will learn models that predict future reward in addition to predicting future frames and evaluate the performance of our architectures in model-based RL.

This work was supported by NSF grant IIS-1526059, Bosch Research, and ONR grant N00014-13-1-0762. Any opinions, findings, conclusions, or recommendations expressed here are those of the authors and do not necessarily reflect the views of the sponsors.

References

Appendix A Network Architectures and Training Details

The network architectures of the proposed models and the baselines are illustrated in Figure 9.

The weight of LSTM is initialized from a uniform distribution of $[-0.08,0.08]$ . The weight of the fully-connected layer from the encoded feature to the factored layer and from the action to the factored layer are initialized from a uniform distribution of $ $and$ [-0.1,0.1]$ respectively.

The total number of iterations is $1.5\times 10^{6}$ , $10^{6}$ , and $10^{6}$ for each training phase (1-step, 3-step, and 5-step). The learning rate is multiplied by $0.9$ after every $10^{5}$ iterations.

Appendix B Informed Exploration

The entire DQN algorithm with informed exploration is described in Algorithm 1.

A feedforward encoding network (illustrated in Figure 10) trained on down-sampled and gray-scaled images is used for computational efficiency. We trained the model on 1-step prediction objective with learning rate of $10^{-4}$ and batch size of $32$ . The pixel values are subtracted by mean pixel values and divided by 128. RMSProp is used with momentum of $0.9$ , (squared) gradient momentum of $0.95$ , and min squared gradient of $0.01$ .

Comparison to Random Exploration.

Figure 11 visualizes the difference between random exploration and informed exploration in two games. In Freeway, where the agent gets rewards by reaching the top lane, the agent moves only around the bottom area in the random exploration, which results in $4.6\times 10^{5}$ steps to get the first reward. On the other hand, the agent moves around all locations in the informed exploration and receives the first reward in $86$ steps. The similar result is found in Ms Pacman.

Application to Deep Q-learning.

The results of the informed exploration using the game emulator and our predictive model are reported in Figure 12 and Table 2. Our DQN replication follows , which uses a smaller CNN than .