Learning to Simulate Dynamic Environments With GameGAN

@article{Kim2020LearningTS,
  title={Learning to Simulate Dynamic Environments With GameGAN},
  author={Seung Wook Kim and Yuhao Zhou and Jonah Philion and Antonio Torralba and Sanja Fidler},
  journal={2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2020},
  pages={1228-1237}
}
Simulation is a crucial component of any robotic system. In order to simulate correctly, we need to write complex rules of the environment: how dynamic agents behave, and how the actions of each of the agents affect the behavior of others. In this paper, we aim to learn a simulator by simply watching an agent interact with an environment. We focus on graphics games as a proxy of the real environment. We introduce GameGAN, a generative model that learns to visually imitate a desired game by… Expand
DriveGAN: Towards a Controllable High-Quality Neural Simulation
TLDR
This work introduces a novel high-quality neural simulator referred to as DriveGAN that achieves controllability by disentangling different components without supervision, andTrain DriveGAN on multiple datasets, including 160 hours of real-world driving data. Expand
Dropout's Dream Land: Generalization from Learned Simulators to Reality
TLDR
The experimental results show that Dropout’s Dream Land is an effective technique to bridge the reality gap between dream environments and reality and improvements are gained by drawing on inspiration from Domain Randomization. Expand
Automatic generation of graphical game assets using GAN
This paper presents an application of the Generative Adversarial Networks (GAN) approach to automatically generate realistic-looking fantasy and science fiction game icons. In this study, we exploredExpand
FitVid: Overfitting in Pixel-Level Video Prediction
TLDR
This paper argues that the inefficient use of parameters in the current video models is the main reason for underfitting, and introduces a new architecture, named FitVid, which is capable of severe overfitting on the common benchmarks while having similar parameter count as the current state-of-the-art models. Expand
Contrastive Learning of Generalized Game Representations
TLDR
The results in a large dataset of sports video games containing 100k images across 175 games and 10 game genres suggest that contrastive learning is better suited for learning generalized game representations compared to conventional supervised learning. Expand
Detecting and Adapting to Novelty in Games
TLDR
A model-based reinforcement learning approach where game state and rules are represented as knowledge graphs allows novelty to be detected as changes in the knowledge graph, assists with the training of deep reinforcement learners, and enables imagination-based re-training. Expand
MC-GAN: A Reimplementation of GameGAN With a Gaming Perspective
TLDR
Multi-Class GamgeGan (MC-GAN), a starting point for the next generation of game engines based on GameGAN, is proposed, which is capable of classifying each desired game element and changing its class to change the element’s nature. Expand
NavRep: Unsupervised Representations for Reinforcement Learning of Robot Navigation in Dynamic Human Environments
TLDR
This work designs ways in which unsupervised learning can be used to assist reinforcement learning for robot navigation, and shows that unsuper supervised learning methods are competitive with end-to-end methods. Expand
Tile Embedding: A General Representation for Level Generation
  • 2021
Entity Embedding as Game Representation
TLDR
This paper presents an autoencoder for deriving what they call "entity embeddings", a consistent way to represent different dynamic entities across multiple games in the same representation. Expand
...
1
2
3
...

References

SHOWING 1-10 OF 43 REFERENCES
Game Engine Learning from Video
TLDR
This work presents a novel approach to learn a forward simulation model via simple search over pixel input and demonstrates the significant improvement in predicting future states compared with a baseline CNN and applies the learned model to train a game playing agent. Expand
ViZDoom: A Doom-based AI research platform for visual reinforcement learning
TLDR
A novel test-bed platform for reinforcement learning research from raw visual information which employs the first-person perspective in a semi-realistic 3D world and confirms the utility of ViZDoom as an AI research platform and implies that visual reinforcement learning in 3D realistic first- person perspective environments is feasible. Expand
Model-Based Reinforcement Learning for Atari
TLDR
Simulated Policy Learning (SimPLe), a complete model-based deep RL algorithm based on video prediction models, is described and a comparison of several model architectures is presented, including a novel architecture that yields the best results in the authors' setting. Expand
Recurrent Environment Simulators
TLDR
This work addresses the issue of computationally inefficiency with a model that does not need to generate a high-dimensional image at each time-step and can be used to improve exploration and is adaptable to many diverse environments. Expand
Stochastic Adversarial Video Prediction
TLDR
This work shows that latent variational variable models that explicitly model underlying stochasticity and adversarially-trained models that aim to produce naturalistic images are in fact complementary and combines the two to produce predictions that look more realistic to human raters and better cover the range of possible futures. Expand
Learning by Cheating
TLDR
This work shows that this challenging learning problem can be simplified by decomposing it into two stages and uses the presented approach to train a vision-based autonomous driving system that substantially outperforms the state of the art on the CARLA benchmark and the recent NoCrash benchmark. Expand
Action-Conditional Video Prediction using Deep Networks in Atari Games
TLDR
This paper is the first to make and evaluate long-term predictions on high-dimensional video conditioned by control inputs and proposes and evaluates two deep neural network architectures that consist of encoding, action-conditional transformation, and decoding layers based on convolutional neural networks and recurrent neural networks. Expand
Generative Adversarial Nets
We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and aExpand
Combining neural networks and tree search for task and motion planning in challenging environments
TLDR
This work integrates Monte Carlo Tree Search with hierarchical neural net policies trained on expressive LTL specifications to find deep neural networks representing both low-level control policies and task-level “option policies” that achieve high-level goals. Expand
Asynchronous Methods for Deep Reinforcement Learning
TLDR
A conceptually simple and lightweight framework for deep reinforcement learning that uses asynchronous gradient descent for optimization of deep neural network controllers and shows that asynchronous actor-critic succeeds on a wide variety of continuous motor control problems as well as on a new task of navigating random 3D mazes using a visual input. Expand
...
1
2
3
4
5
...