• Corpus ID: 7771457

Learning Physical Intuition of Block Towers by Example

  title={Learning Physical Intuition of Block Towers by Example},
  author={Adam Lerer and Sam Gross and Rob Fergus},
Wooden blocks are a common toy for infants, allowing them to develop motor skills and gain intuition about the physical behavior of the world. [] Key Method This data allows us to train large convolutional network models which can accurately predict the outcome, as well as estimating the block trajectories. The models are also able to generalize in two important ways: (i) to new physical scenarios, e.g. towers with an additional block and (ii) to images of real wooden blocks, where it obtains a performance…

Figures and Tables from this paper

IntPhys 2019: A Benchmark for Visual Intuitive Physics Understanding

An evaluation benchmark which diagnoses how much a given system understands about physics by testing whether it can tell apart well matched videos of possible versus impossible events constructed with a game engine is proposed.

Acquiring Target Stacking Skills by Goal-Parameterized Deep Reinforcement Learning

A synthetic block stacking environment with physics simulation is created in which the agent can learn a policy end-to-end through trial and error and a deep reinforcement learning framework that learns policies which are parametrized by a goal is proposed.

Visual Stability Prediction and Its Application to Manipulation

A learning-based approach based on simulated data that predicts stability of towers comprised of wooden blocks under different conditions and quantities related to the potential fall of the towers that in turn enables successful stacking.

Learning to Poke by Poking: Experiential Learning of Intuitive Physics

A novel approach based on deep neural networks is proposed for modeling the dynamics of robot's interactions directly from images, by jointly estimating forward and inverse models of dynamics.

Neural Allocentric Intuitive Physics Prediction from Real Videos

This work proposes a new perspective on the problem of learning intuitive physics that is inspired by the spatial memory representation of objects and spaces in human brains, in particular the co-existence of egocentric and allocentric spatial representations.

Unsupervised Intuitive Physics from Visual Observations

It is demonstrated for the first time that it is possible to learn reliable extrapolators of the object trajectories from raw videos alone, without any form of external supervision and with no more prior knowledge than the choice of a convolutional neural network architecture.

Unsupervised Intuitive Physics from Past Experiences

This work introduces the idea of summarizing past experiences in a very compact manner, in this case using dynamic images, and shows that this can be used to solve the problem well and efficiently.

Probing Physics Knowledge Using Tools from Developmental Psychology

This work introduces the VOE technique, and describes a set of probe datasets inspired by classic test stimuli from developmental psychology that are tested on a baseline deep learning system, as well as on a physics learning dataset recently posed by another research group.

Learning Manipulation under Physics Constraints with Visual Perception

A deep reinforcement learning framework that learns policies for stacking tasks which are parametrized by a target structure is proposed that is inspired by the intuitive physics in humans and compared with a more traditional model-based approach.

Generalizable Features From Unsupervised Learning

It is demonstrated that an unsupervised model, trained to predict future frames of a video sequence of stable and unstable block configurations, can yield features that support extrapolating stability prediction to blocks configurations outside the training set distribution.



To Fall Or Not To Fall: A Visual Approach to Physical Stability Prediction

A learning-based approach based on simulated data that predicts stability of towers comprised of wooden blocks under different conditions and quantities related to the potential fall of the towers by an end-to-end approach that directly predicts stability and related quantities from appearance.

Galileo: Perceiving Physical Object Properties by Integrating a Physics Engine with Deep Learning

This study points towards an account of human vision with generative physical knowledge at its core, and various recognition models as helpers leading to efficient inference.

Simulation as an engine of physical scene understanding

This work proposes a model based on an “intuitive physics engine,” a cognitive mechanism similar to computer engines that simulate rich physics in video games and graphics, but that uses approximate, probabilistic simulations to make robust and fast inferences in complex natural scenes where crucial information is unobserved.

Consistent physics underlying ballistic motion prediction

It is argued that people can apply correct physical principles consistently to simulate the world forward, and that the tasks that require implicit prediction (catching the ball and cutting the string) can be explained by veridical physical principles.

Continuous control with deep reinforcement learning

This work presents an actor-critic, model-free algorithm based on the deterministic policy gradient that can operate over continuous action spaces, and demonstrates that for many of the tasks the algorithm can learn policies end-to-end: directly from raw pixel inputs.

Human-level control through deep reinforcement learning

This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks.

Anticipating Human Activities Using Object Affordances for Reactive Robotic Response

This work represents each possible future using an anticipatory temporal conditional random field (ATCRF) that models the rich spatial-temporal relations through object affordances and represents each ATCRF as a particle and represents the distribution over the potential futures using a set of particles.

Mechanical reasoning by mental simulation

  • M. Hegarty
  • Psychology
    Trends in Cognitive Sciences
  • 2004

Learning to See by Moving

It is found that using the same number of training images, features learnt using egomotion as supervision compare favourably to features learnt with class-label as supervision on the tasks of scene recognition, object recognition, visual odometry and keypoint matching.

3D Reasoning from Blocks to Stability

This paper proposes a new algorithm to parse a single RGB-D image with 3D block units while jointly reasoning about the segments, volumes, supporting relationships, and object stability, and designs an energy function for representing the quality of the block representation based on these properties.