# Variational Optimization Based Reinforcement Learning for Infinite Dimensional Stochastic Systems

@inproceedings{Evans2019VariationalOB, title={Variational Optimization Based Reinforcement Learning for Infinite Dimensional Stochastic Systems}, author={Ethan N. Evans and Marcus A. Pereira and George I. Boutselis and Evangelos A. Theodorou}, booktitle={CoRL}, year={2019} }

Systems involving Partial Differential Equations (PDEs) have recently become more popular among the machine learning community. However prior methods usually treat infinite dimensional problems in finite dimensions with Reduced Order Models. This leads to committing to specific approximation schemes and subsequent derivation of control laws. Additionally, prior work does not consider spatio-temporal descriptions of noise that realistically represent the stochastic nature of physical systems. In…

## 5 Citations

### Spatio-Temporal Stochastic Optimization: Theory and Applications to Optimal Control and Co-Design

- Computer ScienceRobotics: Science and Systems
- 2020

This work derives an optimization algorithm on Hilbert spaces for nonlinear PDEs with an additive spatio-temporal description of white noise, and performs joint RL-type optimization of the feedback control law and the actuator design over episodes.

### Leveraging Stochasticity for Open Loop and Model Predictive Control of Spatio-Temporal Systems

- MathematicsEntropy
- 2021

Simulated experiments explore the application of a measure-theoretic description of spatio-temporal systems by describing them as evolutionary processes on Hilbert spaces, which yields a variational optimization framework for controlling stochastic fields.

### Stochastic spatio-temporal optimization for control and co-design of systems in robotics and applied physics

- Computer ScienceAuton. Robots
- 2022

A novel sampling-based stochastic optimization framework based entirely in Hilbert spaces suitable for the general class of semi-linear SPDEs which describes many systems in robotics and applied physics which is utilized for simultaneous policy optimization and actuator co-design optimization.

### Stochastic Control Problems with Unbounded Control Operators: solutions through generalized derivatives

- Mathematics
- 2021

A specific concept of partial derivative is introduced, designed for stochastic control problems in Hilbert spaces, and a method is developed to prove that the associated HJB equation has a solution with enough regularity to find optimal controls in feedback form.

### Leveraging Stochasticity for Open Loop and Model Predictive Control of Complex Fluid Systems

- Mathematics
- 2019

Stochastic Spatio-Temporal processes are prevalent across domains ranging from modeling of plasma to the turbulence in ﬂuids to the wave function of quantum systems. This letter studies a…

## References

SHOWING 1-10 OF 31 REFERENCES

### Stochastic optimal control in infinite dimension : dynamic programming and HJB equations

- Mathematics
- 2017

Providing an introduction to stochastic optimal control in infinite dimensions, this book gives a complete account of the theory of second-order HJB equations in infinite-dimensional Hilbert spaces,…

### Deep Dynamical Modeling and Control of Unsteady Fluid Flows

- EngineeringNeurIPS
- 2018

The proposed approach, grounded in Koopman theory, is shown to produce stable dynamical models that can predict the time evolution of the cylinder system over extended time horizons and is able to find a straightforward, interpretable control law for suppressing vortex shedding in the wake of the cylinders.

### Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

- Computer ScienceJ. Mach. Learn. Res.
- 2011

This work describes and analyze an apparatus for adaptively modifying the proximal function, which significantly simplifies setting a learning rate and results in regret guarantees that are provably as good as the best proximal functions that can be chosen in hindsight.

### Analysis of coordination in multi-agent systems through partial difference equations

- Mathematics, Computer ScienceIEEE Transactions on Automatic Control
- 2006

This note introduces the framework of partial difference equations over graphs for analyzing the behavior of multi-agent systems equipped with decentralized control schemes and shows that the resulting PdEs enjoy properties that are similar to those of well-known PDEs like the heat equation, thus allowing to exploit physical-based reasoning for conjecturing formation properties.

### An Introduction to Computational Stochastic PDEs

- Computer Science
- 2014

This book offers graduate students and researchers powerful tools for understanding uncertainty quantification for risk analysis and theory is developed in tandem with state-of-the art computational methods through worked examples, exercises, theorems and proofs.

### Model predictive control of nonlinear stochastic PDEs: Application to a sputtering process

- Mathematics2009 American Control Conference
- 2009

The proposed predictive controller can successfully drive the norm of the state variance of the stochastic KSE to a desired level in the presence of significant model parameter uncertainties.

### Nonlinear Stochastic Control and Information Theoretic Dualities: Connections, Interdependencies and Thermodynamic Interpretations

- Computer Science, MathematicsEntropy
- 2015

This paper extends connections between recent developments on the linearly-solvable stochastic optimal control framework with early work in control theory based on the fundamental dualities between free energy and relative entropy to nonlinear stochastically systems with non-affine controls by using the generalized version of the Feynman–Kac lemma.

### Gradient-Based Adaptive Stochastic Search for Non-Differentiable Optimization

- Computer ScienceIEEE Transactions on Automatic Control
- 2014

The algorithm combines the robustness feature of stochastic search from considering a population of candidate solutions with the relative fast convergence speed of classical gradient methods by exploiting local differentiable structures.

### Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control

- Engineering, Computer ScienceJournal of Fluid Mechanics
- 2019

It is shown that, in a two-dimensional simulation of the Kármán vortex street at moderate Reynolds number, the artificial neural network is able to learn an active control strategy from experimenting with the mass flow rates of two jets on the sides of a cylinder.

### Aggressive driving with model predictive path integral control

- Computer Science2016 IEEE International Conference on Robotics and Automation (ICRA)
- 2016

A model predictive control algorithm designed for optimizing non-linear systems subject to complex cost criteria using a stochastic optimal control framework using a fundamental relationship between the information theoretic notions of free energy and relative entropy is presented.