Backprop as Functor: A compositional perspective on supervised learning

@article{Fong2019BackpropAF,
  title={Backprop as Functor: A compositional perspective on supervised learning},
  author={Brendan Fong and David I. Spivak and R{\'e}my Tuy{\'e}ras},
  journal={2019 34th Annual ACM/IEEE Symposium on Logic in Computer Science (LICS)},
  year={2019},
  pages={1-13}
}
A supervised learning algorithm searches over a set of functions $A\rightarrow B$ parametrised by a space $P$ to find the best approximation to some ideal function $f:A\rightarrow B$. It does this by taking examples $(a, f(a))\in A\times B$, and updating the parameter according to some rule. We define a category where these update rules may be composed, and show that gradient descent-with respect to a fixed step size and an error function satisfying a certain property-defines a monoidal functor… 

Figures from this paper

Learners' languages
In “Backprop as functor”, the authors show that the fundamental elements of deep learning—gradient descent and backpropagation—can be conceptualized as a strong monoidal functor Para(Euc) → Learn
Learning Functors using Gradient Descent
TLDR
A category-theoretic formalism around a neural network system called CycleGAN, a general approach to unpaired image-to-image translation that has been getting attention in the recent years, is built and it is shown that enforcing cycle-consistencies amounts to enforcing composition invariants in this category.
Compositional Deep Learning
TLDR
This thesis builds a category-theoretic formalism around a class of neural networks exemplified by CycleGAN, and uses the framework to conceive a novel neural network architecture whose goal is to learn the task of object insertion and object deletion in images with unpaired data.
Categorical Stochastic Processes and Likelihood
  • Dan Shiebler
  • Mathematics, Computer Science
    Compositionality
  • 2021
TLDR
A category-theoretic perspective on the relationship between probabilistic modeling and gradient based optimization is taken and a way to compose the likelihood functions of these models is defined.
A Probabilistic Generative Model of Free Categories
TLDR
It is shown how acyclic directed wiring diagrams can model specifications for morphisms, which the model can use to generate morphisms and the free category prior achieves competitive reconstruction performance on the Omniglot dataset.
Categorical Foundations of Gradient-Based Learning
TLDR
A categorical foundation of gradientbased machine learning algorithms in terms of lenses, parametrised maps, and reverse derivative categories is proposed, which encompasses a variety of gradient descent algorithms such as ADAM, AdaGrad, and Nesterov momentum.
Deep Learning, Grammar Transfer, and Transportation Theory
TLDR
G grammar transfer is used to demonstrate a paradigm that connects artificial intelligence and human intelligence and it is demonstrated that this learning model can learn a grammar intelligently in general, but fails to follow the optimal way of learning.
Dioptics: a Common Generalization of Open Games and Gradient-Based Learners
TLDR
It is shown that this category of gradient-based learners embeds naturally into the category of learners (with a choice of update rule and loss function), and that composing this embedding with reverse-mode automatic differentiation recovers the backpropagation functor L of [FST18].
Lenses and Learners
TLDR
This paper shows both that there is a faithful, identity-on-objects symmetric monoidal functor embedding a category of asymmetric lenses into the category of learners, and furthermore there is such a functorEmbedding the categories of learners into a categories of symmetric lenses.
Characterizing the invariances of learning algorithms using category theory
  • K. Harris
  • Mathematics, Computer Science
    ArXiv
  • 2019
TLDR
The framework for an invariant learning algorithm is a natural transformation between two functors from the product of these categories to the category of sets, representing training datasets and learned functions respectively.
...
...

References

SHOWING 1-10 OF 21 REFERENCES
Lenses and Learners
TLDR
This paper shows both that there is a faithful, identity-on-objects symmetric monoidal functor embedding a category of asymmetric lenses into the category of learners, and furthermore there is such a functorEmbedding the categories of learners into a categories of symmetric lenses.
Relational lenses: a language for updatable views
TLDR
The approach is to define a bi-directional query language, in which every expression can be read bot(from left to right) as a view definition and (from right to left) as an update policy.
A compositional framework for Markov processes
TLDR
It is proved that black boxing gives a symmetric monoidal dagger functor sending open detailed balanced Markov processes to open circuits made of linear resistors, and described how to “black box” an open Markov process.
The algebra of open and interconnected systems
TLDR
This thesis develops the theory of hypergraph categories and introduces the tools of decorated cospans and corelations, a more powerful version that permits construction of all hyper graph categories and hypergraph functors.
Geometric Deep Learning: Going beyond Euclidean data
TLDR
Deep neural networks are used for solving a broad range of problems from computer vision, natural-language processing, and audio analysis where the invariances of these structures are built into networks used to model them.
Algebras of Open Dynamical Systems on the Operad of Wiring Diagrams
TLDR
This paper uses the language of operads to study the algebraic nature of assembling complex dynamical systems from an interconnection of simpler ones, and defines two W-algebras, G and L, which associate semantic content to the structures in W.
From open learners to open games
TLDR
It is proved that there is a faithful symmetric monoidal functor from the former to the latter, which means that any supervised neural network can be seen as an open game in a canonical way.
Understanding deep image representations by inverting them
Image representations, from SIFT and Bag of Visual Words to Convolutional Neural Networks (CNNs), are a crucial component of almost any image understanding system. Nevertheless, our understanding of
Categories for the Working Mathematician
I. Categories, Functors and Natural Transformations.- 1. Axioms for Categories.- 2. Categories.- 3. Functors.- 4. Natural Transformations.- 5. Monics, Epis, and Zeros.- 6. Foundations.- 7. Large
Picturing Quantum Processes: A First Course in Quantum Theory and Diagrammatic Reasoning
TLDR
This entirely diagrammatic presentation of quantum theory represents the culmination of ten years of research, uniting classical techniques in linear algebra and Hilbert spaces with cutting-edge developments in quantum computation and foundations.
...
...