# Group Equivariant Stand-Alone Self-Attention For Vision

@article{Romero2021GroupES, title={Group Equivariant Stand-Alone Self-Attention For Vision}, author={David W. Romero and Jean-Baptiste Cordonnier}, journal={ArXiv}, year={2021}, volume={abs/2010.00977} }

We provide a general self-attention formulation to impose group equivariance to arbitrary symmetry groups. This is achieved by defining positional encodings that are invariant to the action of the group considered. Since the group acts on the positional encoding directly, group equivariant self-attention networks (GSA-Nets) are steerable by nature. Our experiments on vision benchmarks demonstrate consistent improvements of GSA-Nets over non-equivariant self-attention networks.

## Figures and Tables from this paper

## 27 Citations

### LieTransformer: Equivariant self-attention for Lie Groups

- Computer ScienceICML
- 2021

The LieTransformer is proposed, an architecture composed of LieSelfAttention layers that are equivariant to arbitrary Lie groups and their discrete subgroups that are competitive to baseline methods on a wide range of tasks.

### Beyond permutation equivariance in graph networks

- Computer Science, MathematicsArXiv
- 2021

We introduce a novel architecture for graph networks which is equivariant to the Euclidean group in n-dimensions, and is additionally able to deal with affine transformations. Our model is designed…

### E(n) Equivariant Graph Neural Networks

- Computer ScienceICML
- 2021

This paper introduces a new model to learn graph neural networks equivariant to rotations, translations, reﬂections and permutations called E( n ) Equivariant Graph Neural Networks (EGNNs), which does not require computationally expensive higher-order representations in intermediate layers while it still achieves competitive or better performance.

### CKConv: Continuous Kernel Convolution For Sequential Data

- Computer ScienceArXiv
- 2021

Conventional neural architectures for sequential data present important limitations and can be solved by formulating convolutional kernels in CNNs as continuous functions, which allows us to model arbitrarily long sequences in a parallel manner, within a single operation, and without relying on any form of recurrence.

### Frame Averaging for Equivariant Shape Space Learning

- Computer Science2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- 2022

This paper presents a framework for incorporating equivariance in encoders and decoders by adapting the recent Frame Averaging framework and constructing autoencoders equivariant to piecewise Euclidean motions applied to different parts of the shape.

### Efficient Equivariant Network

- Computer ScienceNeurIPS
- 2021

This work proposes a general framework of previous equivariant models, which includes G-CNNs andEquivariant self-attention layers as special cases, and explicitly decomposes the feature aggregation operation into a kernel generator and an encoder, and decouple the spatial and extra geometric dimensions in the computation.

### FedMCSA: Personalized Federated Learning via Model Components Self-Attention

- Computer ScienceArXiv
- 2022

A new framework, federated model components self-attention (FedMCSA), is proposed to handle Non-IID data in FL, which employs model componentsSelf-att attention mechanism to granularly promote cooperation between di-erent clients and can signiﬁcantly improve the performance of FL.

### Unified Fourier-based Kernel and Nonlinearity Design for Equivariant Networks on Homogeneous Spaces

- MathematicsICML
- 2022

We introduce a uniﬁed framework for group equivariant networks on homogeneous spaces derived from a Fourier perspective. We consider tensor-valued feature ﬁelds, before and after a convolutional…

### Equivariant Mesh Attention Networks

- Computer ScienceArXiv
- 2022

This paper presents an attention-based architecture for mesh data that is provably equivariant to all transformations mentioned above, and shows that the choice of RelTan features is preferable to simple node positions.

### Equivariance Discovery by Learned Parameter-Sharing

- Computer ScienceAISTATS
- 2022

The partition distance is proposed to use the partition distance to empirically quantify the accuracy of the recovered equivariance, and the approach recovers known equivariances, such as permutations and shifts, on sum of numbers and spatially-invariant data.

## References

SHOWING 1-10 OF 56 REFERENCES

### Attentive Group Equivariant Convolutional Networks

- Computer ScienceICML
- 2020

Attentive group equivariant convolutions are presented, a generalization of the group convolution, in which attention is applied during the course of convolution to accentuate meaningful symmetry combinations and suppress non-plausible, misleading ones.

### Co-Attentive Equivariant Neural Networks: Focusing Equivariance On Transformations Co-Occurring In Data

- Computer ScienceICLR
- 2020

This work modify conventional equivariant feature mappings such that they are able to attend to the set of co-occurring transformations in data and generalize this notion to act on groups consisting of multiple symmetries.

### Equivariance Through Parameter-Sharing

- Computer Science, MathematicsICML
- 2017

This work shows that ϕW is equivariant with respect to G-action iff G explains the symmetries of the network parameters W, and proposes two parameter-sharing schemes to induce the desirable symmetry on W.

### General E(2)-Equivariant Steerable CNNs

- Computer Science, MathematicsNeurIPS
- 2019

The theory of Steerable CNNs yields constraints on the convolution kernels which depend on group representations describing the transformation laws of feature spaces, and it is shown that these constraints for arbitrary group representations can be reduced to constraints under irreducible representations.

### Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data

- MathematicsICML
- 2020

A general method to construct a convolutional layer that is equivariant to transformations from any specified Lie group with a surjective exponential map is proposed, enabling rapid prototyping and exact conservation of linear and angular momentum.

### Group Equivariant Capsule Networks

- Computer ScienceNeurIPS
- 2018

The group equivariant capsule networks are presented, a framework to introduce guaranteed equivariance and invariance properties to the capsule network idea and are able to combine the strengths of both approaches in one deep neural network architecture.

### Group Equivariant Convolutional Networks

- Computer ScienceICML
- 2016

Group equivariant Convolutional Neural Networks (G-CNNs), a natural generalization of convolutional neural networks that reduces sample complexity by exploiting symmetries and achieves state of the art results on CI- FAR10 and rotated MNIST.

### Scale-Equivariant Steerable Networks

- Computer ScienceICLR
- 2020

This work pays attention to scale changes, which regularly appear in various tasks due to the changing distances between the objects and the camera, and introduces the general theory for building scale-equivariant convolutional networks with steerable filters.

### Stand-Alone Self-Attention in Vision Models

- Computer ScienceNeurIPS
- 2019

The results establish that stand-alone self-attention is an important addition to the vision practitioner's toolbox and is especially impactful when used in later layers.

### On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups

- MathematicsICML
- 2018

It is proved that (given some natural constraints) convolutional structure is not just a sufficient, but also a necessary condition for equivariance to the action of a compact group.