Skip to search formSkip to main contentSkip to account menu

What’s Hidden in a Randomly Weighted Neural Network?

It is empirically show that as randomly weighted neural networks with fixed weights grow wider and deeper, an ``untrained subnetwork" approaches a network with learned weights in accuracy.
View on IEEE (opens in a new tab)

Soft Threshold Weight Reparameterization for Learnable Sparsity

STR is a simple mechanism which learns effective sparsity budgets that contrast with popular heuristics that boosts the accuracy over existing results by up to 10% in the ultra sparse (99%) regime and can also be used to induce low-rank (structured sparsity) in RNNs.
View PDF on arXiv (opens in a new tab)

Supermasks in Superposition

The Supermasks in Superposition (SupSup) model, capable of sequentially learning thousands of tasks without catastrophic forgetting, is presented and it is found that a single gradient step is often sufficient to identify the correct mask, even among 2500 tasks.
View PDF on arXiv (opens in a new tab)

Improving Shape Deformation in Unsupervised Image-to-Image Translation

This work introduces a discriminator with dilated convolutions that is able to use information from across the entire image to train a more context-aware generator, coupled with a multi-scale perceptual loss that is better able to represent error in the underlying shape of objects.
View PDF on arXiv (opens in a new tab)

Forward Compatible Training for Large-Scale Embedding Retrieval Systems

This work proposes a new learning paradigm for representation learning: forward compatible training (FCT), and proposes learning side- information, an auxiliary feature for each sample whichitates future updates of the model.
View on IEEE (opens in a new tab)

LLC: Accurate, Multi-purpose Learnt Low-dimensional Binary Codes

This work proposes a novel method for Learning Low-dimensional binary Codes (LLC) for instances as well as classes that is super-efficient while still ensuring nearly optimal classification accuracy for ResNet50 on ImageNet-1K and captures intrinsically important features in the data by discovering an intuitive taxonomy over classes.
View PDF on arXiv (opens in a new tab)

Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent

The tendency for transformer parameters to grow in magnitude during training is studied, and its implications for the emergent representations within self attention layers are studied, suggesting saturation is a new characterization of an inductive bias implicit in GD of particular interest for NLP.
View on ACL (opens in a new tab)

Parameter Norm Growth During Training of Transformers

The tendency of transformer parameters to grow in magnitude during training is studied to find that in certain contexts, GD increases the parameter $L_2$ norm up to a threshold that itself increases with training-set accuracy, which means increasing training accuracy over time enables the norm to increase.

Matryoshka Representations for Adaptive Deployment

This work introduces Matryoshka Representation Learning (MRL), a minimally modifies existing representation learning pipelines and imposes no additional cost during inference and deployment that learns coarse-to-fine representations that are at least as accurate and rich as independently trained low-dimensional representations.
View via Publisher (opens in a new tab)

Forward Compatible Training for Representation Learning

This work proposes a new learning paradigm for representation learning: forward compatible training (FCT), and proposes learning sideinformation, an auxiliary feature for each sample which facilitates future updates of the model.
By clicking accept or continuing to use the site, you agree to the terms outlined in our Privacy Policy (opens in a new tab), Terms of Service (opens in a new tab), and Dataset License (opens in a new tab)