• Corpus ID: 235293989

SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training

@article{Somepalli2021SAINTIN,
  title={SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training},
  author={Gowthami Somepalli and Micah Goldblum and Avi Schwarzschild and C. Bayan Bruss and Tom Goldstein},
  journal={ArXiv},
  year={2021},
  volume={abs/2106.01342}
}
Tabular data underpins numerous high-impact applications of machine learning 1 from fraud detection to genomics and healthcare. Classical approaches to solving 2 tabular problems, such as gradient boosting and random forests, are widely used 3 by practitioners. However, recent deep learning methods have achieved a degree 4 of performance competitive with popular techniques. We devise a hybrid deep 5 learning approach to solving tabular data problems. Our method, SAINT, performs 6 attention over… 
[PDF] Semantic Reader
57 Citations
Highly Influential Citations
14
Background Citations
37
Methods Citations
27

57 Citations

Transfer Learning with Deep Tabular Models

This work finds that transfer learning with deep tabular models provides a definitive advantage over gradient boosted decision tree methods and proposes a pseudo-feature method for cases where the upstream and downstream feature sets differ.

Hopular: Modern Hopfield Networks for Tabular Data

Hopular is a novel Deep Learning architecture for mediumand smallsized datasets, where each layer is equipped with continuous modern Hopfield networks, and surpasses Gradient Boosting, Random Forests, SVMs, and in particular several Deep Learning methods on tabular data.

Graph Neural Network contextual embedding for Deep Learning on Tabular Data

A novel DL model that uses Graph Neural Network (GNN), more specifically Interaction Network (IN), for contextual embedding is introduced, achieving also competitive results when compared to boosted-tree solutions.

Self Supervised Pre-training for Large Scale Tabular Data

This paper proposes a self supervised pre-training strategy that utilizes Manifold Mixup to produce data augmentations for tabular data and performs reconstruction on these augmentations using noise contrastive estimation and mean absolute error losses, both of which are particularly suitable for large scaletabular data.

Sparse tree-based initialization for neural networks

This work proposes a new sparse initialization technique for (potentially deep) multilayer per-ceptrons (MLP): a tree-based procedure is trained to detect feature interactions and the resulting information is used to initialize the network, which is subsequently trained via standard stochastic gradient strategies.

Perturbation of Deep Autoencoder Weights for Model Compression and Classification of Tabular Data

It is revealed that a pretrained deep autoencoder with weight perturbation can outperform traditional machine learning in tabular data classification, whereas baseline fully-connected DNNs yield the worst classification accuracy.

Deep Neural Networks and Tabular Data: A Survey

An in-depth overview of state-of-the-art deep learning methods for tabular data, categorizing these methods into three groups: data transformations, specialized architectures, and regularization models, and indicates that algorithms based on gradient-boosted tree ensembles still mostly outperform deep learning models on supervised learning tasks, suggesting that the research progress on competitive deep learning model development is stagnating.

Tabular Data: Deep Learning is Not All You Need

Tabular Deep Learning when d ≫ n by Using an Auxiliary Knowledge Graph

P LATO is proposed, a machine learning model for tabular data with d ≫ n and an auxiliary KG with input features as nodes that exceeds or matches the prior state-of-the-art, achieving performance improvements of up to 10.19%.

Revisiting Pretraining Objectives for Tabular Deep Learning

This work aims to identify the best practices to pretrain tabular DL models that can be universally applied to different datasets and architectures and shows that using the object target labels during the pretraining stage is beneficial for the downstream performance.
...

References

SHOWING 1-10 OF 55 REFERENCES

Regularization Learning Networks: Deep Learning for Tabular Datasets

RLNs are introduced, which could efficiently learn a single network in datasets that comprise both tabular and unstructured data, such as in the setting of medical imaging accompanied by electronic health records.

Tabular Transformers for Modeling Multivariate Time Series

  • Inkit PadhiYair Schiff Erik Altman
  • Computer Science
    ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
  • 2021
Two architectures for tabular time series are proposed: one for learning representations that can be pre-trained end-to-end and used in downstream tasks, and one that is akin to GPT and can be used for generation of realistic synthetic tabular sequences.

VIME: Extending the Success of Self- and Semi-supervised Learning to Tabular Domain

This paper creates a novel pretext task of estimating mask vectors from corrupted tabular data in addition to the reconstruction pretext task for self-supervised learning, and introduces a noveltabular data augmentation method for selfand semi- supervised learning frameworks.

Net-DNF: Effective Deep Modeling of Tabular Data

This work presents Net-DNF, a novel generic architecture whose inductive bias elicits models whose structure corresponds to logical Boolean formulas in disjunctive normal form (DNF) over affine soft-threshold decision terms, which opens the door to practical end-to-end handling of tabular data using neural networks.

TABBIE: Pretrained Representations of Tabular Data

A simple pretraining objective (corrupt cell detection) is devised that learns exclusively from tabular data and reaches the state-of-the-art on a suite of table-based prediction tasks and requires far less compute to train.

TabNet: Attentive Interpretable Tabular Learning

It is demonstrated that TabNet outperforms other neural network and decision tree variants on a wide range of non-performance-saturated tabular datasets and yields interpretable feature attributions plus insights into the global model behavior.

LightGBM: A Highly Efficient Gradient Boosting Decision Tree

It is proved that, since the data instances with larger gradients play a more important role in the computation of information gain, GOSS can obtain quite accurate estimation of the information gain with a much smaller data size, and is called LightGBM.

AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data

We introduce AutoGluon-Tabular, an open-source AutoML framework that requires only a single line of Python to train highly accurate machine learning models on an unprocessed tabular dataset such as a…

CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features

Patches are cut and pasted among training images where the ground truth labels are also mixed proportionally to the area of the patches, and CutMix consistently outperforms state-of-the-art augmentation strategies on CIFAR and ImageNet classification tasks, as well as on ImageNet weakly-supervised localization task.

Deep Residual Learning for Image Recognition

This work presents a residual learning framework to ease the training of networks that are substantially deeper than those used previously, and provides comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth.
...