• Corpus ID: 243832591

Well-tuned Simple Nets Excel on Tabular Datasets

@inproceedings{Kadra2021WelltunedSN,
  title={Well-tuned Simple Nets Excel on Tabular Datasets},
  author={Arlind Kadra and Marius Thomas Lindauer and Frank Hutter and Josif Grabocka},
  booktitle={NeurIPS},
  year={2021}
}
Tabular datasets are the last “unconquered castle” for deep learning, with traditional ML methods like Gradient-Boosted Decision Trees still performing strongly even against recent specialized neural architectures. In this paper, we hypothesize that the key to boosting the performance of neural networks lies in rethinking the joint and simultaneous application of a large set of modern regularization techniques. As a result, we propose regularizing plain Multilayer Perceptron (MLP) networks by… 

Why do tree-based models still outperform deep learning on tabular data?

TLDR
Results show that tree-based models remain state-of-the-art on medium-sized data even without accounting for their superior speed, and leads to a series of challenges which should guide researchers aiming to build tabular-species NNs.

Deep Neural Networks and Tabular Data: A Survey

TLDR
An in-depth overview of state-of-the-art deep learning methods for tabular data, categorizing these methods into three groups: data transformations, specialized architectures, and regularization models, and indicates that algorithms based on gradient-boosted tree ensembles still mostly outperform deep learning models on supervised learning tasks, suggesting that the research progress on competitive deep learning model development is stagnating.

On Embeddings for Numerical Features in Tabular Deep Learning

TLDR
It is argued that embeddings for numerical features are an underexplored degree of freedom in tabular DL, which allows constructing more powerful DL models and competing with GBDT on some traditionally GBDT-friendly benchmarks.

TabNAS: Rejection Sampling for Neural Architecture Search on Tabular Datasets

TLDR
TabNAS is developed, a new and moreective approach to handle resource constraints in tabular NAS using an RL controller motivated by the idea of rejection sampling, which demonstrates the superiority of TabNAS over previous reward-shaping methods: it develops better models that obey the constraints.

A Framework and Benchmark for Deep Batch Active Learning for Regression

TLDR
An open-source benchmark with 15 large tabular data sets is introduced, which is used to compare different BMDAL methods and shows that a combination of the novel components yields new state-of-the-art results in terms of RMSE and is computationally efficient.

The GatedTabTransformer. An enhanced deep learning architecture for tabular modeling

TLDR
Inspired by gated MLP, linear projections are implemented in the MLP block and multiple activation functions are tested and the importance of specific hyper parameters during training is evaluated.

Resource-Constrained Neural Architecture Search on Tabular Datasets

TLDR
It is argued that search spaces for tabular NAS pose considerable challenges for these existing reward-shaping methods, and a new reinforcement learning (RL) controller is proposed to address these challenges.

GATE: Gated Additive Tree Ensemble for Tabular Classification and Regression

TLDR
GATE uses a gating mechanism, inspired from GRU, as a feature representation learning unit with an in-built feature selection mechanism to combine an ensemble of differentiable, non-linear decision trees, re-weighted with simple self-attention to predict the desired output.

TransTab: Learning Transferable Tabular Transformers Across Tables

TLDR
The goal of TransTab is to convert each sample to a generalizable embedding vector, and then apply stacked transformers for feature encoding, and one methodology insight is combining column description and table cells as the raw input to a gated transformer model.

Stochastic Perturbations of Tabular Features for Non-Deterministic Inference with Automunge

TLDR
The Automunge library for tabular preprocessing is offered as a resource for the practice, which includes options to integrate random sampling or entropy seeding with the support of quantum circuits, representing a new way to channel quantum algorithms into classical learning.

References

SHOWING 1-10 OF 64 REFERENCES

Regularization Learning Networks: Deep Learning for Tabular Datasets

TLDR
RLNs are introduced, which could efficiently learn a single network in datasets that comprise both tabular and unstructured data, such as in the setting of medical imaging accompanied by electronic health records.

Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data

Nowadays, deep neural networks (DNNs) have become the main instrument for machine learning tasks within a wide range of domains, including vision, NLP, and speech. Meanwhile, in an important case of

Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL

TLDR
This paper introduces Auto-PyTorch, which combines multi-fidelity optimization with portfolio construction for warmstarting and ensembling of deep neural networks (DNNs) and common baselines for tabular data to enable fully automated deep learning (AutoDL).

Net-DNF: Effective Deep Modeling of Tabular Data

TLDR
This work presents Net-DNF, a novel generic architecture whose inductive bias elicits models whose structure corresponds to logical Boolean formulas in disjunctive normal form (DNF) over affine soft-threshold decision terms, which opens the door to practical end-to-end handling of tabular data using neural networks.

Improved Regularization of Convolutional Neural Networks with Cutout

TLDR
This paper shows that the simple regularization technique of randomly masking out square regions of input during training, which is called cutout, can be used to improve the robustness and overall performance of convolutional neural networks.

Hyperparameter Ensembles for Robustness and Uncertainty Quantification

TLDR
This paper proposes hyper-deep ensembles, a simple procedure that involves a random search over different hyperparameters, themselves stratified across multiple random initializations, and proposes a parameter efficient version, hyper-batch ensembls, which builds on the layer structure of batch ensembleles and self-tuning networks.

BOHB: Robust and Efficient Hyperparameter Optimization at Scale

TLDR
This work proposes a new practical state-of-the-art hyperparameter optimization method, which consistently outperforms both Bayesian optimization and Hyperband on a wide range of problem types, including high-dimensional toy functions, support vector machines, feed-forward neural networks, Bayesian Neural networks, deep reinforcement learning, and convolutional neural networks.

Snapshot Ensembles: Train 1, get M for free

TLDR
This paper proposes a method to obtain the seemingly contradictory goal of ensembling multiple neural networks at no additional training cost by training a single neural network, converging to several local minima along its optimization path and saving the model parameters.

Lookahead Optimizer: k steps forward, 1 step back

TLDR
Lookahead improves the learning stability and lowers the variance of its inner optimizer with negligible computation and memory cost, and can significantly improve the performance of SGD and Adam, even with their default hyperparameter settings.

AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data

We introduce AutoGluon-Tabular, an open-source AutoML framework that requires only a single line of Python to train highly accurate machine learning models on an unprocessed tabular dataset such as a
...