Corpus ID: 211677668

Deep differentiable forest with sparse attention for the tabular data

@article{Chen2020DeepDF,
  title={Deep differentiable forest with sparse attention for the tabular data},
  author={Yingshi Chen},
  journal={ArXiv},
  year={2020},
  volume={abs/2003.00223}
}
  • Yingshi Chen
  • Published 29 February 2020
  • Computer Science, Mathematics
  • ArXiv
We present a general architecture of deep differentiable forest and its sparse attention mechanism. The differentiable forest has the advantages of both trees and neural networks. Its structure is a simple binary tree, easy to use and understand. It has full differentiability and all variables are learnable parameters. We would train it by the gradient-based optimization method, which shows great power in the training of deep CNN. We find and analyze the attention mechanism in the… Expand
Learning the Markov Decision Process in the Sparse Gaussian Elimination
TLDR
This study proposes a learning-based approach for the sparse Gaussian Elimination and proposes some Q-Learning algorithms for the main modules of sparse solver: minimum degree ordering, task scheduling and adaptive pivoting. Expand
Guided inverse design of surface plasmon polaritons based nanophotonic films via deep learning
In this paper, We present an improved deep convolutional neural network, Guided-Resnet, to design low-cost surface plasmon polaritons (SPP) based nanophotonic films. The input of Guided-Resnet is theExpand
A novel guided deep learning algorithm to design low-cost SPP films
TLDR
A novel guided deep learning algorithm is presented to find optimal solutions to the design of surface plasmon polaritons films (SPP) with both high accuracy and low cost. Expand
Learning Unsplit-field-based PML for the FDTD Method by Deep Differentiable Forest
TLDR
Numerical results illustrate that the proposed alternative unsplit-filed-based absorbing boundary condition (ABC) computation approach for the finite-difference time-domain (FDTD) can not only easily replace the traditional PML, but also be integrated into the FDTD computation process with satisfactory numerical accuracy and compatibility. Expand
An Expedient DDF-Based Implementation of Perfectly Matched Monolayer
TLDR
Results illustrate that the DDF-based PMM could not only replace the traditional PML but be integrated into FDTD computation process with satisfactory numerical accuracy. Expand

References

SHOWING 1-10 OF 26 REFERENCES
Deep Neural Decision Forests
TLDR
A novel approach that unifies classification trees with the representation learning functionality known from deep convolutional networks, by training them in an end-to-end manner by introducing a stochastic and differentiable decision tree model. Expand
Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data
Nowadays, deep neural networks (DNNs) have become the main instrument for machine learning tasks within a wide range of domains, including vision, NLP, and speech. Meanwhile, in an important case ofExpand
Random Hinge Forest for Differentiable Learning
TLDR
This work derives random hinge forests and ferns, focusing on their sparse and efficient nature, their min-max margin property, strategies to initialize them for arbitrary network architectures, and the class of optimizers most suitable for optimizing random hinge forest. Expand
LightGBM: A Highly Efficient Gradient Boosting Decision Tree
TLDR
It is proved that, since the data instances with larger gradients play a more important role in the computation of information gain, GOSS can obtain quite accurate estimation of the information gain with a much smaller data size, and is called LightGBM. Expand
Adam: A Method for Stochastic Optimization
TLDR
This work introduces Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments, and provides a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Expand
Deep Neural Decision Trees
TLDR
This work presents Deep Neural Decision Trees (DNDT) -- tree models realised by neural networks, which can be easily implemented in NN toolkits, and trained with gradient descent rather than greedy splitting. Expand
Multi-Layered Gradient Boosting Decision Trees
TLDR
This work proposes the multi-layered GBDT forest, with an explicit emphasis on exploring the ability to learn hierarchical representations by stacking several layers of regression GBDTs as its building block, and confirms the effectiveness of the model in terms of performance and representation learning ability. Expand
LiteMORT: A memory efficient gradient boosting tree system on adaptive compact distributions
TLDR
A deep analysis of this algorithm, especially the histogram technique, which is a basis for the regulized distribution with compact support, is given, and three new modifications are presented to reduce memory usage and improve accuracy. Expand
Adaptive Neural Trees
TLDR
Adapt neural trees via adaptive neural trees (ANTs) that incorporates representation learning into edges, routing functions and leaf nodes of a decision tree, along with a backpropagation-based training algorithm that adaptively grows the architecture from primitive modules (e.g., convolutional layers). Expand
Quasi-hyperbolic momentum and Adam for deep learning
TLDR
The quasi-hyperbolic momentum algorithm (QHM) is proposed as an extremely simple alteration of momentum SGD, averaging a plain SGD step with a momentum step, and a QH variant of Adam is proposed called QHAdam. Expand
...
1
2
3
...