• Corpus ID: 211677668

Deep differentiable forest with sparse attention for the tabular data

  title={Deep differentiable forest with sparse attention for the tabular data},
  author={Yingshi Chen},
  • Yingshi Chen
  • Published 29 February 2020
  • Computer Science, Mathematics
  • ArXiv
We present a general architecture of deep differentiable forest and its sparse attention mechanism. The differentiable forest has the advantages of both trees and neural networks. Its structure is a simple binary tree, easy to use and understand. It has full differentiability and all variables are learnable parameters. We would train it by the gradient-based optimization method, which shows great power in the training of deep CNN. We find and analyze the attention mechanism in the… 
Learning the Markov Decision Process in the Sparse Gaussian Elimination
This study proposes a learning-based approach for the sparse Gaussian Elimination and proposes some Q-Learning algorithms for the main modules of sparse solver: minimum degree ordering, task scheduling and adaptive pivoting.
Guided inverse design of surface plasmon polaritons based nanophotonic films via deep learning
In this paper, We present an improved deep convolutional neural network, Guided-Resnet, to design low-cost surface plasmon polaritons (SPP) based nanophotonic films. The input of Guided-Resnet is the
A novel guided deep learning algorithm to design low-cost SPP films
A novel guided deep learning algorithm is presented to find optimal solutions to the design of surface plasmon polaritons films (SPP) with both high accuracy and low cost.
Learning Unsplit-field-based PML for the FDTD Method by Deep Differentiable Forest
Numerical results illustrate that the proposed alternative unsplit-filed-based absorbing boundary condition (ABC) computation approach for the finite-difference time-domain (FDTD) can not only easily replace the traditional PML, but also be integrated into the FDTD computation process with satisfactory numerical accuracy and compatibility.
An Expedient DDF-Based Implementation of Perfectly Matched Monolayer
Results illustrate that the DDF-based PMM could not only replace the traditional PML but be integrated into FDTD computation process with satisfactory numerical accuracy.


Deep Neural Decision Forests
A novel approach that unifies classification trees with the representation learning functionality known from deep convolutional networks, by training them in an end-to-end manner by introducing a stochastic and differentiable decision tree model.
Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data
Nowadays, deep neural networks (DNNs) have become the main instrument for machine learning tasks within a wide range of domains, including vision, NLP, and speech. Meanwhile, in an important case of
Random Hinge Forest for Differentiable Learning
This work derives random hinge forests and ferns, focusing on their sparse and efficient nature, their min-max margin property, strategies to initialize them for arbitrary network architectures, and the class of optimizers most suitable for optimizing random hinge forest.
LightGBM: A Highly Efficient Gradient Boosting Decision Tree
It is proved that, since the data instances with larger gradients play a more important role in the computation of information gain, GOSS can obtain quite accurate estimation of the information gain with a much smaller data size, and is called LightGBM.
Adam: A Method for Stochastic Optimization
This work introduces Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments, and provides a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework.
Deep Neural Decision Trees
This work presents Deep Neural Decision Trees (DNDT) -- tree models realised by neural networks, which can be easily implemented in NN toolkits, and trained with gradient descent rather than greedy splitting.
Multi-Layered Gradient Boosting Decision Trees
This work proposes the multi-layered GBDT forest, with an explicit emphasis on exploring the ability to learn hierarchical representations by stacking several layers of regression GBDTs as its building block, and confirms the effectiveness of the model in terms of performance and representation learning ability.
LiteMORT: A memory efficient gradient boosting tree system on adaptive compact distributions
A deep analysis of this algorithm, especially the histogram technique, which is a basis for the regulized distribution with compact support, is given, and three new modifications are presented to reduce memory usage and improve accuracy.
Adaptive Neural Trees
Adapt neural trees via adaptive neural trees (ANTs) that incorporates representation learning into edges, routing functions and leaf nodes of a decision tree, along with a backpropagation-based training algorithm that adaptively grows the architecture from primitive modules (e.g., convolutional layers).
Quasi-hyperbolic momentum and Adam for deep learning
The quasi-hyperbolic momentum algorithm (QHM) is proposed as an extremely simple alteration of momentum SGD, averaging a plain SGD step with a momentum step, and a QH variant of Adam is proposed called QHAdam.