• Corpus ID: 195767546

Efficient Regularized Piecewise-Linear Regression Trees

  title={Efficient Regularized Piecewise-Linear Regression Trees},
  author={Leonidas Lefakis and Oleksandr Zadorozhnyi and Gilles Blanchard},
We present a detailed analysis of the class of regression decision tree algorithms which employ a regulized piecewise-linear node-splitting criterion and have regularized linear models at the leaves. From a theoretic standpoint, based on Rademacher complexity framework, we present new high-probability upper bounds for the generalization error for the proposed classes of regularized regression decision tree algorithms, including LASSO-type, and $\ell_{2}$ regularization for linear models at the… 

Figures and Tables from this paper



Scalable look-ahead linear regression trees

The motivation behind Look-ahead Linear Regression Trees (LLRT) is that out of all the methods proposed to date, there has been no scalable approach to exhaustively evaluate all possible models in the leaf nodes in order to obtain an optimal split.

Efficient Non-greedy Optimization of Decision Trees

It is shown that the problem of finding optimal linear-combination splits for decision trees is related to structured prediction with latent variables, and a convex-concave upper bound on the tree's empirical loss is formed, and the use of stochastic gradient descent for optimization enables effective training with large datasets.

Error Bounds for Piecewise Smooth and Switching Regression

  • Fabien Lauer
  • Mathematics, Computer Science
    IEEE Transactions on Neural Networks and Learning Systems
  • 2020
This paper derives generalization error bounds in piecewise smooth (PWS) regression and switching regression by following the approach based on Rademacher complexities, and shows how to recover a radical dependence.

Risk Bounds for CART Regression Trees

The aim of the Classification And Regression Trees (CART) proposed by Breiman, Friedman, Olshen and Stone [1] in 1984 is to construct some efficient algorithm which gives a piecewise constant


The proposed algorithm, GUIDE, is specifically designed to eliminate variable selection bias, a problem that can undermine the reliability of inferences from a tree structure and allows fast computation speed, natural ex- tension to data sets with categorical variables, and direct detection of local two- variable interactions.

SECRET: a scalable linear regression tree algorithm

A novel regression tree construction algorithm (SECRET) that produces trees of high quality and scales to very large datasets that has accuracy comparable to other linear regression tree algorithms but takes orders of magnitude less computation time for large datasets.

Piecewise polynomial regression trees

The advantages of the proposed estimate are that it possesses a smooth analytic form, is as many times differentiable as the family of weight functions are, and has a decision tree representation.

Random Composite Forests

A new algorithm, RANDOMCOMPOSITEFORESTS (RCF), that is based on forming an ensemble of random composite trees that yields significant performance improvements over both Random Forests and a variant of RCF in several tasks is devised.

Rademacher penalties and structural risk minimization

This work suggests a penalty function to be used in various problems of structural risk minimization, based on the sup-norm of the so-called Rademacher process indexed by the underlying class of functions (sets), and obtains oracle inequalities for the theoretical risk of estimators, obtained by structural minimization of the empirical risk withRademacher penalties.

BART: Bayesian Additive Regression Trees

We develop a Bayesian "sum-of-trees" model where each tree is constrained by a regularization prior to be a weak learner, and fitting and inference are accomplished via an iterative Bayesian