• Corpus ID: 3815895

LightGBM: A Highly Efficient Gradient Boosting Decision Tree

@inproceedings{Ke2017LightGBMAH,
  title={LightGBM: A Highly Efficient Gradient Boosting Decision Tree},
  author={Guolin Ke and Qi Meng and Thomas Finley and Taifeng Wang and Wei Chen and Weidong Ma and Qiwei Ye and Tie-Yan Liu},
  booktitle={NIPS},
  year={2017}
}
Gradient Boosting Decision Tree (GBDT) is a popular machine learning algorithm, and has quite a few effective implementations such as XGBoost and pGBRT. [] Key Method With GOSS, we exclude a significant proportion of data instances with small gradients, and only use the rest to estimate the information gain. We prove that, since the data instances with larger gradients play a more important role in the computation of information gain, GOSS can obtain quite accurate estimation of the information gain with a…

Figures and Tables from this paper

HarpGBDT: Optimizing Gradient Boosting Decision Tree for Parallel Efficiency
TLDR
A new tree growth method that selects the top K candidates of tree nodes to enable the use of more levels of parallelism without sacrificing the algorithm’s accuracy is adopted and HarpGBDT, a new GBDT system designed from the perspective of parallel efficiency optimization is proposed.
A comparative analysis of gradient boosting algorithms
TLDR
A comprehensive comparison between XGBoost, LightGBM, CatBoost, random forests and gradient boosting has been performed and indicates that CatBoost obtains the best results in generalization accuracy and AUC in the studied datasets although the differences are small.
Gradient Boosting With Piece-Wise Linear Regression Trees
TLDR
This paper extends gradient boosting to use piecewise linear regression trees (PL Trees), instead of piecewise constant regression trees, as base learners, and shows that PL Trees can accelerate convergence of GBDT and improve the accuracy.
Benchmarking and Optimization of Gradient Boosted Decision Tree Algorithms
TLDR
This paper evaluates the performance of the GPU acceleration provided by XGBoost, LightGBM and Catboost using large-scale datasets with varying shapes, sparsities and learning tasks and compares the packages in the context of hyper-parameter optimization.
LiteMORT: A memory efficient gradient boosting tree system on adaptive compact distributions
TLDR
A deep analysis of this algorithm, especially the histogram technique, which is a basis for the regulized distribution with compact support, is given, and three new modifications are presented to reduce memory usage and improve accuracy.
An Experimental Evaluation of Large Scale GBDT Systems
TLDR
A quadrant categorization of data management policies based on data partitioning and data storage is introduced and a novel distributed GBDT system named Vero is proposed, which adopts the unexplored composition of vertical partitions and row-store and suits for many large-scale cases.
Block-distributed Gradient Boosted Trees
TLDR
This work achieves communication efficiency by making full use of the data sparsity and adapting the Quickscorer algorithm to the block-distributed setting, and allows more cost-effective scale-out without the need for expensive network communication.
ThunderGBM: Fast GBDTs and Random Forests on GPUs
TLDR
The experimental results show that ThunderGBM outperforms the existing libraries while producing similar models, and can handle high dimensional problems where existing GPU-based libraries fail.
Exploiting GPUs for Efficient Gradient Boosting Decision Tree Training
TLDR
The experimental results show that the algorithm named ThunderGBM can be 10x times faster than the state-of-the-art libraries (i.e., XGBoost, LightGBM and CatBoost) running on a relatively high-end workstation of 20 CPU cores.
Competitive Analysis of the Top Gradient Boosting Machine Learning Algorithms
TLDR
This research performs an exhaustive 360 degree comparative analysis of each of the four state-of-the-art gradient boosting algorithms viz.
...
...

References

SHOWING 1-10 OF 30 REFERENCES
GPU-acceleration for Large-scale Tree Boosting
TLDR
A novel massively parallel algorithm for accelerating the decision tree building procedure on GPUs, which is a crucial step in Gradient Boosted Decision Tree (GBDT) and random forests training and can be used as a drop-in replacement for histogram construction in popular tree boosting systems to improve their scalability.
Boosting with Maximum Adaptive Sampling
TLDR
Experiments in object recognition with two standard computer vision data-sets show that the adaptive methods proposed outperform basic sampling and state-of-the-art bandit methods.
XGBoost: A Scalable Tree Boosting System
TLDR
This paper proposes a novel sparsity-aware algorithm for sparse data and weighted quantile sketch for approximate tree learning and provides insights on cache access patterns, data compression and sharding to build a scalable tree boosting system called XGBoost.
Quickly Boosting Decision Trees - Pruning Underachieving Features Early
TLDR
A bound on the error of a decision stump is proved given its preliminary error on a subset of the training data; the bound may be used to prune unpromising features early in the training process, yielding speedups of an order of magnitude at no cost in the final performance of the classifier.
A Communication-Efficient Parallel Algorithm for Decision Tree
TLDR
Experiments on real-world datasets show that PV-Tree significantly outperforms the existing parallel decision tree algorithms in the tradeoff between accuracy and efficiency.
Best-first Decision Tree Learning
TLDR
This thesis introduces the algorithm for building binary best- first decision trees for classification problems and investigates two new pruning methods that determine an appropriate tree size by combining best-first decision tree growth with cross-validation-based selection of the number of expansions that are performed.
Parallel boosted regression trees for web search ranking
TLDR
A novel method for parallelizing the training of GBRT, which parallelizes the construction of the individual regression trees and operates using the master-worker paradigm, and demonstrates that the loss in accuracy induced due to the histogram approximation in the regression tree creation can be compensated for through slightly deeper trees.
CLOUDS: A Decision Tree Classifier for Large Datasets
TLDR
A novel decision tree classifier called CLOUDS is presented, which samples the splitting points for numeric attributes followed by an estimation step to narrow the search space of the best split and reduces computation and I/O complexity substantially compared to state of the art classifiers.
Generalized Boosted Models: A guide to the gbm package
TLDR
The main purposes of this document is to spell out in detail what the gbm package implements, mostly due to an effort to cast boosting terms into more standard statistical terminology.
Logistic Regression, AdaBoost and Bregman Distances
TLDR
A unified account of boosting and logistic regression in which each learning problem is cast in terms of optimization of Bregman distances, and a parameterized family of algorithms that includes both a sequential- and a parallel-update algorithm as special cases are described, thus showing how the sequential and parallel approaches can themselves be unified.
...
...