Gradient boosting, explained
Google’s course explains it with two kinds of model: a weak one, typically a small decision tree, and a strong one made of many weak ones added together. At each step a new weak model is trained to predict the current error, then added to the strong model to reduce it. When the weak models are decision trees, the result is called gradient boosted trees.
A setting called shrinkage scales each new tree down before it is added. Google compares it to a learning rate: smaller steps learn more slowly but help stop the model from fitting noise. Training stops at a set limit, or when results on held back data stop improving.
Why use gradient boosting?
Google’s course says decision forests, the family gradient boosted trees belong to, work well on table data, such as rows of past orders with a number to predict. Each new tree only has to fix what the model before it got wrong, so the model improves step by step.
Can gradient boosting overfit?
Yes. Google notes that the strong model can begin to overfit, meaning it fits the training data too closely. Shrinkage and stopping when results on a separate check set get worse help prevent that.
Questions
What is gradient boosting in simple terms?
Gradient boosting builds a model from many small decision trees, one after another. Each new tree learns from the mistakes the model still makes.
What are gradient boosted trees?
Gradient boosted trees are gradient boosting with decision trees as the weak models. Google’s course uses this name for the most common form of the method.
Is gradient boosting a kind of machine learning?
Yes. It is a machine learning method that combines many weak models, usually small decision trees, into one stronger model.
Where we use this term
Sources
- Gradient Boosted Decision Trees. Google for Developers, Decision Forests course.
- Decision Forests: Introduction. Google for Developers, Decision Forests course.
- Machine Learning Glossary. Google for Developers.
Definition checked against these sources and our own pages on .