public class GradientTreeBoost extends java.lang.Object implements SoftClassifier<double[]>, java.io.Serializable
Generic gradient boosting at the tth step would fit a regression tree to pseudoresiduals. Let J be the number of its leaves. The tree partitions the input space into J disjoint regions and predicts a constant value in each region. The parameter J controls the maximum allowed level of interaction between variables in the model. With J = 2 (decision stumps), no interaction between variables is allowed. With J = 3 the model may include effects of the interaction between up to two variables, and so on. Hastie et al. comment that typically 4 ≤ J ≤ 8 work well for boosting and results are fairly insensitive to the choice of in this range, J = 2 is insufficient for many applications, and J > 10 is unlikely to be required.
Fitting the training set too closely can lead to degradation of the model's generalization ability. Several socalled regularization techniques reduce this overfitting effect by constraining the fitting procedure. One natural regularization parameter is the number of gradient boosting iterations T (i.e. the number of trees in the model when the base learner is a decision tree). Increasing T reduces the error on training set, but setting it too high may lead to overfitting. An optimal value of T is often selected by monitoring prediction error on a separate validation data set.
Another regularization approach is the shrinkage which times a parameter η (called the "learning rate") to update term. Empirically it has been found that using small learning rates (such as η < 0.1) yields dramatic improvements in model's generalization ability over gradient boosting without shrinking (η = 1). However, it comes at the price of increasing computational time both during training and prediction: lower learning rate requires more iterations.
Soon after the introduction of gradient boosting Friedman proposed a minor modification to the algorithm, motivated by Breiman's bagging method. Specifically, he proposed that at each iteration of the algorithm, a base learner should be fit on a subsample of the training set drawn at random without replacement. Friedman observed a substantial improvement in gradient boosting's accuracy with this modification.
Subsample size is some constant fraction f of the size of the training set. When f = 1, the algorithm is deterministic and identical to the one described above. Smaller values of f introduce randomness into the algorithm and help prevent overfitting, acting as a kind of regularization. The algorithm also becomes faster, because regression trees have to be fit to smaller datasets at each iteration. Typically, f is set to 0.5, meaning that one half of the training set is used to build each base learner.
Also, like in bagging, subsampling allows one to define an outofbag estimate of the prediction performance improvement by evaluating predictions on those observations which were not used in the building of the next base learner. Outofbag estimates help avoid the need for an independent validation dataset, but often underestimate actual performance improvement and the optimal number of iterations.
Gradient tree boosting implementations often also use regularization by limiting the minimum number of observations in trees' terminal nodes. It's used in the tree building process by ignoring any splits that lead to nodes containing fewer than this number of training set instances. Imposing this limit helps to reduce variance in predictions at leaves.
Modifier and Type  Class and Description 

static class 
GradientTreeBoost.Trainer
Trainer for GradientTreeBoost classifiers.

Constructor and Description 

GradientTreeBoost(Attribute[] attributes,
double[][] x,
int[] y,
int ntrees)
Constructor.

GradientTreeBoost(Attribute[] attributes,
double[][] x,
int[] y,
int ntrees,
int maxNodes,
double shrinkage,
double subsample)
Constructor.

GradientTreeBoost(double[][] x,
int[] y,
int ntrees)
Constructor.

GradientTreeBoost(double[][] x,
int[] y,
int ntrees,
int maxNodes,
double shrinkage,
double f)
Constructor.

Modifier and Type  Method and Description 

RegressionTree[] 
getTrees()
Returns the regression trees.

double[] 
importance()
Returns the variable importance.

int 
predict(double[] x)
Predicts the class label of an instance.

int 
predict(double[] x,
double[] posteriori)
Predicts the class label of an instance and also calculate a posteriori
probabilities.

int 
size()
Returns the number of trees in the model.

double[] 
test(double[][] x,
int[] y)
Test the model on a validation dataset.

double[][] 
test(double[][] x,
int[] y,
ClassificationMeasure[] measures)
Test the model on a validation dataset.

void 
trim(int ntrees)
Trims the tree model set to a smaller size in case of overfitting.

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
predict
public GradientTreeBoost(double[][] x, int[] y, int ntrees)
x
 the training instances.y
 the class labels.ntrees
 the number of iterations (trees).public GradientTreeBoost(double[][] x, int[] y, int ntrees, int maxNodes, double shrinkage, double f)
x
 the training instances.y
 the class labels.ntrees
 the number of iterations (trees).maxNodes
 the number of leaves in each tree.shrinkage
 the shrinkage parameter in (0, 1] controls the learning rate of procedure.f
 the sampling rate for stochastic tree boosting.public GradientTreeBoost(Attribute[] attributes, double[][] x, int[] y, int ntrees)
attributes
 the attribute properties.x
 the training instances.y
 the class labels.ntrees
 the number of iterations (trees).public GradientTreeBoost(Attribute[] attributes, double[][] x, int[] y, int ntrees, int maxNodes, double shrinkage, double subsample)
attributes
 the attribute properties.x
 the training instances.y
 the class labels.ntrees
 the number of iterations (trees).maxNodes
 the number of leaves in each tree.shrinkage
 the shrinkage parameter in (0, 1] controls the learning rate of procedure.subsample
 the sampling fraction for stochastic tree boosting.public double[] importance()
public int size()
public void trim(int ntrees)
ntrees
 the new (smaller) size of tree model set.public int predict(double[] x)
Classifier
predict
in interface Classifier<double[]>
x
 the instance to be classified.public int predict(double[] x, double[] posteriori)
SoftClassifier
predict
in interface SoftClassifier<double[]>
x
 the instance to be classified.posteriori
 the array to store a posteriori probabilities on output.public double[] test(double[][] x, int[] y)
x
 the test data set.y
 the test data response values.public double[][] test(double[][] x, int[] y, ClassificationMeasure[] measures)
x
 the test data set.y
 the test data labels.measures
 the performance measures of classification.public RegressionTree[] getTrees()