Package smile.classification
The instance is usually described by a vector of features, which together constitute a description of all known characteristics of the instance. Typically, features are either categorical (also known as nominal, i.e. consisting of one of a set of unordered items, such as a gender of "male" or "female", or a blood type of "A", "B", "AB" or "O"), ordinal (consisting of one of a set of ordered items, e.g. "large", "medium" or "small"), integervalued (e.g. a count of the number of occurrences of a particular word in an email) or realvalued (e.g. a measurement of blood pressure).
Classification normally refers to a supervised procedure, i.e. a procedure that produces an inferred function to predict the output value of new instances based on a training set of pairs consisting of an input object and a desired output value. The inferred function is called a classifier if the output is discrete or a regression function if the output is continuous.
The inferred function should predict the correct output value for any valid input object. This requires the learning algorithm to generalize from the training data to unseen situations in a "reasonable" way.
A wide range of supervised learning algorithms is available, each with its strengths and weaknesses. There is no single learning algorithm that works best on all supervised learning problems. The most widely used learning algorithms are AdaBoost and gradient boosting, support vector machines, linear regression, linear discriminant analysis, logistic regression, naive Bayes, decision trees, knearest neighbor algorithm, and neural networks (multilayer perceptron).
If the feature vectors include features of many different kinds (discrete, discrete ordered, counts, continuous values), some algorithms cannot be easily applied. Many algorithms, including linear regression, logistic regression, neural networks, and nearest neighbor methods, require that the input features be numerical and scaled to similar ranges (e.g., to the [1,1] interval). Methods that employ a distance function, such as nearest neighbor methods and support vector machines with Gaussian kernels, are particularly sensitive to this. An advantage of decision trees (and boosting algorithms based on decision trees) is that they easily handle heterogeneous data.
If the input features contain redundant information (e.g., highly correlated features), some learning algorithms (e.g., linear regression, logistic regression, and distance based methods) will perform poorly because of numerical instabilities. These problems can often be solved by imposing some form of regularization.
If each of the features makes an independent contribution to the output, then algorithms based on linear functions (e.g., linear regression, logistic regression, linear support vector machines, naive Bayes) generally perform well. However, if there are complex interactions among features, then algorithms such as nonlinear support vector machines, decision trees and neural networks work better. Linear methods can also be applied, but the engineer must manually specify the interactions when using them.
There are several major issues to consider in supervised learning:
 Features
 The accuracy of the inferred function depends strongly on how the input
object is represented. Typically, the input object is transformed into
a feature vector, which contains a number of features that are descriptive
of the object. The number of features should not be too large, because of
the curse of dimensionality; but should contain enough information to
accurately predict the output.
There are many algorithms for feature selection that seek to identify the relevant features and discard the irrelevant ones. More generally, dimensionality reduction may seek to map the input data into a lower dimensional space prior to running the supervised learning algorithm.
 Overfitting
 Overfitting occurs when a statistical model describes random error
or noise instead of the underlying relationship. Overfitting generally
occurs when a model is excessively complex, such as having too many
parameters relative to the number of observations. A model which has
been overfit will generally have poor predictive performance, as it can
exaggerate minor fluctuations in the data.
The potential for overfitting depends not only on the number of parameters and data but also the conformability of the model structure with the data shape, and the magnitude of model error compared to the expected level of noise or error in the data.
In order to avoid overfitting, it is necessary to use additional techniques (e.g. crossvalidation, regularization, early stopping, pruning, Bayesian priors on parameters or model comparison), that can indicate when further training is not resulting in better generalization. The basis of some techniques is either (1) to explicitly penalize overly complex models, or (2) to test the model's ability to generalize by evaluating its performance on a set of data not used for training, which is assumed to approximate the typical unseen data that a model will encounter.
 Regularization
 Regularization involves introducing additional information in order
to solve an illposed problem or to prevent overfitting. This information
is usually of the form of a penalty for complexity, such as restrictions
for smoothness or bounds on the vector space norm.
A theoretical justification for regularization is that it attempts to impose Occam's razor on the solution. From a Bayesian point of view, many regularization techniques correspond to imposing certain prior distributions on model parameters.
 Biasvariance tradeoff
 Mean squared error (MSE) can be broken down into two components: variance and squared bias, known as the biasvariance decomposition. Thus in order to minimize the MSE, we need to minimize both the bias and the variance. However, this is not trivial. Therefore, there is a tradeoff between bias and variance.

ClassDescriptionAbstract base class of classifiers.AdaBoost (Adaptive Boosting) classifier with decision trees.Classifier<T>A classifier assigns an input object into one of a given number of categories.Classifier.Trainer<T,
M extends Classifier<T>> The classifier trainer.Map arbitrary class labels to [0, k), where k is the number of classes.Classification trait on DataFrame.DataFrameClassifier.Trainer<M extends DataFrameClassifier>The classifier trainer.Decision tree.Naive Bayes classifier for document classification in NLP.The generation models of naive Bayes classifier.Fisher's linear discriminant.Gradient boosting for classification.A method to calibrate decision function value to probability.KNN<T>Knearest neighbor classifier.Linear discriminant analysis.Logistic regression.Binomial logistic regression.Multinomial logistic regression.Maximum Entropy Classifier.Binomial maximum entropy classifier.Multinomial maximum entropy classifier.Fully connected multilayer perceptron neural network for classification.Naive Bayes classifier.OneVersusOne<T>Onevsone strategy for reducing the problem of multiclass classification to multiple binary classification problems.Onevsrest (or onevsall) strategy for reducing the problem of multiclass classification to multiple binary classification problems.Platt scaling or Platt calibration is a way of transforming the outputs of a classification model into a probability distribution over classes.Quadratic discriminant analysis.Random forest for classification.The base model.RBFNetwork<T>Radial basis function networks.Regularized discriminant analysis.Logistic regression on sparse data.Binomial logistic regression.Multinomial logistic regression.SVM<T>Support vector machines for classification.