# Machine Learning

This page documents all the machine learning algorithms present in the library. In particular, there are algorithms for performing binary classification, regression, clustering, anomaly detection, and feature ranking, as well as algorithms for doing more specialized computations.

A good tutorial and introduction to the general concepts used by most of the objects in this part of the library can be found in the svm example program.

The major design goal of this portion of the library is to provide a highly modular and simple architecture for dealing with kernel algorithms. Towards this end, dlib takes a generic programming approach using C++ templates. In particular, each algorithm is parameterized to allow a user to supply either one of the predefined dlib kernels (e.g. RBF operating on column vectors), or a new user defined kernel. Moreover, the implementations of the algorithms are totally separated from the data on which they operate. This makes the dlib implementation generic enough to operate on any kind of data, be it column vectors, images, or some other form of structured data. All that is necessary is an appropriate kernel.

Primary Algorithms
mlp
krls
kcentroid
linearly_independent_subset_finder
kkmeans
svm_nu_trainer
rvm_trainer
rvm_regression_trainer
rbf_network_trainer
rank_features
svm_pegasos
Trainer Adapters
train_probabilistic_decision_function
reduced_decision_function_trainer
reduced
reduced_decision_function_trainer2
reduced2
batch
verbose_batch
batch_trainer
Kernels
radial_basis_kernel
polynomial_kernel
sigmoid_kernel
linear_kernel
offset_kernel
sparse_radial_basis_kernel
sparse_polynomial_kernel
sparse_sigmoid_kernel
sparse_linear_kernel
Function Objects
decision_function
distance_function
probabilistic_decision_function
normalized_function
Miscellaneous
vector_normalizer
randomize_samples
is_binary_classification_problem
test_binary_decision_function
cross_validate_trainer
cross_validate_trainer_threaded
pick_initial_centers
pick_initial_centers
dlib/svm.h
dlib/svm/kkmeans_abstract.h
This is a function that you can use to seed data clustering algorithms like the kkmeans clustering method. What it does is pick reasonable starting points for clustering by basically trying to find a set of points that are all far away from each other.
kkmeans_ex.cpp.html
mlp
dlib/mlp.h
dlib/mlp/mlp_kernel_abstract.h

This object represents a multilayer layer perceptron network that is trained using the back propagation algorithm. The training algorithm also incorporates the momentum method. That is, each round of back propagation training also adds a fraction of the previous update. This fraction is controlled by the momentum term set in the constructor.

It is worth noting that a MLP is, in general, very inferior to modern kernel algorithms such as the support vector machine. So if you haven't tried any other techniques with your data you really should.

mlp_ex.cpp.html
mlp_kernel_1
dlib/mlp/mlp_kernel_1.h
This is implemented in the obvious way.
kernel_1a
is a typedef for mlp_kernel_1
krls
dlib/svm.h
dlib/svm/krls_abstract.h
This is an implementation of the kernel recursive least squares algorithm described in the paper The Kernel Recursive Least Squares Algorithm by Yaakov Engel.

The long and short of this algorithm is that it is an online kernel based regression algorithm. You give it samples (x,y) and it learns the function f(x) == y. For a detailed description of the algorithm read the above paper.

krls_ex.cpp.html
krls_filter_ex.cpp.html
svm_pegasos
dlib/svm.h
dlib/svm/pegasos_abstract.h
This object implements an online algorithm for training a support vector machine for solving binary classification problems.

The implementation of the Pegasos algorithm used by this object is based on the following excellent paper:

> Pegasos: Primal estimated sub-gradient solver for SVM (2007) by Yoram Singer, Nathan Srebro In ICML

This SVM training algorithm has two interesting properties. First, the pegasos algorithm itself converges to the solution in an amount of time unrelated to the size of the training set (in addition to being quite fast to begin with). This makes it an appropriate algorithm for learning from very large datasets. Second, this object uses the kcentroid object to maintain a sparse approximation of the learned decision function. This means that the number of support vectors in the resulting decision function is also unrelated to the size of the dataset (in normal SVM training algorithms, the number of support vectors grows approximately linearly with the size of the training set).

svm_pegasos_ex.cpp.html
svm_sparse_ex.cpp.html
kkmeans
dlib/svm.h
dlib/svm/kkmeans_abstract.h
This is an implementation of a kernelized k-means clustering algorithm. It performs k-means clustering by using the kcentroid object.
kkmeans_ex.cpp.html
vector_normalizer
dlib/statistics.h
dlib/statistics/statistics_abstract.h
This object represents something that can learn to normalize a set of column vectors. In particular, normalized column vectors should have zero mean and a variance of one.

Also, if desired, this object can use principal component analysis for the purposes of reducing the number of elements in a vector.
svm_ex.cpp.html
linearly_independent_subset_finder
dlib/svm.h
dlib/svm/linearly_independent_subset_finder_abstract.h

This is an implementation of an online algorithm for recursively finding a set of linearly independent vectors in a kernel induced feature space. To use it you decide how large you would like the set to be and then you feed it sample points.

Each time you present it with a new sample point it either keeps the current set of independent points unchanged, or if the new point is "more linearly independent" than one of the points it already has, it replaces the weakly linearly independent point with the new one.

This object uses the Approximately Linearly Dependent metric described in the paper The Kernel Recursive Least Squares Algorithm by Yaakov Engel to decide which points are more linearly independent than others.

kcentroid
dlib/svm.h
dlib/svm/kcentroid_abstract.h

This object represents a weighted sum of sample points in a kernel induced feature space. It can be used to kernelize any algorithm that requires only the ability to perform vector addition, subtraction, scalar multiplication, and inner products.

An example use of this object is as an online algorithm for recursively estimating the centroid of a sequence of training points. This object then allows you to compute the distance between the centroid and any test points. So you can use this object to predict how similar a test point is to the data this object has been trained on (larger distances from the centroid indicate dissimilarity/anomalous points).

The object internally keeps a set of "dictionary vectors" that are used to represent the centroid. It manages these vectors using the sparsification technique described in the paper The Kernel Recursive Least Squares Algorithm by Yaakov Engel. This technique allows us to keep the number of dictionary vectors down to a minimum. In fact, the object has a user selectable tolerance parameter that controls the trade off between accuracy and number of stored dictionary vectors.

kcentroid_ex.cpp.html
train_probabilistic_decision_function
dlib/svm.h
dlib/svm/svm_abstract.h

Trains a probabilistic_decision_function using some sort of batch trainer object such as the svm_nu_trainer or rbf_network_trainer.

The probability model is created by using the technique described in the paper:

> Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods by John C. Platt. Match 26, 1999

svm_ex.cpp.html
rbf_network_trainer
dlib/svm.h
dlib/svm/rbf_network_abstract.h
Trains a radial basis function network and outputs a decision_function.
rvm_regression_trainer
dlib/svm.h
dlib/svm/rvm_abstract.h

Trains a relevance vector machine for solving regression problems. Outputs a decision_function that represents the learned regression function.

The implementation of the RVM training algorithm used by this library is based on the following paper:

> Tipping, M. E. and A. C. Faul (2003). Fast marginal likelihood maximisation for sparse Bayesian models. In C. M. Bishop and B. J. Frey (Eds.), Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics, Key West, FL, Jan 3-6.

rvm_regression_ex.cpp.html
rvm_trainer
dlib/svm.h
dlib/svm/rvm_abstract.h

Trains a relevance vector machine for solving binary classification problems. Outputs a decision_function that represents the learned classifier.

The implementation of the RVM training algorithm used by this library is based on the following paper:

> Tipping, M. E. and A. C. Faul (2003). Fast marginal likelihood maximisation for sparse Bayesian models. In C. M. Bishop and B. J. Frey (Eds.), Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics, Key West, FL, Jan 3-6.

rvm_ex.cpp.html
svm_nu_trainer
dlib/svm.h
dlib/svm/svm_abstract.h

Trains a nu support vector classifier and outputs a decision_function.

The implementation of the nu-svm training algorithm used by this library is based on the following excellent papers:

- Chang and Lin, Training {nu}-Support Vector Classifiers: Theory and Algorithms
- Chih-Chung Chang and Chih-Jen Lin, LIBSVM : a library for support vector machines, 2001. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm

svm_ex.cpp.html
normalized_function
dlib/svm.h
dlib/svm/function_abstract.h
This object represents a container for another function object and an instance of the vector_normalizer object.

It automatically normalizes all inputs before passing them off to the contained function object.
svm_ex.cpp.html
probabilistic_decision_function
dlib/svm.h
dlib/svm/function_abstract.h
This object represents a binary decision function for use with support vector machines. It returns an estimate of the probability that a given sample is in the +1 class.
svm_ex.cpp.html
distance_function
dlib/svm.h
dlib/svm/function_abstract.h
This object represents a point in kernel induced feature space. You may use this object to find the distance from the point it represents to points in input space.
decision_function
dlib/svm.h
dlib/svm/function_abstract.h
This object represents a decision or regression function that was learned by a kernel based learning algorithm.
svm_ex.cpp.html
offset_kernel
dlib/svm.h
dlib/svm/kernel_abstract.h
This object represents a kernel with a fixed value offset added to it.
linear_kernel
dlib/svm.h
dlib/svm/kernel_abstract.h
This object represents a linear function kernel for use with kernel learning machines.
sigmoid_kernel
dlib/svm.h
dlib/svm/kernel_abstract.h
This object represents a sigmoid kernel for use with kernel learning machines.
polynomial_kernel
dlib/svm.h
dlib/svm/kernel_abstract.h
This object represents a polynomial kernel for use with kernel learning machines.
radial_basis_kernel
dlib/svm.h
dlib/svm/kernel_abstract.h
This object represents a radial basis function kernel for use with kernel learning machines.
svm_ex.cpp.html
sparse_linear_kernel
dlib/svm.h
dlib/svm/sparse_kernel_abstract.h
This object represents a linear function kernel for use with kernel learning machines that operate on sparse vectors.
svm_sparse_ex.cpp.html
sparse_sigmoid_kernel
dlib/svm.h
dlib/svm/sparse_kernel_abstract.h
This object represents a sigmoid kernel for use with kernel learning machines that operate on sparse vectors.
sparse_polynomial_kernel
dlib/svm.h
dlib/svm/sparse_kernel_abstract.h
This object represents a polynomial kernel for use with kernel learning machines that operate sparse vectors.
sparse_radial_basis_kernel
dlib/svm.h
dlib/svm/sparse_kernel_abstract.h
This object represents a radial basis function kernel for use with kernel learning machines that operate sparse vectors.
is_binary_classification_problem
dlib/svm.h
dlib/svm/svm_abstract.h
This function simply takes two vectors, the first containing feature vectors and the second containing labels, and reports back if the two could possibly contain data for a well formed classification problem.
randomize_samples
dlib/svm.h
dlib/svm/svm_abstract.h
Randomizes the order of samples in a column vector containing sample data.
svm_ex.cpp.html
rank_features
dlib/svm.h
dlib/svm/feature_ranking_abstract.h
Finds a ranking of the top N (a user supplied parameter) features in a set of data from a two class classification problem. It does this by computing the distance between the centroids of both classes in kernel defined feature space. Good features are then ones that result in the biggest separation between the two centroids.
rank_features_ex.cpp.html
batch
dlib/svm.h
dlib/svm/pegasos_abstract.h
This is a convenience function for creating batch_trainer objects.
svm_pegasos_ex.cpp.html
verbose_batch
dlib/svm.h
dlib/svm/pegasos_abstract.h
This is a convenience function for creating batch_trainer objects. This function generates a batch_trainer that will print status messages to standard output so that you can observe the progress of a training algorithm.
svm_pegasos_ex.cpp.html
batch_trainer
dlib/svm.h
dlib/svm/pegasos_abstract.h
This is a batch trainer object that is meant to wrap online trainer objects that create decision_functions. It turns an online learning algorithm such as svm_pegasos into a batch learning object. This allows you to use objects like svm_pegasos with functions (e.g. cross_validate_trainer) that expect batch mode training objects.
reduced
dlib/svm.h
dlib/svm/reduced_abstract.h
This is a convenience function for creating reduced_decision_function_trainer objects.
reduced_decision_function_trainer
dlib/svm.h
dlib/svm/reduced_abstract.h
This is a batch trainer object that is meant to wrap other batch trainer objects that create decision_function objects. It performs post processing on the output decision_function objects with the intent of representing the decision_function with fewer support vectors.
reduced2
dlib/svm.h
dlib/svm/reduced_abstract.h
This is a convenience function for creating reduced_decision_function_trainer2 objects.
svm_ex.cpp.html
reduced_decision_function_trainer2
dlib/svm.h
dlib/svm/reduced_abstract.h

This is a batch trainer object that is meant to wrap other batch trainer objects that create decision_function objects. It performs post processing on the output decision_function objects with the intent of representing the decision_function with fewer support vectors.

It begins by performing the same post processing as the reduced_decision_function_trainer object but it also performs a global gradient based optimization to further improve the results.

svm_ex.cpp.html
test_binary_decision_function
dlib/svm.h
dlib/svm/svm_abstract.h
Tests a decision_function that represents a binary decision function and returns the test accuracy.

cross_validate_trainer_threaded
dlib/svm_threaded.h
dlib/svm/svm_threaded_abstract.h
Performs k-fold cross validation on a user supplied trainer object such as the svm_nu_trainer or rbf_network_trainer. This function does the same thing as cross_validate_trainer except this function also allows you to specify how many threads of execution to use. So you can use this function to take advantage of a multi-core system to perform cross validation faster.
cross_validate_trainer
dlib/svm.h
dlib/svm/svm_abstract.h
Performs k-fold cross validation on a user supplied trainer object such as the svm_nu_trainer or rbf_network_trainer.
svm_ex.cpp.html