Skip to content

GitLab

"src/vscode:/vscode.git/clone" did not exist on "314a75f367a0d41e158edfc2edbb54eb0a1ae235"
Commit 2f4217c7 authored by Davis King's avatar Davis King
Browse files

Updated the example programs regarding model selection. 

--HG--
extra : convert_revision : svn%3Afdd8eb12-d10e-0410-9acb-85c331704f74/trunk%403256
parent 69d28ee0
Hide whitespace changes
Inline Side-by-side
Showing with 181 additions and 116 deletions
examples/model_selection_ex.cpp
View file @ 2f4217c7
// The contents of this file are in the public domain. See LICENSE_FOR_EXAMPLE_PROGRAMS.txt
/*
This is an example that shows you some reasonable ways you can perform
This is an example that shows some reasonable ways you can perform
model selection with the dlib C++ Library.
This example creates a simple set of data and then shows you how to use
......@@ -13,6 +13,9 @@
from the origin are labeled +1 and all other points are labeled
as -1.
As an side, you should probably read the svm_ex.cpp and matrix_ex.cpp example
programs before you read this one.
*/
......@@ -26,10 +29,7 @@ using namespace dlib;
// operate. So the first thing we do here is declare a convenient typedef.
// This typedef declares a matrix with 2 rows and 1 column. It will be the
// object that contains each of our 2 dimensional samples. (Note that if you wanted
// more than 2 features in this vector you can simply change the 2 to something else.
// Or if you don't know how many features you want until runtime then you can put a 0
// here and use the matrix.set_size() member function)
// object that contains each of our 2 dimensional samples.
typedef matrix<double, 2, 1> sample_type;
// This is a typedef for the type of kernel we are going to use in this example.
......@@ -39,8 +39,18 @@ typedef radial_basis_kernel<sample_type> kernel_type;
class cross_validation_objective
{
/*!
WHAT THIS OBJECT REPRESENTS
This object is a simple function object that takes a set of model
parameters and returns a number indicating how "good" they are. It
does this by performing 10 fold cross validation on our dataset
and reporting the accuracy.
See below in main() for how this object gets used.
!*/
public:
cross_validation_objective (
......@@ -52,15 +62,27 @@ public:
const matrix<double>& params
) const
{
// Pull out the two SVM model parameters. Note that, in this case,
// I have setup the parameter search to operate in log scale so we have
// to remember to call exp() to put the parameters back into a normal scale.
const double gamma = exp(params(0));
const double nu = exp(params(1));
// Make an SVM trainer and tell it what the parameters are supposed to be.
svm_nu_trainer<kernel_type> trainer;
trainer.set_kernel(kernel_type(gamma));
trainer.set_nu(nu);
// Finally, perform 10-fold cross validation and then print and return the results.
matrix<double> result = cross_validate_trainer(trainer, samples, labels, 10);
cout << "gamma: " << setw(11) << gamma << " nu: " << setw(11) << nu << " cross validation accuracy: " << result;
// Here I'm just summing the accuracy on each class. However, you could do something else.
// For example, your application might require a 90% accuracy on class +1 and so you could
// heavily penalize results that didn't obtain the desired accuracy. Or similarly, you
// might use the roc_c1_trainer() function to adjust the trainer output so that it always
// obtained roughly a 90% accuracy on class +1. In that case returning the sum of the two
// class accuracies might then be appropriate.
return sum(result);
}
......@@ -69,131 +91,174 @@ public:
};
int main()
{
// Now we make objects to contain our samples and their respective labels.
std::vector<sample_type> samples;
std::vector<double> labels;
// Now lets put some data into our samples and labels objects. We do this
// by looping over a bunch of points and labeling them according to their
// distance from the origin.
for (int r = -20; r <= 20; ++r)
try
{
for (int c = -20; c <= 20; ++c)
{
sample_type samp;
samp(0) = r;
samp(1) = c;
samples.push_back(samp);
// if this point is less than 10 from the origin
if (sqrt((double)r*r + c*c) <= 10)
labels.push_back(+1);
else
labels.push_back(-1);
// Now we make objects to contain our samples and their respective labels.
std::vector<sample_type> samples;
std::vector<double> labels;
// Now lets put some data into our samples and labels objects. We do this
// by looping over a bunch of points and labeling them according to their
// distance from the origin.
for (double r = -20; r <= 20; r += 0.8)
{
for (double c = -20; c <= 20; c += 0.8)
{
sample_type samp;
samp(0) = r;
samp(1) = c;
samples.push_back(samp);
// if this point is less than 10 from the origin
if (sqrt(r*r + c*c) <= 10)
labels.push_back(+1);
else
labels.push_back(-1);
}
}
}
// Here we normalize all the samples by subtracting their mean and dividing by their standard deviation.
// This is generally a good idea since it often heads off numerical stability problems and also
// prevents one large feature from smothering others. Doing this doesn't matter much in this example
// so I'm just doing this here so you can see an easy way to accomplish this with
// the library.
vector_normalizer<sample_type> normalizer;
// let the normalizer learn the mean and standard deviation of the samples
normalizer.train(samples);
// now normalize each sample
for (unsigned long i = 0; i < samples.size(); ++i)
samples[i] = normalizer(samples[i]);
// Now that we have some data we want to train on it. However, there are two parameters to the
// training. These are the nu and gamma parameters. Our choice for these parameters will
// influence how good the resulting decision function is. To test how good a particular choice
// of these parameters are we can use the cross_validate_trainer() function to perform n-fold cross
// validation on our training data. However, there is a problem with the way we have sampled
// our distribution above. The problem is that there is a definite ordering to the samples.
// That is, the first half of the samples look like they are from a different distribution
// than the second half do. This would screw up the cross validation process but we can
// fix it by randomizing the order of the samples with the following function call.
randomize_samples(samples, labels);
// The nu parameter has a maximum value that is dependent on the ratio of the +1 to -1
// labels in the training data. This function finds that value.
const double max_nu = maximum_nu(labels);
// here we make an instance of the svm_nu_trainer object that uses our kernel type.
svm_nu_trainer<kernel_type> trainer;
cout << "Generated " << samples.size() << " points" << endl;
// Here we normalize all the samples by subtracting their mean and dividing by their standard deviation.
// This is generally a good idea since it often heads off numerical stability problems and also
// prevents one large feature from smothering others. Doing this doesn't matter much in this example
// so I'm just doing this here so you can see an easy way to accomplish this with
// the library.
vector_normalizer<sample_type> normalizer;
// let the normalizer learn the mean and standard deviation of the samples
normalizer.train(samples);
// now normalize each sample
for (unsigned long i = 0; i < samples.size(); ++i)
samples[i] = normalizer(samples[i]);
// Now that we have some data we want to train on it. However, there are two parameters to the
// training. These are the nu and gamma parameters. Our choice for these parameters will
// influence how good the resulting decision function is. To test how good a particular choice
// of these parameters are we can use the cross_validate_trainer() function to perform n-fold cross
// validation on our training data. However, there is a problem with the way we have sampled
// our distribution above. The problem is that there is a definite ordering to the samples.
// That is, the first half of the samples look like they are from a different distribution
// than the second half do. This would screw up the cross validation process but we can
// fix it by randomizing the order of the samples with the following function call.
randomize_samples(samples, labels);
// The nu parameter has a maximum value that is dependent on the ratio of the +1 to -1
// labels in the training data. This function finds that value.
const double max_nu = maximum_nu(labels);
// The first kind of model selection we will do is a simple grid search. That is, below we just
// generate a fixed grid of points (each point represents one possible setting of the model parameters).
// and test each via cross validation.
// This code generates a 4x4 grid of logarithmically spaced points. The result is a matrix
// with 2 rows and 16 columns where each column represents one of our points.
matrix<double> params = cartesian_product(logspace(log10(5.0), log10(1e-5), 4), // gamma parameter
logspace(log10(max_nu), log10(1e-5), 4) // nu parameter
);
// As an aside, if you wanted to do a grid search over points of dimensionality more than two
// you would just nest calls to cartesian_product(). You can also use linspace() to generate
// linearly spaced points if that is more appropriate for the parameters you are working with.
// Next we loop over all the points we generated and check how good each is.
cout << "Doing a grid search" << endl;
matrix<double> best_result(2,1);
best_result = 0;
double best_gamma, best_nu;
set_all_elements(best_result, 0);
for (long col = 0; col < params.nc(); ++col)
{
// pull out the current set of model parameters
const double gamma = params(0, col);
const double nu = params(1, col);
// setup a training object using our current parameters
svm_nu_trainer<kernel_type> trainer;
trainer.set_kernel(kernel_type(gamma));
trainer.set_nu(nu);
// Finally, do 10 fold cross validation and then check if the results are the best we have seen so far.
matrix<double> result = cross_validate_trainer(trainer, samples, labels, 10);
cout << "gamma: " << setw(11) << gamma << " nu: " << setw(11) << nu << " cross validation accuracy: " << result;
// save the best results
if (sum(result) > sum(best_result))
{
best_result = result;
best_gamma = gamma;
best_nu = nu;
}
}
cout << "\n best result of grid search: " << sum(best_result) << endl;
cout << " best gamma: " << best_gamma << " best nu: " << best_nu << endl;
// Lets do a simple grid search
matrix<double> params = cartesian_product(logspace(log10(20.0), log10(1e-5), 4), // gamma parameter
logspace(log10(max_nu), log10(1e-5), 4) // nu parameter
);
cout << "Doing a grid search" << endl;
matrix<double> best_result(2,1);
best_result = 0;
double best_gamma, best_nu;
set_all_elements(best_result, 0);
for (long col = 0; col < params.nc(); ++col)
{
const double gamma = params(0, col);
const double nu = params(1, col);
// Grid search is a very simple brute force method. Below we try out the BOBYQA algorithm.
// It is a routine that performs optimization of a function in the absence of derivatives.
trainer.set_kernel(kernel_type(gamma));
trainer.set_nu(nu);
matrix<double> result = cross_validate_trainer(trainer, samples, labels, 10);
cout << "gamma: " << setw(11) << gamma << " nu: " << setw(11) << nu << " cross validation accuracy: " << result;
if (sum(result) > sum(best_result))
{
best_result = result;
best_gamma = gamma;
best_nu = nu;
}
}
cout << "\n\n Try the BOBYQA algorithm" << endl;
cout << "\n best result of grid search: " << sum(best_result) << endl;
cout << " best gamma: " << best_gamma << " best nu: " << best_nu << endl;
// We need to supply a starting point for the optimization. Here we are using the best
// result of the grid search. Generally, you want to try and give a reasonable starting
// point due to the possibility of the optimization getting stuck in a local minima.
params.set_size(2,1);
params = best_gamma, // initial gamma
best_nu; // initial nu
// now lets try out the BOBYQA algorithm
// We also need to supply lower and upper bounds for the search.
matrix<double> lower_bound(2,1), upper_bound(2,1);
lower_bound = 1e-7, // smallest allowed gamma
1e-7; // smallest allowed nu
upper_bound = 100, // largest allowed gamma
max_nu; // largest allowed nu
cout << "\n\n Try the BOBYQA algorithm" << endl;
params.set_size(2,1);
params = best_gamma, // initial gamma
best_nu; // initial nu
// For the gamma and nu SVM parameters it is generally a good idea to search
// in log space. So I'm just converting them into log space here before
// we start the optimization.
params = log(params);
lower_bound = log(lower_bound);
upper_bound = log(upper_bound);
matrix<double> lower_bound(2,1), upper_bound(2,1);
lower_bound = 1e-7, // smallest allowed gamma
1e-7; // smallest allowed nu
upper_bound = 100, // largest allowed gamma
max_nu; // largest allowed nu
// Finally, ask BOBYQA to look for the best set of parameters. Note that we are using the
// cross validation function object defined at the top of the file.
double best_score = find_max_bobyqa(
cross_validation_objective(samples, labels), // Function to maximize
params, // starting point
params.size()*2 + 1, // See BOBYQA docs, generally size*2+1 is a good setting for this
lower_bound, // lower bound
upper_bound, // upper bound
min(upper_bound-lower_bound)/10, // search radius
0.01, // desired accuracy
100 // max number of allowable calls to cross_validation_objective()
);
params = log(params);
lower_bound = log(lower_bound);
upper_bound = log(upper_bound);
// Don't forget to convert back from log scale to normal scale
params = exp(params);
double best_score = find_max_bobyqa(cross_validation_objective(samples, labels),
params,
params.size()*2 + 1,
lower_bound,
upper_bound,
min(upper_bound-lower_bound)/10,
0.01,
100
);
cout << " best result of BOBYQA: " << best_score << endl;
cout << " best gamma: " << params(0) << " best nu: " << params(1) << endl;
params = exp(params);
cout << " best result of BOBYQA: " << best_score << endl;
cout << " best gamma: " << params(0) << " best nu: " << params(1) << endl;
// Also note that the find_max_bobyqa() function only works for optimization problems
// with 2 variables or more. If you have only a single variable then you should use
// the find_max_single_variable() function.
}
catch (exception& e)
{
cout << e.what() << endl;
}
}
examples/rvm_ex.cpp
View file @ 2f4217c7
......@@ -100,9 +100,9 @@ int main()
rvm_trainer<kernel_type> trainer;
// Now we loop over some different gamma values to see how good they are. Note
// that this is just a simple brute force way to try out a few possible parameter
// choices. You may want to investigate more sophisticated strategies for determining
// good parameter choices.
// that this is a very simple way to try out a few possible parameter choices. You
// should look at the model_selection_ex.cpp program for examples of more sophisticated
// strategies for determining good parameter choices.
cout << "doing cross validation" << endl;
for (double gamma = 0.00001; gamma <= 1; gamma += 0.1)
{
......
examples/svm_ex.cpp
View file @ 2f4217c7
......@@ -102,9 +102,9 @@ int main()
svm_nu_trainer<kernel_type> trainer;
// Now we loop over some different nu and gamma values to see how good they are. Note
// that this is just a simple brute force way to try out a few possible parameter
// choices. You may want to investigate more sophisticated strategies for determining
// good parameter choices.
// that this is a very simple way to try out a few possible parameter choices. You
// should look at the model_selection_ex.cpp program for examples of more sophisticated
// strategies for determining good parameter choices.
cout << "doing cross validation" << endl;
for (double gamma = 0.00001; gamma <= 1; gamma += 0.1)
{
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment