potok/opencv
1
0
Fork 0
mirror of https://github.com/opencv/opencv.git synced 2026-07-31 08:13:04 +04:00
Code Packages Releases Wiki Activity

propagated some more fixes from 2.3 branch to the trunk

Browse Source
This commit is contained in:
Vadim Pisarevsky
2011-06-29 22:06:42 +00:00
parent dacd265424
commit b204e73d9a
54 changed files with 9120 additions and 3049 deletions
Download Patch File Download Diff File Expand all files Collapse all files
modules/ml/doc/boosting.rst
+142 -119
View File
@@ -3,13 +3,14 @@
Boosting
========
.. highlight:: cpp
A common machine learning task is supervised learning. In supervised learning, the goal is to learn the functional relationship
:math:`F: y = F(x)` between the input
:math:`x` and the output
:math:`y` . Predicting the qualitative output is called classification, while predicting the quantitative output is called regression.
:math:`y` . Predicting the qualitative output is called *classification*, while predicting the quantitative output is called *regression*.
Boosting is a powerful learning concept that provides a solution to the supervised classification learning task. It combines the performance of many "weak" classifiers to produce a powerful 'committee'
:ref:`[HTF01] <HTF01>` . A weak classifier is only required to be better than chance, and thus can be very simple and computationally inexpensive. However, many of them smartly combine results to a strong classifier that often outperforms most "monolithic" strong classifiers such as SVMs and Neural Networks.??
Boosting is a powerful learning concept that provides a solution to the supervised classification learning task. It combines the performance of many "weak" classifiers to produce a powerful committee [HTF01]_. A weak classifier is only required to be better than chance, and thus can be very simple and computationally inexpensive. However, many of them smartly combine results to a strong classifier that often outperforms most "monolithic" strong classifiers such as SVMs and Neural Networks.
Decision trees are the most popular weak classifiers used in boosting schemes. Often the simplest decision trees with only a single split node per tree (called ``stumps`` ) are sufficient.
@@ -19,15 +20,16 @@ The boosted model is based on
:math:`x_i \in{R^K}` and
:math:`y_i \in{-1, +1}` .
:math:`x_i` is a
:math:`K` -component vector. Each component encodes a feature relevant for the learning task at hand. The desired two-class output is encoded as -1 and +1.
:math:`K` -component vector. Each component encodes a feature relevant to the learning task at hand. The desired two-class output is encoded as -1 and +1.
Different variants of boosting are known as Discrete Adaboost, Real AdaBoost, LogitBoost, and Gentle AdaBoost
:ref:`[FHT98] <FHT98>` . All of them are very similar in their overall structure. Therefore, this chapter focuses only on the standard two-class Discrete AdaBoost algorithm as shown in the box below??. Each sample is initially assigned the same weight (step 2). Then, a weak classifier
Different variants of boosting are known as Discrete Adaboost, Real AdaBoost, LogitBoost, and Gentle AdaBoost [FHT98]_. All of them are very similar in their overall structure. Therefore, this chapter focuses only on the standard two-class Discrete AdaBoost algorithm, outlined below. Initially the same weight is assigned to each sample (step 2). Then, a weak classifier
:math:`f_{m(x)}` is trained on the weighted training data (step 3a). Its weighted training error and scaling factor
:math:`c_m` is computed (step 3b). The weights are increased for training samples that have been misclassified (step 3c). All weights are then normalized, and the process of finding the next weak classifier continues for another
:math:`M` -1 times. The final classifier
:math:`F(x)` is the sign of the weighted sum over the individual weak classifiers (step 4).
**Two-class Discrete AdaBoost Algorithm**
#.
Set
:math:`N` examples
@@ -39,175 +41,196 @@ Different variants of boosting are known as Discrete Adaboost, Real AdaBoost, Lo
:math:`w_i = 1/N, i = 1,...,N` .
#.
Repeat for
:math:`m` =
:math:`1,2,...,M` :
Repeat for :math:`m = 1,2,...,M` :
#.
Fit the classifier
:math:`f_m(x) \in{-1,1}` , using weights
:math:`w_i` on the training data.
3.1. Fit the classifier :math:`f_m(x) \in{-1,1}`, using weights :math:`w_i` on the training data.
#.
Compute
:math:`err_m = E_w [1_{(y =\neq f_m(x))}], c_m = log((1 - err_m)/err_m)` .
3.2. Compute :math:`err_m = E_w [1_{(y \neq f_m(x))}], c_m = log((1 - err_m)/err_m)` .
#.
Set
:math:`w_i \Leftarrow w_i exp[c_m 1_{(y_i \neq f_m(x_i))}], i = 1,2,...,N,` and renormalize so that
:math:`\Sigma i w_i = 1` .
3.3. Set :math:`w_i \Leftarrow w_i exp[c_m 1_{(y_i \neq f_m(x_i))}], i = 1,2,...,N,` and renormalize so that :math:`\Sigma i w_i = 1` .
#.
Output the classifier sign
:math:`[\Sigma m = 1M c_m f_m(x)]` .
Two-class Discrete AdaBoost Algorithm: Training (steps 1 to 3) and Evaluation (step 4)??you need to revise this section. what is this? a title for the image that is missing?
#. Classify new samples *x* using the formula: :math:`\textrm{sign} (\Sigma m = 1M c_m f_m(x))` .
**NOTE:**
Similar to the classical boosting methods, the current implementation supports two-class classifiers only. For M
:math:`>` two classes, there is the
**AdaBoost.MH**
algorithm (described in
:ref:`[FHT98] <FHT98>` ) that reduces the problem to the two-class problem, yet with a much larger training set.
.. note:: Similar to the classical boosting methods, the current implementation supports two-class classifiers only. For ``M > 2`` classes, there is the **AdaBoost.MH** algorithm (described in [FHT98]_) that reduces the problem to the two-class problem, yet with a much larger training set.
To reduce computation time for boosted models without substantially losing accuracy, the influence trimming technique may be employed. As the training algorithm proceeds and the number of trees in the ensemble is increased, a larger number of the training samples are classified correctly and with increasing confidence, thereby those samples receive smaller weights on the subsequent iterations. Examples with a very low relative weight have a small impact on the weak classifier training. Thus, such examples may be excluded during the weak classifier training without having much effect on the induced classifier. This process is controlled with the ``weight_trim_rate`` parameter. Only examples with the summary fraction ``weight_trim_rate`` of the total weight mass are used in the weak classifier training. Note that the weights for
To reduce computation time for boosted models without substantially losing accuracy, the influence trimming technique can be employed. As the training algorithm proceeds and the number of trees in the ensemble is increased, a larger number of the training samples are classified correctly and with increasing confidence, thereby those samples receive smaller weights on the subsequent iterations. Examples with a very low relative weight have a small impact on the weak classifier training. Thus, such examples may be excluded during the weak classifier training without having much effect on the induced classifier. This process is controlled with the ``weight_trim_rate`` parameter. Only examples with the summary fraction ``weight_trim_rate`` of the total weight mass are used in the weak classifier training. Note that the weights for
**all**
training examples are recomputed at each training iteration. Examples deleted at a particular iteration may be used again for learning some of the weak classifiers further
:ref:`[FHT98] <FHT98>` .
training examples are recomputed at each training iteration. Examples deleted at a particular iteration may be used again for learning some of the weak classifiers further [FHT98]_.
.. _HTF01:??what is this meant to be? it doesn't work
.. [HTF01] Hastie, T., Tibshirani, R., Friedman, J. H. *The Elements of Statistical Learning: Data Mining, Inference, and Prediction*. Springer Series in Statistics. 2001.
[HTF01] Hastie, T., Tibshirani, R., Friedman, J. H. *The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Series in Statistics*. 2001.
.. _FHT98:??the same comment
[FHT98] Friedman, J. H., Hastie, T. and Tibshirani, R. Additive Logistic Regression: a Statistical View of Boosting. Technical Report, Dept. of Statistics*, Stanford University, 1998.
.. index:: CvBoostParams
.. _CvBoostParams:
.. [FHT98] Friedman, J. H., Hastie, T. and Tibshirani, R. *Additive Logistic Regression: a Statistical View of Boosting*. Technical Report, Dept. of Statistics, Stanford University, 1998.
CvBoostParams
-------------
.. c:type:: CvBoostParams
.. ocv:class:: CvBoostParams
Boosting training parameters ::
Boosting training parameters.
struct CvBoostParams : public CvDTreeParams
The structure is derived from :ocv:class:`CvDTreeParams` but not all of the decision tree parameters are supported. In particular, cross-validation is not supported.
All parameters are public. You can initialize them by a constructor and then override some of them directly if you want.
CvBoostParams::CvBoostParams
----------------------------
The constructors.
.. ocv:function:: CvBoostParams::CvBoostParams()
.. ocv:function:: CvBoostParams::CvBoostParams( int boost_type, int weak_count, double weight_trim_rate, int max_depth, bool use_surrogates, const float* priors )
:param boost_type: Type of the boosting algorithm. Possible values are:
* **CvBoost::DISCRETE** Discrete AbaBoost.
* **CvBoost::REAL** Real AdaBoost. It is a technique that utilizes confidence-rated predictions and works well with categorical data.
* **CvBoost::LOGIT** LogitBoost. It can produce good regression fits.
* **CvBoost::GENTLE** Gentle AdaBoost. It puts less weight on outlier data points and for that reason is often good with regression data.
Gentle AdaBoost and Real AdaBoost are often the preferable choices.
:param weak_count: The number of weak classifiers.
:param weight_trim_rate: A threshold between 0 and 1 used to save computational time. Samples with summary weight :math:`\leq 1 - weight\_trim\_rate` do not participate in the *next* iteration of training. Set this parameter to 0 to turn off this functionality.
See :ocv:func:`CvDTreeParams::CvDTreeParams` for description of other parameters.
Also there is one structure member that you can set directly:
.. ocv:member:: int split_criteria
Splitting criteria used to choose optimal splits during a weak tree construction. Possible values are:
* **CvBoost::DEFAULT** Use the default for the particular boosting method, see below.
* **CvBoost::GINI** Use Gini index. This is default option for Real AdaBoost; may be also used for Discrete AdaBoost.
* **CvBoost::MISCLASS** Use misclassification rate. This is default option for Discrete AdaBoost; may be also used for Real AdaBoost.
* **CvBoost::SQERR** Use least squares criteria. This is default and the only option for LogitBoost and Gentle AdaBoost.
Default parameters are:
::
CvBoostParams::CvBoostParams()
{
int boost_type;
int weak_count;
int split_criteria;
double weight_trim_rate;
CvBoostParams();
CvBoostParams( int boost_type, int weak_count, double weight_trim_rate,
int max_depth, bool use_surrogates, const float* priors );
};
The structure is derived from
:ref:`CvDTreeParams` but not all of the decision tree parameters are supported. In particular, cross-validation is not supported.
.. index:: CvBoostTree
.. _CvBoostTree:
boost_type = CvBoost::REAL;
weak_count = 100;
weight_trim_rate = 0.95;
cv_folds = 0;
max_depth = 1;
}
CvBoostTree
-----------
.. c:type:: CvBoostTree
.. ocv:class:: CvBoostTree
Weak tree classifier ::
The weak tree classifier, a component of the boosted tree classifier :ocv:class:`CvBoost`, is a derivative of :ocv:class:`CvDTree`. Normally, there is no need to use the weak classifiers directly. However, they can be accessed as elements of the sequence :ocv:member:`CvBoost::weak`, retrieved by :ocv:func:`CvBoost::get_weak_predictors`.
class CvBoostTree: public CvDTree
{
public:
CvBoostTree();
virtual ~CvBoostTree();
virtual bool train( CvDTreeTrainData* _train_data,
const Mat& subsample_idx, CvBoost* ensemble );
virtual void scale( double s );
virtual void read( CvFileStorage* fs, CvFileNode* node,
CvBoost* ensemble, CvDTreeTrainData* _data );
virtual void clear();
protected:
...
CvBoost* ensemble;
};
The weak classifier, a component of the boosted tree classifier
:ref:`CvBoost` , is a derivative of
:ref:`CvDTree` . Normally, there is no need to use the weak classifiers directly. However, they can be accessed as elements of the sequence ``CvBoost::weak`` , retrieved by ``CvBoost::get_weak_predictors`` .
**Note:**
In case of LogitBoost and Gentle AdaBoost, each weak predictor is a regression tree, rather than a classification tree. Even in case of Discrete AdaBoost and Real AdaBoost, the ``CvBoostTree::predict`` return value ( ``CvDTreeNode::value`` ) is not an output class label. A negative value "votes" for class
#
0, a positive - for class
#
1. The votes are weighted. The weight of each individual tree may be increased or decreased using the method ``CvBoostTree::scale`` .
.. index:: CvBoost
.. note:: In case of LogitBoost and Gentle AdaBoost, each weak predictor is a regression tree, rather than a classification tree. Even in case of Discrete AdaBoost and Real AdaBoost, the ``CvBoostTree::predict`` return value (:ocv:member:`CvDTreeNode::value`) is not an output class label. A negative value "votes" for class #0, a positive value - for class #1. The votes are weighted. The weight of each individual tree may be increased or decreased using the method ``CvBoostTree::scale``.
CvBoost
-------
.. ocv:class:: CvBoost
Boosted tree classifier, derived from :ocv:class:`CvStatModel`
Boosted tree classifier derived from :ocv:class:`CvStatModel`.
.. index:: CvBoost::train
CvBoost::CvBoost
----------------
Default and training constructors.
.. _CvBoost::train:
.. ocv:function:: CvBoost::CvBoost()
.. ocv:function:: CvBoost::CvBoost( const Mat& trainData, int tflag, const Mat& responses, const Mat& varIdx=Mat(), const Mat& sampleIdx=Mat(), const Mat& varType=Mat(), const Mat& missingDataMask=Mat(), CvBoostParams params=CvBoostParams() )
.. ocv:function::CvBoost::CvBoost( const CvMat* trainData, int tflag, const CvMat* responses, const CvMat* varIdx=0, const CvMat* sampleIdx=0, const CvMat* varType=0, const CvMat* missingDataMask=0, CvBoostParams params=CvBoostParams() )
.. ocv:pyfunction:: cv2.Boost(trainData, tflag, responses[, varIdx[, sampleIdx[, varType[, missingDataMask[, params]]]]]) -> <Boost object>
The constructors follow conventions of :ocv:func:`CvStatModel::CvStatModel`. See :ocv:func:`CvStatModel::train` for parameters descriptions.
CvBoost::train
--------------
.. ocv:function:: bool CvBoost::train( const Mat& _train_data, int _tflag, const Mat& _responses, const Mat& _var_idx=Mat(), const Mat& _sample_idx=Mat(), const Mat& _var_type=Mat(), const Mat& _missing_mask=Mat(), CvBoostParams params=CvBoostParams(), bool update=false )
Trains a boosted tree classifier.
Trains a boosted tree classifier.
.. ocv:function:: bool CvBoost::train( const Mat& trainData, int tflag, const Mat& responses, const Mat& varIdx=Mat(), const Mat& sampleIdx=Mat(), const Mat& varType=Mat(), const Mat& missingDataMask=Mat(), CvBoostParams params=CvBoostParams(), bool update=false )
The train method follows the common template. The last parameter ``update`` specifies whether the classifier needs to be updated (the new weak tree classifiers added to the existing ensemble) or the classifier needs to be rebuilt from scratch. The responses must be categorical, which means that boosted trees cannot be built for regression, and there should be two classes.
.. ocv:function::bool CvBoost::train( const CvMat* trainData, int tflag, const CvMat* responses, const CvMat* varIdx=0, const CvMat* sampleIdx=0, const CvMat* varType=0, const CvMat* missingDataMask=0, CvBoostParams params=CvBoostParams(), bool update=false )
.. index:: CvBoost::predict
.. ocv:function::bool CvBoost::train( CvMLData* data, CvBoostParams params=CvBoostParams(), bool update=false )
.. _CvBoost::predict:
.. ocv:pyfunction:: cv2.Boost.train(trainData, tflag, responses[, varIdx[, sampleIdx[, varType[, missingDataMask[, params[, update]]]]]]) -> retval
:param update: Specifies whether the classifier needs to be updated (``true``, the new weak tree classifiers added to the existing ensemble) or the classifier needs to be rebuilt from scratch (``false``).
The train method follows the common template of :ocv:func:`CvStatModel::train`. The responses must be categorical, which means that boosted trees cannot be built for regression, and there should be two classes.
CvBoost::predict
----------------
.. ocv:function:: float CvBoost::predict( const Mat& sample, const Mat& missing=Mat(), const Range& slice=Range::all(), bool rawMode=false, bool returnSum=false ) const
Predicts a response for an input sample.
Predicts a response for an input sample.
.. ocv:function:: float CvBoost::predict( const Mat& sample, const Mat& missing=Mat(), const Range& slice=Range::all(), bool rawMode=false, bool returnSum=false ) const
The method ``CvBoost::predict`` runs the sample through the trees in the ensemble and returns the output class label based on the weighted voting.
.. ocv:function::float CvBoost::predict( const CvMat* sample, const CvMat* missing=0, CvMat* weak_responses=0, CvSlice slice=CV_WHOLE_SEQ, bool raw_mode=false, bool return_sum=false ) const
.. index:: CvBoost::prune
.. ocv:pyfunction:: cv2.Boost.predict(sample[, missing[, slice[, rawMode[, returnSum]]]]) -> retval
.. _CvBoost::prune:
:param sample: Input sample.
:param missing: Optional mask of missing measurements. To handle missing measurements, the weak classifiers must include surrogate splits (see ``CvDTreeParams::use_surrogates``).
:param weak_responses: Optional output parameter, a floating-point vector with responses of each individual weak classifier. The number of elements in the vector must be equal to the slice length.
:param slice: Continuous subset of the sequence of weak classifiers to be used for prediction. By default, all the weak classifiers are used.
:param raw_mode: Normally, it should be set to ``false``.
:param return_sum: If ``true`` then return sum of votes instead of the class label.
The method runs the sample through the trees in the ensemble and returns the output class label based on the weighted voting.
CvBoost::prune
--------------
.. ocv:function:: void CvBoost::prune( CvSlice slice )
Removes the specified weak classifiers.
Removes the specified weak classifiers.
.. ocv:function::void CvBoost::prune( CvSlice slice )
.. ocv:pyfunction:: cv2.Boost.prune(slice) -> None
:param slice: Continuous subset of the sequence of weak classifiers to be removed.
The method removes the specified weak classifiers from the sequence.
**Note:**
.. note:: Do not confuse this method with the pruning of individual decision trees, which is currently not supported.
Do not confuse this method with the pruning of individual decision trees, which is currently not supported.
.. index:: CvBoost::get_weak_predictors
CvBoost::calc_error
-------------------
Returns error of the boosted tree classifier.
.. ocv:function::float CvBoost::calc_error( CvMLData* _data, int type , std::vector<float> *resp = 0 )
The method is identical to :ocv:func:`CvDTree::calc_error` but uses the boosted tree classifier as predictor.
.. _CvBoost::get_weak_predictors:
CvBoost::get_weak_predictors
----------------------------
.. ocv:function:: CvSeq* CvBoost::get_weak_predictors()
Returns the sequence of weak tree classifiers.
Returns the sequence of weak tree classifiers.
.. ocv:function::CvSeq* CvBoost::get_weak_predictors()
The method returns the sequence of weak classifiers. Each element of the sequence is a pointer to the ``CvBoostTree`` class or, probably, to some of its derivatives.
The method returns the sequence of weak classifiers. Each element of the sequence is a pointer to the :ocv:class:`CvBoostTree` class or to some of its derivatives.
CvBoost::get_params
-------------------
Returns current parameters of the boosted tree classifier.
.. ocv:function:: const CvBoostParams& CvBoost::get_params() const
CvBoost::get_data
-----------------
Returns used train data of the boosted tree classifier.
.. ocv:function::const CvDTreeTrainData* CvBoost::get_data() const
modules/ml/doc/decision_trees.rst
+182 -285
View File
@@ -1,15 +1,14 @@
Decision Trees
==============
The ML classes discussed in this section implement Classification and Regression Tree algorithms described in `[Breiman84] <#paper_Breiman84>`_
.
The ML classes discussed in this section implement Classification and Regression Tree algorithms described in [Breiman84]_.
The class
:ref:`CvDTree` represents a single decision tree that may be used alone, or as a base class in tree ensembles (see
:ocv:class:`CvDTree` represents a single decision tree that may be used alone or as a base class in tree ensembles (see
:ref:`Boosting` and
:ref:`Random Trees` ).
A decision tree is a binary tree (tree where each non-leaf node has exactly two child nodes). It can be used either for classification or for regression. For classification, each tree leaf is marked with a class label; multiple leafs may have the same label. For regression, a constant is also assigned to each tree leaf, so the approximation function is piecewise constant.
A decision tree is a binary tree (tree where each non-leaf node has two child nodes). It can be used either for classification or for regression. For classification, each tree leaf is marked with a class label; multiple leaves may have the same label. For regression, a constant is also assigned to each tree leaf, so the approximation function is piecewise constant.
Predicting with Decision Trees
------------------------------
@@ -22,7 +21,7 @@ value of a certain variable whose index is stored in the observed
node. The following variables are possible:
*
**Ordered variables.** The variable value is compared with a threshold that is also stored in the node). If the value is less than the threshold, the procedure goes to the left. Otherwise, it goes to the right. For example, if the weight is less than 1 kilogram, the procedure goes to the left, else to the right.
**Ordered variables.** The variable value is compared with a threshold that is also stored in the node. If the value is less than the threshold, the procedure goes to the left. Otherwise, it goes to the right. For example, if the weight is less than 1 kilogram, the procedure goes to the left, else to the right.
*
**Categorical variables.** A discrete variable value is tested to see whether it belongs to a certain subset of values (also stored in the node) from a limited set of values the variable could take. If it does, the procedure goes to the left. Otherwise, it goes to the right. For example, if the color is green or red, go to the left, else to the right.
@@ -36,7 +35,7 @@ Sometimes, certain features of the input vector are missed (for example, in the
Training Decision Trees
-----------------------
The tree is built recursively, starting from the root node. All training data (feature vectors and responses) is used to split the root node. In each node the optimum decision rule (the best "primary" split) is found based on some criteria. In ML, ``gini`` "purity" criteria are used for classification, and sum of squared errors is used for regression. Then, if necessary, the surrogate splits are found. They resemble the results of the primary split on the training data. All the data is divided using the primary and the surrogate splits (like it is done in the prediction procedure) between the left and the right child node. Then, the procedure recursively splits both left and right nodes. At each node the recursive procedure may stop (that is, stop splitting the node further) in one of the following cases:
The tree is built recursively, starting from the root node. All training data (feature vectors and responses) is used to split the root node. In each node the optimum decision rule (the best "primary" split) is found based on some criteria. In machine learning, ``gini`` "purity" criteria are used for classification, and sum of squared errors is used for regression. Then, if necessary, the surrogate splits are found. They resemble the results of the primary split on the training data. All the data is divided using the primary and the surrogate splits (like it is done in the prediction procedure) between the left and the right child node. Then, the procedure recursively splits both left and right nodes. At each node the recursive procedure may stop (that is, stop splitting the node further) in one of the following cases:
* Depth of the constructed tree branch has reached the specified maximum value.
@@ -46,7 +45,7 @@ The tree is built recursively, starting from the root node. All training data (f
* The best found split does not give any noticeable improvement compared to a random choice.
When the tree is built, it may be pruned using a cross-validation procedure, if necessary. That is, some branches of the tree that may lead to the model overfitting are cut off. Normally, this procedure is only applied to standalone decision trees. Tree ensembles usually build trees that are small enough and use their own protection schemes against overfitting.
When the tree is built, it may be pruned using a cross-validation procedure, if necessary. That is, some branches of the tree that may lead to the model overfitting are cut off. Normally, this procedure is only applied to standalone decision trees. Usually tree ensembles build trees that are small enough and use their own protection schemes against overfitting.
Variable Importance
-------------------
@@ -55,106 +54,130 @@ Besides the prediction that is an obvious use of decision trees, the tree can be
Importance of each variable is computed over all the splits on this variable in the tree, primary and surrogate ones. Thus, to compute variable importance correctly, the surrogate splits must be enabled in the training parameters, even if there is no missing data.
[Breiman84] Breiman, L., Friedman, J. Olshen, R. and Stone, C. (1984), *Classification and Regression Trees*, Wadsworth.
.. index:: CvDTreeSplit
.. _CvDTreeSplit:
CvDTreeSplit
------------
.. c:type:: CvDTreeSplit
Decision tree node split ::
struct CvDTreeSplit
{
int var_idx;
int inversed;
float quality;
CvDTreeSplit* next;
union
{
int subset[2];
struct
{
float c;
int split_point;
}
ord;
};
};
.. ocv:class:: CvDTreeSplit
.. index:: CvDTreeNode
The structure represents a possible decision tree node split. It has public members:
.. _CvDTreeNode:
.. ocv:member:: int var_idx
Index of variable on which the split is created.
.. ocv:member:: int inversed
If it is not null then inverse split rule is used that is left and right branches are exchanged in the rule expressions below.
.. ocv:member:: float quality
The split quality, a positive number. It is used to choose the best primary split, then to choose and sort the surrogate splits. After the tree is constructed, it is also used to compute variable importance.
.. ocv:member:: CvDTreeSplit* next
Pointer to the next split in the node list of splits.
.. ocv:member:: int subset[2]
Bit array indicating the value subset in case of split on a categorical variable. The rule is:
::
if var_value in subset
then next_node <- left
else next_node <- right
.. ocv:member:: float ord.c
The threshold value in case of split on an ordered variable. The rule is:
::
if var_value < c
then next_node<-left
else next_node<-right
.. ocv:member:: int ord.split_point
Used internally by the training algorithm.
CvDTreeNode
-----------
.. c:type:: CvDTreeNode
.. ocv:class:: CvDTreeNode
Decision tree node ::
struct CvDTreeNode
{
int class_idx;
int Tn;
double value;
The structure represents a node in a decision tree. It has public members:
CvDTreeNode* parent;
CvDTreeNode* left;
CvDTreeNode* right;
.. ocv:member:: int class_idx
CvDTreeSplit* split;
Class index normalized to 0..class_count-1 range and assigned to the node. It is used internally in classification trees and tree ensembles.
int sample_count;
int depth;
...
};
.. ocv:member:: int Tn
Tree index in a ordered sequence of pruned trees. The indices are used during and after the pruning procedure. The root node has the maximum value ``Tn`` of the whole tree, child nodes have ``Tn`` less than or equal to the parent's ``Tn``, and nodes with :math:`Tn \leq CvDTree::pruned\_tree\_idx` are not used at prediction stage (the corresponding branches are considered as cut-off), even if they have not been physically deleted from the tree at the pruning stage.
.. ocv:member:: double value
Value at the node: a class label in case of classification or estimated function value in case of regression.
.. ocv:member:: CvDTreeNode* parent
Pointer to the parent node.
.. ocv:mebmer:: CvDTreeNode* left
Pointer to the left child node.
.. ocv:member:: CvDTreeNode* right
Pointer to the right child node.
.. ocv:member:: CvDTreeSplit* split
Pointer to the first (primary) split in the node list of splits.
.. ocv:mebmer:: int sample_count
The number of samples that fall into the node at the training stage. It is used to resolve the difficult cases - when the variable for the primary split is missing and all the variables for other surrogate splits are missing too. In this case the sample is directed to the left if ``left->sample_count > right->sample_count`` and to the right otherwise.
.. ocv:member:: int depth
Depth of the node. The root node depth is 0, the child nodes depth is the parent's depth + 1.
Other numerous fields of ``CvDTreeNode`` are used internally at the training stage.
.. index:: CvDTreeParams
.. _CvDTreeParams:
CvDTreeParams
-------------
.. c:type:: CvDTreeParams
Decision tree training parameters.
.. ocv:class:: CvDTreeParams
The structure contains all the decision tree training parameters. You can initialize it by default constructor and then override any parameters directly before training, or the structure may be fully initialized using the advanced variant of the constructor.
.. index:: CvDTreeParams::CvDTreeParams
.. _CvDTreeParams::CvDTreeParams
CvDTreeParams::CvDTreeParams
----------------------------
The constructors.
.. ocv:function:: CvDTreeParams::CvDTreeParams()
.. ocv:function:: CvDTreeParams( int max_depth, int min_sample_count, float regression_accuracy, bool use_surrogates, int max_categories, int cv_folds, bool use_1se_rule, bool truncate_pruned_tree, const float* priors )
:param max_depth: The maximum number of levels in a tree. The depth of a constructed tree may be smaller due to other termination criterias or pruning of the tree.
.. ocv:function:: CvDTreeParams::CvDTreeParams( int max_depth, int min_sample_count, float regression_accuracy, bool use_surrogates, int max_categories, int cv_folds, bool use_1se_rule, bool truncate_pruned_tree, const float* priors )
:param max_depth: The maximum possible depth of the tree. That is the training algorithms attempts to split a node while its depth is less than ``max_depth``. The actual depth may be smaller if the other termination criteria are met (see the outline of the training procedure in the beginning of the section), and/or if the tree is pruned.
:param min_sample_count: If the number of samples in a node is less than this parameter then the node will not be splitted.
:param regression_accuracy: Termination criteria for regression trees. If all absolute differences between an estimated value in a node and values of train samples in this node are less than this parameter then the node will not be splitted.
:param use_surrogates: If true then surrogate splits will be built. These splits allow to work with missing data.
:param use_surrogates: If true then surrogate splits will be built. These splits allow to work with missing data and compute variable importance correctly.
:param max_categories: Cluster possible values of a categorical variable into ``K`` :math:`\leq` ``max_categories`` clusters to find a suboptimal split. The clustering is applied only in n>2-class classification problems for categorical variables with ``N > max_categories`` possible values. See the Learning OpenCV book (page 489) for more detailed explanation.
:param max_categories: Cluster possible values of a categorical variable into ``K`` :math:`\leq` ``max_categories`` clusters to find a suboptimal split. If a discrete variable, on which the training procedure tries to make a split, takes more than ``max_categories`` values, the precise best subset estimation may take a very long time because the algorithm is exponential. Instead, many decision trees engines (including ML) try to find sub-optimal split in this case by clustering all the samples into ``max_categories`` clusters that is some categories are merged together. The clustering is applied only in ``n``>2-class classification problems for categorical variables with ``N > max_categories`` possible values. In case of regression and 2-class classification the optimal split can be found efficiently without employing clustering, thus the parameter is not used in these cases.
:param cv_folds: If ``cv_folds > 1`` then prune a tree with ``K``-fold cross-validation where ``K`` is equal to ``cv_folds``.
:param use_1se_rule: If true then a pruning will be harsher. This will make a tree more compact but a bit less accurate.
:param use_1se_rule: If true then a pruning will be harsher. This will make a tree more compact and more resistant to the training data noise but a bit less accurate.
:param truncate_pruned_tree: If true then pruned branches are removed completely from the tree. Otherwise they are retained and it is possible to get the unpruned tree or prune the tree differently by changing ``CvDTree::pruned_tree_idx`` parameter.
:param truncate_pruned_tree: If true then pruned branches are physically removed from the tree. Otherwise they are retained and it is possible to get results from the original unpruned (or pruned less aggressively) tree by decreasing ``CvDTree::pruned_tree_idx`` parameter.
:param priors: Weights of prediction categories which determine relative weights that you give to misclassification. That is, if the weight of the first category is 1 and the weight of the second category is 10, then each mistake in predicting the second category is equivalent to making 10 mistakes in predicting the first category.
:param priors: The array of a priori class probabilities, sorted by the class label value. The parameter can be used to tune the decision tree preferences toward a certain class. For example, if you want to detect some rare anomaly occurrence, the training base will likely contain much more normal cases than anomalies, so a very good classification performance will be achieved just by considering every case as normal. To avoid this, the priors can be specified, where the anomaly probability is artificially increased (up to 0.5 or even greater), so the weight of the misclassified anomalies becomes much bigger, and the tree is adjusted properly. You can also think about this parameter as weights of prediction categories which determine relative weights that you give to misclassification. That is, if the weight of the first category is 1 and the weight of the second category is 10, then each mistake in predicting the second category is equivalent to making 10 mistakes in predicting the first category.
The default constructor initializes all the parameters with the default values tuned for the standalone classification tree:
@@ -166,261 +189,135 @@ The default constructor initializes all the parameters with the default values t
{}
.. index:: CvDTreeTrainData
.. _CvDTreeTrainData:
CvDTreeTrainData
----------------
.. c:type:: CvDTreeTrainData
.. ocv:class:: CvDTreeTrainData
Decision tree training data and shared data for tree ensembles ::
Decision tree training data and shared data for tree ensembles. The structure is mostly used internally for storing both standalone trees and tree ensembles efficiently. Basically, it contains the following types of information:
struct CvDTreeTrainData
{
CvDTreeTrainData();
CvDTreeTrainData( const Mat& _train_data, int _tflag,
const Mat& _responses, const Mat& _var_idx=Mat(),
const Mat& _sample_idx=Mat(), const Mat& _var_type=Mat(),
const Mat& _missing_mask=Mat(),
const CvDTreeParams& _params=CvDTreeParams(),
bool _shared=false, bool _add_labels=false );
virtual ~CvDTreeTrainData();
#. Training parameters, an instance of :ocv:class:`CvDTreeParams`.
virtual void set_data( const Mat& _train_data, int _tflag,
const Mat& _responses, const Mat& _var_idx=Mat(),
const Mat& _sample_idx=Mat(), const Mat& _var_type=Mat(),
const Mat& _missing_mask=Mat(),
const CvDTreeParams& _params=CvDTreeParams(),
bool _shared=false, bool _add_labels=false,
bool _update_data=false );
virtual void get_vectors( const Mat& _subsample_idx,
float* values, uchar* missing, float* responses,
bool get_class_idx=false );
virtual CvDTreeNode* subsample_data( const Mat& _subsample_idx );
virtual void write_params( CvFileStorage* fs );
virtual void read_params( CvFileStorage* fs, CvFileNode* node );
// release all the data
virtual void clear();
int get_num_classes() const;
int get_var_type(int vi) const;
int get_work_var_count() const;
virtual int* get_class_labels( CvDTreeNode* n );
virtual float* get_ord_responses( CvDTreeNode* n );
virtual int* get_labels( CvDTreeNode* n );
virtual int* get_cat_var_data( CvDTreeNode* n, int vi );
virtual CvPair32s32f* get_ord_var_data( CvDTreeNode* n, int vi );
virtual int get_child_buf_idx( CvDTreeNode* n );
////////////////////////////////////
virtual bool set_params( const CvDTreeParams& params );
virtual CvDTreeNode* new_node( CvDTreeNode* parent, int count,
int storage_idx, int offset );
virtual CvDTreeSplit* new_split_ord( int vi, float cmp_val,
int split_point, int inversed, float quality );
virtual CvDTreeSplit* new_split_cat( int vi, float quality );
virtual void free_node_data( CvDTreeNode* node );
virtual void free_train_data();
virtual void free_node( CvDTreeNode* node );
int sample_count, var_all, var_count, max_c_count;
int ord_var_count, cat_var_count;
bool have_labels, have_priors;
bool is_classifier;
int buf_count, buf_size;
bool shared;
Mat& cat_count;
Mat& cat_ofs;
Mat& cat_map;
Mat& counts;
Mat& buf;
Mat& direction;
Mat& split_buf;
Mat& var_idx;
Mat& var_type; // i-th element =
// k<0 - ordered
// k>=0 - categorical, see k-th element of cat_* arrays
Mat& priors;
CvDTreeParams params;
CvMemStorage* tree_storage;
CvMemStorage* temp_storage;
CvDTreeNode* data_root;
CvSet* node_heap;
CvSet* split_heap;
CvSet* cv_heap;
CvSet* nv_heap;
CvRNG rng;
};
This structure is mostly used internally for storing both standalone trees and tree ensembles efficiently. Basically, it contains the following types of information:
#. Training parameters, an instance of :ref:`CvDTreeParams`.
#. Training data, preprocessed to find the best splits more efficiently. For tree ensembles, this preprocessed data is reused by all trees. Additionally, the training data characteristics shared by all trees in the ensemble are stored here: variable types, the number of classes, class label compression map, and so on.
#. Training data preprocessed to find the best splits more efficiently. For tree ensembles, this preprocessed data is reused by all trees. Additionally, the training data characteristics shared by all trees in the ensemble are stored here: variable types, the number of classes, a class label compression map, and so on.
#. Buffers, memory storages for tree nodes, splits, and other elements of the constructed trees.
There are two ways of using this structure. In simple cases (for example, a standalone tree or the ready-to-use "black box" tree ensemble from ML, like
There are two ways of using this structure. In simple cases (for example, a standalone tree or the ready-to-use "black box" tree ensemble from machine learning, like
:ref:`Random Trees` or
:ref:`Boosting` ), there is no need to care or even to know about the structure. You just construct the needed statistical model, train it, and use it. The ``CvDTreeTrainData`` structure is constructed and used internally. However, for custom tree algorithms or another sophisticated cases, the structure may be constructed and used explicitly. The scheme is the following:
#.
The structure is initialized using the default constructor, followed by ``set_data`` , or it is built using the full form of constructor. The parameter ``_shared`` must be set to ``true`` .
The structure is initialized using the default constructor, followed by ``set_data``, or it is built using the full form of constructor. The parameter ``_shared`` must be set to ``true``.
#.
One or more trees are trained using this data (see the special form of the method ``CvDTree::train`` ).
One or more trees are trained using this data (see the special form of the method :ocv:func:`CvDTree::train`).
#.
The structure is released as soon as all the trees using it are released.
.. index:: CvDTree
.. _CvDTree:
CvDTree
-------
.. c:type:: CvDTree
.. ocv:class:: CvDTree
Decision tree ::
The class implements a decision tree as described in the beginning of this section.
class CvDTree : public CvStatModel
{
public:
CvDTree();
virtual ~CvDTree();
virtual bool train( const Mat& _train_data, int _tflag,
const Mat& _responses, const Mat& _var_idx=Mat(),
const Mat& _sample_idx=Mat(), const Mat& _var_type=Mat(),
const Mat& _missing_mask=Mat(),
CvDTreeParams params=CvDTreeParams() );
virtual bool train( CvDTreeTrainData* _train_data,
const Mat& _subsample_idx );
virtual CvDTreeNode* predict( const Mat& _sample,
const Mat& _missing_data_mask=Mat(),
bool raw_mode=false ) const;
virtual const Mat& get_var_importance();
virtual void clear();
virtual void read( CvFileStorage* fs, CvFileNode* node );
virtual void write( CvFileStorage* fs, const char* name );
// special read & write methods for trees in the tree ensembles
virtual void read( CvFileStorage* fs, CvFileNode* node,
CvDTreeTrainData* data );
virtual void write( CvFileStorage* fs );
const CvDTreeNode* get_root() const;
int get_pruned_tree_idx() const;
CvDTreeTrainData* get_data();
protected:
virtual bool do_train( const Mat& _subsample_idx );
virtual void try_split_node( CvDTreeNode* n );
virtual void split_node_data( CvDTreeNode* n );
virtual CvDTreeSplit* find_best_split( CvDTreeNode* n );
virtual CvDTreeSplit* find_split_ord_class( CvDTreeNode* n, int vi );
virtual CvDTreeSplit* find_split_cat_class( CvDTreeNode* n, int vi );
virtual CvDTreeSplit* find_split_ord_reg( CvDTreeNode* n, int vi );
virtual CvDTreeSplit* find_split_cat_reg( CvDTreeNode* n, int vi );
virtual CvDTreeSplit* find_surrogate_split_ord( CvDTreeNode* n, int vi );
virtual CvDTreeSplit* find_surrogate_split_cat( CvDTreeNode* n, int vi );
virtual double calc_node_dir( CvDTreeNode* node );
virtual void complete_node_dir( CvDTreeNode* node );
virtual void cluster_categories( const int* vectors, int vector_count,
int var_count, int* sums, int k, int* cluster_labels );
virtual void calc_node_value( CvDTreeNode* node );
virtual void prune_cv();
virtual double update_tree_rnc( int T, int fold );
virtual int cut_tree( int T, int fold, double min_alpha );
virtual void free_prune_data(bool cut_tree);
virtual void free_tree();
virtual void write_node( CvFileStorage* fs, CvDTreeNode* node );
virtual void write_split( CvFileStorage* fs, CvDTreeSplit* split );
virtual CvDTreeNode* read_node( CvFileStorage* fs,
CvFileNode* node,
CvDTreeNode* parent );
virtual CvDTreeSplit* read_split( CvFileStorage* fs, CvFileNode* node );
virtual void write_tree_nodes( CvFileStorage* fs );
virtual void read_tree_nodes( CvFileStorage* fs, CvFileNode* node );
CvDTreeNode* root;
int pruned_tree_idx;
Mat& var_importance;
CvDTreeTrainData* data;
};
.. index:: CvDTree::train
.. _CvDTree::train:
CvDTree::train
--------------
.. ocv:function:: bool CvDTree::train( const Mat& _train_data, int _tflag, const Mat& _responses, const Mat& _var_idx=Mat(), const Mat& _sample_idx=Mat(), const Mat& _var_type=Mat(), const Mat& _missing_mask=Mat(), CvDTreeParams params=CvDTreeParams() )
Trains a decision tree.
.. ocv:function:: bool CvDTree::train( CvDTreeTrainData* _train_data, const Mat& _subsample_idx )
.. ocv:function:: bool CvDTree::train( const Mat& train_data, int tflag, const Mat& responses, const Mat& var_idx=Mat(), const Mat& sample_idx=Mat(), const Mat& var_type=Mat(), const Mat& missing_mask=Mat(), CvDTreeParams params=CvDTreeParams() )
Trains a decision tree.
.. ocv:function::bool CvDTree::train( const CvMat* trainData, int tflag, const CvMat* responses, const CvMat* varIdx=0, const CvMat* sampleIdx=0, const CvMat* varType=0, const CvMat* missingDataMask=0, CvDTreeParams params=CvDTreeParams() )
There are two ``train`` methods in ``CvDTree`` :
.. ocv:function::bool CvDTree::train( CvMLData* trainData, CvDTreeParams params=CvDTreeParams() )
* The first method follows the generic ``CvStatModel::train`` conventions. It is the most complete form. Both data layouts ( ``_tflag=CV_ROW_SAMPLE`` and ``_tflag=CV_COL_SAMPLE`` ) are supported, as well as sample and variable subsets, missing measurements, arbitrary combinations of input and output variable types, and so on. The last parameter contains all of the necessary training parameters (see the
:ref:`CvDTreeParams` description).
.. ocv:function::bool CvDTree::train( CvDTreeTrainData* trainData, const CvMat* subsampleIdx )
* The second method ``train`` is mostly used for building tree ensembles. It takes the pre-constructed
:ref:`CvDTreeTrainData` instance and an optional subset of the training set. The indices in ``_subsample_idx`` are counted relatively to the ``_sample_idx`` , passed to the ``CvDTreeTrainData`` constructor. For example, if ``_sample_idx=[1, 5, 7, 100]`` , then ``_subsample_idx=[0,3]`` means that the samples ``[1, 100]`` of the original training set are used.
.. ocv:pyfunction:: cv2.DTree.train(trainData, tflag, responses[, varIdx[, sampleIdx[, varType[, missingDataMask[, params]]]]]) -> retval
There are four ``train`` methods in :ocv:class:`CvDTree`:
* The **first two** methods follow the generic :ocv:func:`CvStatModel::train` conventions. It is the most complete form. Both data layouts (``tflag=CV_ROW_SAMPLE`` and ``tflag=CV_COL_SAMPLE``) are supported, as well as sample and variable subsets, missing measurements, arbitrary combinations of input and output variable types, and so on. The last parameter contains all of the necessary training parameters (see the :ocv:class:`CvDTreeParams` description).
* The **third** method uses :ocv:class:`CvMLData` to pass training data to a decision tree.
* The **last** method ``train`` is mostly used for building tree ensembles. It takes the pre-constructed :ocv:class:`CvDTreeTrainData` instance and an optional subset of the training set. The indices in ``subsampleIdx`` are counted relatively to the ``_sample_idx`` , passed to the ``CvDTreeTrainData`` constructor. For example, if ``_sample_idx=[1, 5, 7, 100]`` , then ``subsampleIdx=[0,3]`` means that the samples ``[1, 100]`` of the original training set are used.
.. index:: CvDTree::predict
.. _CvDTree::predict:
CvDTree::predict
----------------
.. ocv:function:: CvDTreeNode* CvDTree::predict( const Mat& _sample, const Mat& _missing_data_mask=Mat(), bool raw_mode=false ) const
Returns the leaf node of a decision tree corresponding to the input vector.
Returns the leaf node of a decision tree corresponding to the input vector.
.. ocv:function:: CvDTreeNode* CvDTree::predict( const Mat& sample, const Mat& missingDataMask=Mat(), bool preprocessedInput=false ) const
The method takes the feature vector and an optional missing measurement mask as input, traverses the decision tree, and returns the reached leaf node as output. The prediction result, either the class label or the estimated function value, may be retrieved as the ``value`` field of the
:ref:`CvDTreeNode` structure, for example: dtree-
:math:`>` predict(sample,mask)-
:math:`>` value.
.. ocv:function::CvDTreeNode* CvDTree::predict( const CvMat* sample, const CvMat* missingDataMask=0, bool preprocessedInput=false ) const
The last parameter is normally set to ``false`` , implying a regular
input. If it is ``true`` , the method assumes that all the values of
the discrete input variables have been already normalized to
:math:`0` to
:math:`num\_of\_categories_i-1` ranges since the decision tree uses such
normalized representation internally. It is useful for faster prediction
with tree ensembles. For ordered input variables, the flag is not used.
.. ocv:pyfunction:: cv2.DTree.predict(sample[, missingDataMask[, preprocessedInput]]) -> retval
:param sample: Sample for prediction.
:param missingDataMask: Optional input missing measurement mask.
:param preprocessedInput: This parameter is normally set to ``false``, implying a regular input. If it is ``true``, the method assumes that all the values of the discrete input variables have been already normalized to :math:`0` to :math:`num\_of\_categories_i-1` ranges since the decision tree uses such normalized representation internally. It is useful for faster prediction with tree ensembles. For ordered input variables, the flag is not used.
The method traverses the decision tree and returns the reached leaf node as output. The prediction result, either the class label or the estimated function value, may be retrieved as the ``value`` field of the :ocv:class:`CvDTreeNode` structure, for example: ``dtree->predict(sample,mask)->value``.
CvDTree::calc_error
-------------------
Returns error of the decision tree.
.. ocv:function::float CvDTree::calc_error( CvMLData* trainData, int type, std::vector<float> *resp = 0 )
:param data: Data for the decision tree.
:param type: Type of error. Possible values are:
* **CV_TRAIN_ERROR** Error on train samples.
* **CV_TEST_ERROR** Erron on test samples.
:param resp: If it is not null then size of this vector will be set to the number of samples and each element will be set to result of prediction on the corresponding sample.
The method calculates error of the decision tree. In case of classification it is the percentage of incorrectly classified samples and in case of regression it is the mean of squared errors on samples.
CvDTree::getVarImportance
-------------------------
Returns the variable importance array.
.. ocv:function:: Mat CvDTree::getVarImportance()
.. ocv:function::const CvMat* CvDTree::get_var_importance()
.. ocv:pyfunction:: cv2.DTree.getVarImportance() -> importanceVector
CvDTree::get_root
-----------------
Returns the root of the decision tree.
.. ocv:function:: const CvDTreeNode* CvDTree::get_root() const
CvDTree::get_pruned_tree_idx
----------------------------
Returns the ``CvDTree::pruned_tree_idx`` parameter.
.. ocv:function:: int CvDTree::get_pruned_tree_idx() const
The parameter ``DTree::pruned_tree_idx`` is used to prune a decision tree. See the ``CvDTreeNode::Tn`` parameter.
CvDTree::get_data
-----------------
Returns used train data of the decision tree.
.. ocv:function::const CvDTreeTrainData* CvDTree::get_data() const
Example: building a tree for classifying mushrooms. See the ``mushroom.cpp`` sample that demonstrates how to build and use the
decision tree.
.. [Breiman84] Breiman, L., Friedman, J. Olshen, R. and Stone, C. (1984), *Classification and Regression Trees*, Wadsworth.
modules/ml/doc/expectation_maximization.rst
+182 -100
View File
@@ -1,7 +1,9 @@
Expectation Maximization
========================
The EM (Expectation Maximization) algorithm estimates the parameters of the multivariate probability density function in the form of a Gaussian mixture distribution with a specified number of mixtures.
.. highlight:: cpp
The Expectation Maximization(EM) algorithm estimates the parameters of the multivariate probability density function in the form of a Gaussian mixture distribution with a specified number of mixtures.
Consider the set of the N feature vectors
{ :math:`x_1, x_2,...,x_{N}` } from a d-dimensional Euclidean space drawn from a Gaussian mixture:
@@ -59,7 +61,7 @@ At the second step (Maximization step or M-step), the mixture parameter estimate
Alternatively, the algorithm may start with the M-step when the initial values for
:math:`p_{i,k}` can be provided. Another alternative when
:math:`p_{i,k}` are unknown is to use a simpler clustering algorithm to pre-cluster the input samples and thus obtain initial
:math:`p_{i,k}` . Often (including ML) the
:math:`p_{i,k}` . Often (including macnine learning) the
:ref:`kmeans` algorithm is used for that purpose.
One of the main problems of the EM algorithm is a large number
@@ -83,121 +85,87 @@ already a good enough approximation).
*
Bilmes98 J. A. Bilmes. *A Gentle Tutorial of the EM Algorithm and its Application to Parameter Estimation for Gaussian Mixture and Hidden Markov Models*. Technical Report TR-97-021, International Computer Science Institute and Computer Science Division, University of California at Berkeley, April 1998.
.. index:: CvEMParams
.. _CvEMParams:
CvEMParams
----------
.. c:type:: CvEMParams
.. ocv:class:: CvEMParams
Parameters of the EM algorithm ::
Parameters of the EM algorithm. All parameters are public. You can initialize them by a constructor and then override some of them directly if you want.
struct CvEMParams
CvEMParams::CvEMParams
----------------------
The constructors
.. ocv:function:: CvEMParams::CvEMParams()
.. ocv:function:: CvEMParams::CvEMParams( int nclusters, int cov_mat_type=CvEM::COV_MAT_DIAGONAL, int start_step=CvEM::START_AUTO_STEP, CvTermCriteria term_crit=cvTermCriteria(CV_TERMCRIT_ITER+CV_TERMCRIT_EPS, 100, FLT_EPSILON), const CvMat* probs=0, const CvMat* weights=0, const CvMat* means=0, const CvMat** covs=0 )
:param nclusters: The number of mixture components in the gaussian mixture model. Some of EM implementation could determine the optimal number of mixtures within a specified value range, but that is not the case in ML yet.
:param cov_mat_type: Constraint on covariance matrices which defines type of matrices. Possible values are:
* **CvEM::COV_MAT_SPHERICAL** A scaled identity matrix :math:`\mu_k * I`. There is the only parameter :math:`\mu_k` to be estimated for earch matrix. The option may be used in special cases, when the constraint is relevant, or as a first step in the optimization (for example in case when the data is preprocessed with PCA). The results of such preliminary estimation may be passed again to the optimization procedure, this time with ``cov_mat_type=CvEM::COV_MAT_DIAGONAL``.
* **CvEM::COV_MAT_DIAGONAL** A diagonal matrix with positive diagonal elements. The number of free parameters is ``d`` for each matrix. This is most commonly used option yielding good estimation results.
* **CvEM::COV_MAT_GENERIC** A symmetric positively defined matrix. The number of free parameters in each matrix is about :math:`d^2/2`. It is not recommended to use this option, unless there is pretty accurate initial estimation of the parameters and/or a huge number of training samples.
:param start_step: The start step of the EM algorithm:
* **CvEM::START_E_STEP** Start with Expectation step. You need to provide means :math:`a_k` of mixture components to use this option. Optionally you can pass weights :math:`\pi_k` and covariance matrices :math:`S_k` of mixture components.
* **CvEM::START_M_STEP** Start with Maximization step. You need to provide initial probabilites :math:`p_{i,k}` to use this option.
* **CvEM::START_AUTO_STEP** Start with Expectation step. You need not provide any parameters because they will be estimated by the k-means algorithm.
:param term_crit: The termination criteria of the EM algorithm. The EM algorithm can be terminated by the number of iterations ``term_crit.max_iter`` (number of M-steps) or when relative change of likelihood logarithm is less than ``term_crit.epsilon``.
:param probs: Initial probabilities :math:`p_{i,k}` of sample :math:`i` to belong to mixture component :math:`k`. It is a floating-point matrix of :math:`nsamples \times nclusters` size. It is used and must be not NULL only when ``start_step=CvEM::START_M_STEP``.
:param weights: Initial weights :math:`\pi_k` of mixture components. It is a floating-point vector with :math:`nclusters` elements. It is used (if not NULL) only when ``start_step=CvEM::START_E_STEP``.
:param means: Initial means :math:`a_k` of mixture components. It is a floating-point matrix of :math:`nclusters \times dims` size. It is used used and must be not NULL only when ``start_step=CvEM::START_E_STEP``.
:param covs: Initial covariance matrices :math:`S_k` of mixture components. Each of covariance matrices is a valid square floating-point matrix of :math:`dims \times dims` size. It is used (if not NULL) only when ``start_step=CvEM::START_E_STEP``.
The default constructor represents a rough rule-of-the-thumb:
::
CvEMParams() : nclusters(10), cov_mat_type(1/*CvEM::COV_MAT_DIAGONAL*/),
start_step(0/*CvEM::START_AUTO_STEP*/), probs(0), weights(0), means(0), covs(0)
{
CvEMParams() : nclusters(10), cov_mat_type(CvEM::COV_MAT_DIAGONAL),
start_step(CvEM::START_AUTO_STEP), probs(0), weights(0), means(0),
covs(0)
{
term_crit=cvTermCriteria( CV_TERMCRIT_ITER+CV_TERMCRIT_EPS,
100, FLT_EPSILON );
}
CvEMParams( int _nclusters, int _cov_mat_type=1/*CvEM::COV_MAT_DIAGONAL*/,
int _start_step=0/*CvEM::START_AUTO_STEP*/,
CvTermCriteria _term_crit=cvTermCriteria(
CV_TERMCRIT_ITER+CV_TERMCRIT_EPS,
100, FLT_EPSILON),
const CvMat* _probs=0, const CvMat* _weights=0,
const CvMat* _means=0, const CvMat** _covs=0 ) :
nclusters(_nclusters), cov_mat_type(_cov_mat_type),
start_step(_start_step),
probs(_probs), weights(_weights), means(_means), covs(_covs),
term_crit(_term_crit)
{}
int nclusters;
int cov_mat_type;
int start_step;
const CvMat* probs;
const CvMat* weights;
const CvMat* means;
const CvMat** covs;
CvTermCriteria term_crit;
};
term_crit=cvTermCriteria( CV_TERMCRIT_ITER+CV_TERMCRIT_EPS, 100, FLT_EPSILON );
}
The structure has two constructors. The default one represents a rough rule-of-the-thumb. With another one it is possible to override a variety of parameters from a single number of mixtures (the only essential problem-dependent parameter) to initial values for the mixture parameters.
With another contstructor it is possible to override a variety of parameters from a single number of mixtures (the only essential problem-dependent parameter) to initial values for the mixture parameters.
.. index:: CvEM
.. _CvEM:
CvEM
----
.. c:type:: CvEM
.. ocv:class:: CvEM
EM model ::
The class implements the EM algorithm as described in the beginning of this section.
class CV_EXPORTS CvEM : public CvStatModel
{
public:
// Type of covariance matrices
enum { COV_MAT_SPHERICAL=0, COV_MAT_DIAGONAL=1, COV_MAT_GENERIC=2 };
// Initial step
enum { START_E_STEP=1, START_M_STEP=2, START_AUTO_STEP=0 };
CvEM();
CvEM( const Mat& samples, const Mat& sample_idx=Mat(),
CvEMParams params=CvEMParams(), Mat* labels=0 );
virtual ~CvEM();
virtual bool train( const Mat& samples, const Mat& sample_idx=Mat(),
CvEMParams params=CvEMParams(), Mat* labels=0 );
virtual float predict( const Mat& sample, Mat& probs ) const;
virtual void clear();
int get_nclusters() const { return params.nclusters; }
const Mat& get_means() const { return means; }
const Mat&* get_covs() const { return covs; }
const Mat& get_weights() const { return weights; }
const Mat& get_probs() const { return probs; }
protected:
virtual void set_params( const CvEMParams& params,
const CvVectors& train_data );
virtual void init_em( const CvVectors& train_data );
virtual double run_em( const CvVectors& train_data );
virtual void init_auto( const CvVectors& samples );
virtual void kmeans( const CvVectors& train_data, int nclusters,
Mat& labels, CvTermCriteria criteria,
const Mat& means );
CvEMParams params;
double log_likelihood;
Mat& means;
Mat&* covs;
Mat& weights;
Mat& probs;
Mat& log_weight_div_det;
Mat& inv_eigen_values;
Mat&* cov_rotate_mats;
};
.. index:: CvEM::train
.. _CvEM::train:
CvEM::train
-----------
Estimates the Gaussian mixture parameters from a sample set.
.. ocv:function:: void CvEM::train( const Mat& samples, const Mat& sample_idx=Mat(), CvEMParams params=CvEMParams(), Mat* labels=0 )
Estimates the Gaussian mixture parameters from a sample set.
.. ocv:function:: bool CvEM::train( const CvMat* samples, const CvMat* sampleIdx=0, CvEMParams params=CvEMParams(), CvMat* labels=0 )
.. ocv:pyfunction:: cv2.EM.train(samples[, sampleIdx[, params]]) -> retval, labels
:param samples: Samples from which the Gaussian mixture model will be estimated.
:param sample_idx: Mask of samples to use. All samples are used by default.
:param params: Parameters of the EM algorithm.
:param labels: The optional output "class label" for each sample: :math:`\texttt{labels}_i=\texttt{arg max}_k(p_{i,k}), i=1..N` (indices of the most probable mixture component for each sample).
Unlike many of the ML models, EM is an unsupervised learning algorithm and it does not take responses (class labels or function values) as input. Instead, it computes the
*Maximum Likelihood Estimate* of the Gaussian mixture parameters from an input sample set, stores all the parameters inside the structure:
@@ -205,12 +173,126 @@ Unlike many of the ML models, EM is an unsupervised learning algorithm and it do
:math:`a_k` in ``means`` ,
:math:`S_k` in ``covs[k]``,
:math:`\pi_k` in ``weights`` , and optionally computes the output "class label" for each sample:
:math:`\texttt{labels}_i=\texttt{arg max}_k(p_{i,k}), i=1..N` (indices of the most probable mixture for each sample).
:math:`\texttt{labels}_i=\texttt{arg max}_k(p_{i,k}), i=1..N` (indices of the most probable mixture component for each sample).
The trained model can be used further for prediction, just like any other classifier. The trained model is similar to the
:ref:`Bayes classifier`.
For example of clustering random samples of multi-Gaussian distribution using EM see em.cpp sample in OpenCV distribution.
For an example of clustering random samples of the multi-Gaussian distribution using EM, see ``em.cpp`` sample in the OpenCV distribution.
CvEM::predict
-------------
Returns a mixture component index of a sample.
.. ocv:function:: float CvEM::predict( const Mat& sample, Mat* probs=0 ) const
.. ocv:function:: float CvEM::predict( const CvMat* sample, CvMat* probs ) const
.. ocv:pyfunction:: cv2.EM.predict(sample) -> retval, probs
:param sample: A sample for classification.
:param probs: If it is not null then the method will write posterior probabilities of each component given the sample data to this parameter.
CvEM::getNClusters
------------------
Returns the number of mixture components :math:`M` in the gaussian mixture model.
.. ocv:function:: int CvEM::getNClusters() const
.. ocv:function:: int CvEM::get_nclusters() const
.. ocv:pyfunction:: cv2.EM.getNClusters() -> retval
CvEM::getMeans
------------------
Returns mixture means :math:`a_k`.
.. ocv:function:: Mat CvEM::getMeans() const
.. ocv:function:: const CvMat* CvEM::get_means() const
.. ocv:pyfunction:: cv2.EM.getMeans() -> means
CvEM::getCovs
-------------
Returns mixture covariance matrices :math:`S_k`.
.. ocv:function:: void CvEM::getCovs(std::vector<cv::Mat>& covs) const
.. ocv:function:: const CvMat** CvEM::get_covs() const
.. ocv:pyfunction:: cv2.EM.getCovs([covs]) -> covs
CvEM::getWeights
----------------
Returns mixture weights :math:`\pi_k`.
.. ocv:function:: Mat CvEM::getWeights() const
.. ocv:function:: const CvMat* CvEM::get_weights() const
.. ocv:pyfunction:: cv2.EM.getWeights() -> weights
CvEM::getProbs
--------------
Returns vectors of probabilities for each training sample.
.. ocv:function:: Mat CvEM::getProbs() const
.. ocv:function:: const CvMat* CvEM::get_probs() const
.. ocv:pyfunction:: cv2.EM.getProbs() -> probs
For each training sample :math:`i` (that have been passed to the constructor or to :ocv:func:`CvEM::train`) returns probabilites :math:`p_{i,k}` to belong to a mixture component :math:`k`.
CvEM::getLikelihood
-------------------
Returns logarithm of likelihood.
.. ocv:function:: double CvEM::getLikelihood() const
.. ocv:function:: double CvEM::get_log_likelihood() const
.. ocv:pyfunction:: cv2.EM.getLikelihood() -> likelihood
CvEM::getLikelihoodDelta
------------------------
Returns difference between logarithm of likelihood on the last iteration and logarithm of likelihood on the previous iteration.
.. ocv:function:: double CvEM::getLikelihoodDelta() const
.. ocv:function:: double CvEM::get_log_likelihood_delta() const
.. ocv:pyfunction:: cv2.EM.getLikelihoodDelta() -> likelihood delta
CvEM::write_params
------------------
Writes used parameters of the EM algorithm to a file storage.
.. ocv:function:: void CvEM::write_params( CvFileStorage* fs ) const
:param fs: A file storage where parameters will be written.
CvEM::read_params
-----------------
Reads parameters of the EM algorithm.
.. ocv:function:: void CvEM::read_params( CvFileStorage* fs, CvFileNode* node )
:param fs: A file storage with parameters of the EM algorithm.
:param node: The parent map. If it is NULL, the function searches a node with parameters in all the top-level nodes (streams), starting with the first one.
The function reads EM parameters from the specified file storage node. For example of clustering random samples of multi-Gaussian distribution using EM see em.cpp sample in OpenCV distribution.
modules/ml/doc/gradient_boosted_trees.rst
+126 -227
View File
@@ -3,24 +3,24 @@
Gradient Boosted Trees
======================
Gradient Boosted Trees (GBT) is a generalized boosting algorithm, introduced by
.. highlight:: cpp
Gradient Boosted Trees (GBT) is a generalized boosting algorithm introduced by
Jerome Friedman: http://www.salfordsystems.com/doc/GreedyFuncApproxSS.pdf .
In contrast to AdaBoost.M1 algorithm GBT can deal with both multiclass
classification and regression problems. More than that it can use any
In contrast to the AdaBoost.M1 algorithm, GBT can deal with both multiclass
classification and regression problems. Moreover, it can use any
differential loss function, some popular ones are implemented.
Decision trees (:ref:`CvDTree`) usage as base learners allows to process ordered
Decision trees (:ocv:class:`CvDTree`) usage as base learners allows to process ordered
and categorical variables.
.. _Training the GBT model:
Training the GBT model
----------------------
Gradient Boosted Trees model represents an ensemble of single regression trees,
that are built in a greedy fashion. Training procedure is an iterative proccess
similar to the numerical optimazation via gradient descent method. Summary loss
on the training set depends only from the current model predictions on the
Gradient Boosted Trees model represents an ensemble of single regression trees
built in a greedy fashion. Training procedure is an iterative proccess
similar to the numerical optimization via the gradient descent method. Summary loss
on the training set depends only on the current model predictions for the
thaining samples, in other words
:math:`\sum^N_{i=1}L(y_i, F(x_i)) \equiv \mathcal{L}(F(x_1), F(x_2), ... , F(x_N))
\equiv \mathcal{L}(F)`. And the :math:`\mathcal{L}(F)`
@@ -30,12 +30,13 @@ gradient can be computed as follows:
grad(\mathcal{L}(F)) = \left( \dfrac{\partial{L(y_1, F(x_1))}}{\partial{F(x_1)}},
\dfrac{\partial{L(y_2, F(x_2))}}{\partial{F(x_2)}}, ... ,
\dfrac{\partial{L(y_N, F(x_N))}}{\partial{F(x_N)}} \right) .
On every training step a single regression tree is built to predict an
antigradient vector components. Step length is computed corresponding to the
loss function and separately for every region determined by the tree leaf, and
can be eliminated by changing leaves' values directly.
The main scheme of the training proccess is shown below.
At every training step, a single regression tree is built to predict an
antigradient vector components. Step length is computed corresponding to the
loss function and separately for every region determined by the tree leaf. It
can be eliminated by changing values of the leaves directly.
See below the main scheme of the training proccess:
#.
Find the best constant model.
@@ -52,114 +53,90 @@ The main scheme of the training proccess is shown below.
Add the tree to the model.
The following loss functions are implemented:
The following loss functions are implemented for regression problems:
*for regression problems:*
#.
*
Squared loss (``CvGBTrees::SQUARED_LOSS``):
:math:`L(y,f(x))=\dfrac{1}{2}(y-f(x))^2`
#.
*
Absolute loss (``CvGBTrees::ABSOLUTE_LOSS``):
:math:`L(y,f(x))=|y-f(x)|`
#.
*
Huber loss (``CvGBTrees::HUBER_LOSS``):
:math:`L(y,f(x)) = \left\{ \begin{array}{lr}
\delta\cdot\left(|y-f(x)|-\dfrac{\delta}{2}\right) & : |y-f(x)|>\delta\\
\dfrac{1}{2}\cdot(y-f(x))^2 & : |y-f(x)|\leq\delta \end{array} \right.`,
where :math:`\delta` is the :math:`\alpha`-quantile estimation of the
where :math:`\delta` is the :math:`\alpha`-quantile estimation of the
:math:`|y-f(x)|`. In the current implementation :math:`\alpha=0.2`.
*for classification problems:*
4.
The following loss functions are implemented for classification problems:
*
Deviance or cross-entropy loss (``CvGBTrees::DEVIANCE_LOSS``):
:math:`K` functions are built, one function for each output class, and
:math:`L(y,f_1(x),...,f_K(x)) = -\sum^K_{k=0}1(y=k)\ln{p_k(x)}`,
where :math:`p_k(x)=\dfrac{\exp{f_k(x)}}{\sum^K_{i=1}\exp{f_i(x)}}`
is the estimation of the probability that :math:`y=k`.
is the estimation of the probability of :math:`y=k`.
In the end we get the model in the following form:
As a result, you get the following model:
.. math:: f(x) = f_0 + \nu\cdot\sum^M_{i=1}T_i(x) ,
where :math:`f_0` is the initial guess (the best constant model) and :math:`\nu`
where :math:`f_0` is an initial guess (the best constant model) and :math:`\nu`
is a regularization parameter from the interval :math:`(0,1]`, futher called
*shrinkage*.
.. _Predicting with GBT model:
Predicting with GBT model
Predicting with the GBT Model
-------------------------
To get the GBT model prediciton it is needed to compute the sum of responses of
all the trees in the ensemble. For regression problems it is the answer, and
for classification problems the result is :math:`\arg\max_{i=1..K}(f_i(x))`.
To get the GBT model prediciton, you need to compute the sum of responses of
all the trees in the ensemble. For regression problems, it is the answer.
For classification problems, the result is :math:`\arg\max_{i=1..K}(f_i(x))`.
.. highlight:: cpp
.. index:: CvGBTreesParams
.. _CvGBTreesParams:
CvGBTreesParams
---------------
.. c:type:: CvGBTreesParams
.. ocv:class:: CvGBTreesParams
GBT training parameters ::
struct CvGBTreesParams : public CvDTreeParams
{
int weak_count;
int loss_function_type;
float subsample_portion;
float shrinkage;
CvGBTreesParams();
CvGBTreesParams( int loss_function_type, int weak_count, float shrinkage,
float subsample_portion, int max_depth, bool use_surrogates );
};
GBT training parameters.
The structure contains parameters for each sigle decision tree in the ensemble,
as well as the whole model characteristics. The structure is derived from
:ref:`CvDTreeParams` but not all of the decision tree parameters are supported:
cross-validation, pruning and class priorities are not used. The whole
parameters list is shown below:
:ocv:class:`CvDTreeParams` but not all of the decision tree parameters are supported:
cross-validation, pruning, and class priorities are not used.
``weak_count``
CvGBTreesParams::CvGBTreesParams
--------------------------------
.. ocv:function:: CvGBTreesParams::CvGBTreesParams()
The count of boosting algorithm iterations. ``weak_count*K`` -- is the total
count of trees in the GBT model, where ``K`` is the output classes count
(equal to one in the case of regression).
``loss_function_type``
.. ocv:function:: CvGBTreesParams::CvGBTreesParams( int loss_function_type, int weak_count, float shrinkage, float subsample_portion, int max_depth, bool use_surrogates )
The type of the loss function used for training
:param loss_function_type: Type of the loss function used for training
(see :ref:`Training the GBT model`). It must be one of the
following: ``CvGBTrees::SQUARED_LOSS``, ``CvGBTrees::ABSOLUTE_LOSS``,
following types: ``CvGBTrees::SQUARED_LOSS``, ``CvGBTrees::ABSOLUTE_LOSS``,
``CvGBTrees::HUBER_LOSS``, ``CvGBTrees::DEVIANCE_LOSS``. The first three
ones are used for the case of regression problems, and the last one for
types are used for regression problems, and the last one for
classification.
:param weak_count: Count of boosting algorithm iterations. ``weak_count*K`` is the total
count of trees in the GBT model, where ``K`` is the output classes count
(equal to one in case of a regression).
:param shrinkage: Regularization parameter (see :ref:`Training the GBT model`).
``shrinkage``
:param subsample_portion: Portion of the whole training set used for each algorithm iteration.
Subset is generated randomly. For more information see
http://www.salfordsystems.com/doc/StochasticBoostingSS.pdf.
Regularization parameter (see :ref:`Training the GBT model`).
``subsample_portion``
:param max_depth: Maximal depth of each decision tree in the ensemble (see :ocv:class:`CvDTree`).
The portion of the whole training set used on each algorithm iteration.
Subset is generated randomly
(For more information see
http://www.salfordsystems.com/doc/StochasticBoostingSS.pdf).
``max_depth``
The maximal depth of each decision tree in the ensemble (see :ref:`CvDTree`).
``use_surrogates``
If ``true`` surrogate splits are built (see :ref:`CvDTree`).
:param use_surrogates: If ``true``, surrogate splits are built (see :ocv:class:`CvDTree`).
By default the following constructor is used:
@@ -168,204 +145,126 @@ By default the following constructor is used:
CvGBTreesParams(CvGBTrees::SQUARED_LOSS, 200, 0.8f, 0.01f, 3, false)
: CvDTreeParams( 3, 10, 0, false, 10, 0, false, false, 0 )
.. index:: CvGBTrees
.. _CvGBTrees:
CvGBTrees
---------
.. c:type:: CvGBTrees
.. ocv:class:: CvGBTrees
GBT model ::
The class implements the Gradient boosted tree model as described in the beginning of this section.
class CvGBTrees : public CvStatModel
{
public:
CvGBTrees::CvGBTrees
--------------------
Default and training constructors.
enum {SQUARED_LOSS=0, ABSOLUTE_LOSS, HUBER_LOSS=3, DEVIANCE_LOSS};
.. ocv:function:: CvGBTrees::CvGBTrees()
CvGBTrees();
CvGBTrees( const cv::Mat& trainData, int tflag,
const Mat& responses, const Mat& varIdx=Mat(),
const Mat& sampleIdx=Mat(), const cv::Mat& varType=Mat(),
const Mat& missingDataMask=Mat(),
CvGBTreesParams params=CvGBTreesParams() );
.. ocv:function:: CvGBTrees::CvGBTrees( const Mat& trainData, int tflag, const Mat& responses, const Mat& varIdx=Mat(), const Mat& sampleIdx=Mat(), const Mat& varType=Mat(), const Mat& missingDataMask=Mat(), CvGBTreesParams params=CvGBTreesParams() )
virtual ~CvGBTrees();
virtual bool train( const Mat& trainData, int tflag,
const Mat& responses, const Mat& varIdx=Mat(),
const Mat& sampleIdx=Mat(), const Mat& varType=Mat(),
const Mat& missingDataMask=Mat(),
CvGBTreesParams params=CvGBTreesParams(),
bool update=false );
virtual bool train( CvMLData* data,
CvGBTreesParams params=CvGBTreesParams(),
bool update=false );
.. ocv:function::CvGBTrees::CvGBTrees( const CvMat* trainData, int tflag, const CvMat* responses, const CvMat* varIdx=0, const CvMat* sampleIdx=0, const CvMat* varType=0, const CvMat* missingDataMask=0, CvGBTreesParams params=CvGBTreesParams() )
virtual float predict( const Mat& sample, const Mat& missing=Mat(),
const Range& slice = Range::all(),
int k=-1 ) const;
.. ocv:pyfunction:: cv2.GBTrees([trainData, tflag, responses[, varIdx[, sampleIdx[, varType[, missingDataMask[, params]]]]]]) -> <GBTrees object>
virtual void clear();
virtual float calc_error( CvMLData* _data, int type,
std::vector<float> *resp = 0 );
virtual void write( CvFileStorage* fs, const char* name ) const;
virtual void read( CvFileStorage* fs, CvFileNode* node );
protected:
CvDTreeTrainData* data;
CvGBTreesParams params;
CvSeq** weak;
Mat& orig_response;
Mat& sum_response;
Mat& sum_response_tmp;
Mat& weak_eval;
Mat& sample_idx;
Mat& subsample_train;
Mat& subsample_test;
Mat& missing;
Mat& class_labels;
RNG* rng;
int class_count;
float delta;
float base_value;
...
};
.. index:: CvGBTrees::train
.. _CvGBTrees::train:
The constructors follow conventions of :ocv:func:`CvStatModel::CvStatModel`. See :ocv:func:`CvStatModel::train` for parameters descriptions.
CvGBTrees::train
----------------
.. c:function:: bool train(const Mat & trainData, int tflag, const Mat & responses, const Mat & varIdx=Mat(), const Mat & sampleIdx=Mat(), const Mat & varType=Mat(), const Mat & missingDataMask=Mat(), CvGBTreesParams params=CvGBTreesParams(), bool update=false)
Trains a Gradient boosted tree model.
.. c:function:: bool train(CvMLData* data, CvGBTreesParams params=CvGBTreesParams(), bool update=false)
.. ocv:function:: bool CvGBTrees::train(const Mat& trainData, int tflag, const Mat& responses, const Mat& varIdx=Mat(), const Mat& sampleIdx=Mat(), const Mat& varType=Mat(), const Mat& missingDataMask=Mat(), CvGBTreesParams params=CvGBTreesParams(), bool update=false)
.. ocv:function::bool CvGBTrees::train( const CvMat* trainData, int tflag, const CvMat* responses, const CvMat* varIdx=0, const CvMat* sampleIdx=0, const CvMat* varType=0, const CvMat* missingDataMask=0, CvGBTreesParams params=CvGBTreesParams(), bool update=false )
.. ocv:function::bool CvGBTrees::train(CvMLData* data, CvGBTreesParams params=CvGBTreesParams(), bool update=false)
.. ocv:pyfunction:: cv2.GBTrees.train(trainData, tflag, responses[, varIdx[, sampleIdx[, varType[, missingDataMask[, params[, update]]]]]]) -> retval
Trains a Gradient boosted tree model.
The first train method follows the common template (see :ref:`CvStatModel::train`).
The first train method follows the common template (see :ocv:func:`CvStatModel::train`).
Both ``tflag`` values (``CV_ROW_SAMPLE``, ``CV_COL_SAMPLE``) are supported.
``trainData`` must be of ``CV_32F`` type. ``responses`` must be a matrix of type
``CV_32S`` or ``CV_32F``, in both cases it is converted into the ``CV_32F``
``trainData`` must be of the ``CV_32F`` type. ``responses`` must be a matrix of type
``CV_32S`` or ``CV_32F``. In both cases it is converted into the ``CV_32F``
matrix inside the training procedure. ``varIdx`` and ``sampleIdx`` must be a
list of indices (``CV_32S``), or a mask (``CV_8U`` or ``CV_8S``). ``update`` is
list of indices (``CV_32S``) or a mask (``CV_8U`` or ``CV_8S``). ``update`` is
a dummy parameter.
The second form of :ref:`CvGBTrees::train` function uses :ref:`CvMLData` as a
The second form of :ocv:func:`CvGBTrees::train` function uses :ocv:class:`CvMLData` as a
data set container. ``update`` is still a dummy parameter.
All parameters specific to the GBT model are passed into the training function
as a :ref:`CvGBTreesParams` structure.
as a :ocv:class:`CvGBTreesParams` structure.
.. index:: CvGBTrees::predict
.. _CvGBTrees::predict:
CvGBTrees::predict
------------------
.. c:function:: float predict(const Mat & sample, const Mat & missing=Mat(), const Range & slice = Range::all(), int k=-1) const
Predicts a response for an input sample.
Predicts a response for an input sample.
The method predicts the response, corresponding to the given sample
(see :ref:`Predicting with GBT model`).
The result is either the class label or the estimated function value.
:c:func:`predict` method allows to use the parallel version of the GBT model
prediction if the OpenCV is built with the TBB library. In this case predicitons
of single trees are computed in a parallel fashion.
.. ocv:function:: float CvGBTrees::predict(const Mat& sample, const Mat& missing=Mat(), const Range& slice = Range::all(), int k=-1) const
``sample``
.. ocv:function::float CvGBTrees::predict( const CvMat* sample, const CvMat* missing=0, CvMat* weakResponses=0, CvSlice slice = CV_WHOLE_SEQ, int k=-1 ) const
An input feature vector, that has the same format as every training set
element. Hence, if not all the variables were actualy used while training,
``sample`` have to contain fictive values on the appropriate places.
.. ocv:pyfunction:: cv2.GBTrees.predict(sample[, missing[, slice[, k]]]) -> retval
:param sample: Input feature vector that has the same format as every training set
element. If not all the variables were actualy used during training,
``sample`` contains forged values at the appropriate places.
``missing``
The missing values mask. The one dimentional matrix of the same size as
``sample`` having a ``CV_8U`` type. ``1`` corresponds to the missing value
:param missing: Missing values mask, which is a dimentional matrix of the same size as
``sample`` having the ``CV_8U`` type. ``1`` corresponds to the missing value
in the same position in the ``sample`` vector. If there are no missing values
in the feature vector empty matrix can be passed instead of the missing mask.
in the feature vector, an empty matrix can be passed instead of the missing mask.
``weak_responses``
In addition to the prediciton of the whole model all the trees' predcitions
can be obtained by passing a ``weak_responses`` matrix with :math:`K` rows,
where :math:`K` is the output classes count (1 for the case of regression)
and having as many columns as the ``slice`` length.
:param weak_responses: Matrix used to obtain predictions of all the trees.
The matrix has :math:`K` rows,
where :math:`K` is the count of output classes (1 for the regression case).
The matrix has as many columns as the ``slice`` length.
``slice``
Defines the part of the ensemble used for prediction.
All trees are used when ``slice = Range::all()``. This parameter is useful to
:param slice: Parameter defining the part of the ensemble used for prediction.
If ``slice = Range::all()``, all trees are used. Use this parameter to
get predictions of the GBT models with different ensemble sizes learning
only the one model actually.
``k``
In the case of the classification problem not the one, but :math:`K` tree
ensembles are built (see :ref:`Training the GBT model`). By passing this
parameter the ouput can be changed to sum of the trees' predictions in the
``k``'th ensemble only. To get the total GBT model prediction ``k`` value
must be -1. For regression problems ``k`` have to be equal to -1 also.
only one model.
:param k: Number of tree ensembles built in case of the classification problem
(see :ref:`Training the GBT model`). Use this
parameter to change the ouput to sum of the trees' predictions in the
``k``-th ensemble only. To get the total GBT model prediction, ``k`` value
must be -1. For regression problems, ``k`` is also equal to -1.
The method predicts the response corresponding to the given sample
(see :ref:`Predicting with the GBT model`).
The result is either the class label or the estimated function value. The
:ocv:func:`predict` method enables using the parallel version of the GBT model
prediction if the OpenCV is built with the TBB library. In this case, predictions
of single trees are computed in a parallel fashion.
.. index:: CvGBTrees::clear
.. _CvGBTrees::clear:
CvGBTrees::clear
----------------
.. c:function:: void clear()
Clears the model.
Clears the model.
.. ocv:function:: void CvGBTrees::clear()
Deletes the data set information, all the weak models and sets all internal
variables to the initial state. Is called in :ref:`CvGBTrees::train` and in the
.. ocv:pyfunction:: cv2.GBTrees.clear() -> None
The function deletes the data set information and all the weak models and sets all internal
variables to the initial state. The function is called in :ocv:func:`CvGBTrees::train` and in the
destructor.
.. index:: CvGBTrees::calc_error
.. _CvGBTrees::calc_error:
CvGBTrees::calc_error
---------------------
.. c:function:: float calc_error( CvMLData* _data, int type, std::vector<float> *resp = 0 )
Calculates a training or testing error.
Calculates training or testing error.
If the :ref:`CvMLData` data is used to store the data set :c:func:`calc_error` can be
used to get the training or testing error easily and (optionally) all predictions
on the training/testing set. If TBB library is used, the error is computed in a
parallel way: predictions for different samples are computed at the same time.
In the case of regression problem mean squared error is returned. For
classifications the result is the misclassification error in percent.
.. ocv:function:: float CvGBTrees::calc_error( CvMLData* _data, int type, std::vector<float> *resp = 0 )
``_data``
Data set.
:param _data: Data set.
``type``
Defines what error should be computed: train (``CV_TRAIN_ERROR``) or test
:param type: Parameter defining the error that should be computed: train (``CV_TRAIN_ERROR``) or test
(``CV_TEST_ERROR``).
``resp``
If not ``0`` a vector of predictions on the corresponding data set is
:param resp: If non-zero, a vector of predictions on the corresponding data set is
returned.
If the :ocv:class:`CvMLData` data is used to store the data set, :ocv:func:`calc_error` can be
used to get a training/testing error easily and (optionally) all predictions
on the training/testing set. If the Intel* TBB* library is used, the error is computed in a
parallel way, namely, predictions for different samples are computed at the same time.
In case of a regression problem, a mean squared error is returned. For
classifications, the result is a misclassification error in percent.
modules/ml/doc/k_nearest_neighbors.rst
+77 -59
View File
@@ -1,93 +1,111 @@
K Nearest Neighbors
K-Nearest Neighbors
===================
The algorithm caches all training samples and predicts the response for a new sample by analyzing a certain number (
**K**
) of the nearest neighbors of the sample (using voting, calculating weighted sum, and so on). The method is sometimes referred to as "learning by example" because for prediction it looks for the feature vector with a known response that is closest to the given vector.
.. highlight:: cpp
.. index:: CvKNearest
.. _CvKNearest:
The algorithm caches all training samples and predicts the response for a new sample by analyzing a certain number (**K**) of the nearest neighbors of the sample using voting, calculating weighted sum, and so on. The method is sometimes referred to as "learning by example" because for prediction it looks for the feature vector with a known response that is closest to the given vector.
CvKNearest
----------
.. c:type:: CvKNearest
.. ocv:class:: CvKNearest
K-Nearest Neighbors model ::
The class implements K-Nearest Neighbors model as described in the beginning of this section.
class CvKNearest : public CvStatModel
{
public:
CvKNearest::CvKNearest
----------------------
Default and training constructors.
CvKNearest();
virtual ~CvKNearest();
.. ocv:function:: CvKNearest::CvKNearest()
CvKNearest( const Mat& _train_data, const Mat& _responses,
const Mat& _sample_idx=Mat(), bool _is_regression=false, int max_k=32 );
.. ocv:function:: CvKNearest::CvKNearest( const Mat& trainData, const Mat& responses, const Mat& sampleIdx=Mat(), bool isRegression=false, int max_k=32 )
virtual bool train( const Mat& _train_data, const Mat& _responses,
const Mat& _sample_idx=Mat(), bool is_regression=false,
int _max_k=32, bool _update_base=false );
.. ocv:function::CvKNearest::CvKNearest( const CvMat* trainData, const CvMat* responses, const CvMat* sampleIdx=0, bool isRegression=false, int max_k=32 )
virtual float find_nearest( const Mat& _samples, int k, Mat* results=0,
const float** neighbors=0, Mat* neighbor_responses=0, Mat* dist=0 ) const;
virtual void clear();
int get_max_k() const;
int get_var_count() const;
int get_sample_count() const;
bool is_regression() const;
protected:
...
};
.. index:: CvKNearest::train
.. _CvKNearest::train:
See :ocv:func:`CvKNearest::train` for additional parameters descriptions.
CvKNearest::train
-----------------
.. ocv:function:: bool CvKNearest::train( const Mat& _train_data, const Mat& _responses, const Mat& _sample_idx=Mat(), bool is_regression=false, int _max_k=32, bool _update_base=false )
Trains the model.
Trains the model.
.. ocv:function:: bool CvKNearest::train( const Mat& trainData, const Mat& responses, const Mat& sampleIdx=Mat(), bool isRegression=false, int maxK=32, bool updateBase=false )
.. ocv:function::bool CvKNearest::train( const CvMat* trainData, const CvMat* responses, const CvMat* sampleIdx=0, bool is_regression=false, int maxK=32, bool updateBase=false )
.. ocv:pyfunction:: cv2.KNearest.train(trainData, responses[, sampleIdx[, isRegression[, maxK[, updateBase]]]]) -> retval
:param isRegression: Type of the problem: ``true`` for regression and ``false`` for classification.
:param maxK: Number of maximum neighbors that may be passed to the method :ocv:func:`CvKNearest::find_nearest`.
:param updateBase: Specifies whether the model is trained from scratch (``update_base=false``), or it is updated using the new training data (``update_base=true``). In the latter case, the parameter ``maxK`` must not be larger than the original value.
The method trains the K-Nearest model. It follows the conventions of the generic :ocv:func:`CvStataModel::train` approach with the following limitations:
The method trains the K-Nearest model. It follows the conventions of the generic ``train`` "method" with the following limitations:
* Only ``CV_ROW_SAMPLE`` data layout is supported.
* Input variables are all ordered.
* Output variables can be either categorical ( ``is_regression=false`` ) or ordered ( ``is_regression=true`` ).
* Variable subsets ( ``var_idx`` ) and missing measurements are not supported.
The parameter ``_max_k`` specifies the number of maximum neighbors that may be passed to the method ``find_nearest`` .
The parameter ``_update_base`` specifies whether the model is trained from scratch
( ``_update_base=false`` ), or it is updated using the new training data ( ``_update_base=true`` ). In the latter case, the parameter ``_max_k`` must not be larger than the original value.
.. index:: CvKNearest::find_nearest
.. _CvKNearest::find_nearest:
* Variable subsets (``var_idx``) and missing measurements are not supported.
CvKNearest::find_nearest
------------------------
.. ocv:function:: float CvKNearest::find_nearest( const Mat& _samples, int k, Mat* results=0, const float** neighbors=0, Mat* neighbor_responses=0, Mat* dist=0 ) const
Finds the neighbors and predicts responses for input vectors.
Finds the neighbors for input vectors.
.. ocv:function:: float CvKNearest::find_nearest( const Mat& samples, int k, Mat* results=0, const float** neighbors=0, Mat* neighborResponses=0, Mat* dist=0 ) const
For each input vector (a row of the matrix ``_samples`` ), the method finds the
:math:`\texttt{k} \le
\texttt{get\_max\_k()}` nearest neighbor. In case of regression,
the predicted result is a mean value of the particular vector's
neighbor responses. In case of classification, the class is determined
by voting.
.. ocv:function:: float CvKNearest::find_nearest( const Mat& samples, int k, Mat& results, Mat& neighborResponses, Mat& dists) const
For a custom classification/regression prediction, the method can optionally return pointers to the neighbor vectors themselves ( ``neighbors`` , an array of ``k*_samples->rows`` pointers), their corresponding output values ( ``neighbor_responses`` , a vector of ``k*_samples->rows`` elements), and the distances from the input vectors to the neighbors ( ``dist`` , also a vector of ``k*_samples->rows`` elements).
.. ocv:function::float CvKNearest::find_nearest( const CvMat* samples, int k, CvMat* results=0, const float** neighbors=0, CvMat* neighborResponses=0, CvMat* dist=0 ) const
.. ocv:pyfunction:: cv2.KNearest.find_nearest(samples, k[, results[, neighborResponses[, dists]]]) -> retval, results, neighborResponses, dists
:param samples: Input samples stored by rows. It is a single-precision floating-point matrix of :math:`number\_of\_samples \times number\_of\_features` size.
:param k: Number of used nearest neighbors. It must satisfy constraint: :math:`k \le` :ocv:func:`CvKNearest::get_max_k`.
:param results: Vector with results of prediction (regression or classification) for each input sample. It is a single-precision floating-point vector with ``number_of_samples`` elements.
:param neighbors: Optional output pointers to the neighbor vectors themselves. It is an array of ``k*samples->rows`` pointers.
:param neighborResponses: Optional output values for corresponding ``neighbors``. It is a single-precision floating-point matrix of :math:`number\_of\_samples \times k` size.
:param dist: Optional output distances from the input vectors to the corresponding ``neighbors``. It is a single-precision floating-point matrix of :math:`number\_of\_samples \times k` size.
For each input vector (a row of the matrix ``samples``), the method finds the ``k`` nearest neighbors. In case of regression, the predicted result is a mean value of the particular vector's neighbor responses. In case of classification, the class is determined by voting.
For each input vector, the neighbors are sorted by their distances to the vector.
In case of C++ interface you can use output pointers to empty matrices and the function will allocate memory itself.
If only a single input vector is passed, all output matrices are optional and the predicted value is returned by the method.
The sample below (currently using the obsolete ``CvMat`` structures) demonstrates the use of the k-nearest classifier for 2D point classification ::
CvKNearest::get_max_k
---------------------
Returns the number of maximum neighbors that may be passed to the method :ocv:func:`CvKNearest::find_nearest`.
.. ocv:function:: int CvKNearest::get_max_k() const
CvKNearest::get_var_count
-------------------------
Returns the number of used features (variables count).
.. ocv:function:: int CvKNearest::get_var_count() const
CvKNearest::get_sample_count
----------------------------
Returns the total number of train samples.
.. ocv:function:: int CvKNearest::get_sample_count() const
CvKNearest::is_regression
-------------------------
Returns type of the problem: ``true`` for regression and ``false`` for classification.
.. ocv:function:: bool CvKNearest::is_regression() const
The sample below (currently using the obsolete ``CvMat`` structures) demonstrates the use of the k-nearest classifier for 2D point classification: ::
#include "ml.h"
#include "highgui.h"
modules/ml/doc/mldata.rst
+88 -29
View File
@@ -3,13 +3,13 @@ MLData
.. highlight:: cpp
For the machine learning algorithms usage it is often that data set is saved in file of format like .csv. The supported format file must contains the table of predictors and responses values, each row of the table must correspond to one sample. Missing values are supported. Famous UC Irvine Machine Learning Repository (http://archive.ics.uci.edu/ml/) provides many stored in such format data sets to the machine learning community. The class MLData has been implemented to ease the loading data for the training one of the existing in OpenCV machine learning algorithm. For float values only separator ``'.'`` is supported.
For the machine learning algorithms, the data set is often stored in a file of the ``.csv``-like format. The file contains a table of predictor and response values where each row of the table corresponds to a sample. Missing values are supported. The UC Irvine Machine Learning Repository (http://archive.ics.uci.edu/ml/) provides many data sets stored in such a format to the machine learning community. The class ``MLData`` is implemented to easily load the data for training one of the OpenCV machine learning algorithms. For float values, only the ``'.'`` separator is supported.
CvMLData
--------
.. ocv:class:: CvMLData
The class to load the data from .csv file.
Class for loading the data from a ``.csv`` file.
::
class CV_EXPORTS CvMLData
@@ -56,135 +56,190 @@ The class to load the data from .csv file.
CvMLData::read_csv
------------------
Reads the data set from a ``.csv``-like ``filename`` file and stores all read values in a matrix.
.. ocv:function:: int CvMLData::read_csv(const char* filename);
This method reads the data set from .csv-like file named ``filename`` and store all read values in one matrix. While reading the method tries to define variables (predictors and response) type: ordered or categorical. If some value of the variable is not a number (e.g. contains the letters) exept a label for missing value, then the type of the variable is set to ``CV_VAR_CATEGORICAL``. If all unmissing values of the variable are the numbers, then the type of the variable is set to ``CV_VAR_ORDERED``. So default definition of variables types works correctly for all cases except the case of categorical variable that has numerical class labeles. In such case the type ``CV_VAR_ORDERED`` will be set and user should change the type to ``CV_VAR_CATEGORICAL`` using method :ocv:func:`CvMLData::change_var_type`. For categorical variables the common map is built to convert string class label to the numerical class label and this map can be got by :ocv:func:`CvMLData::get_class_labels_map`. Also while reading the data the method constructs the mask of missing values (e.g. values are egual to `'?'`).
:param filename: The input file name
While reading the data, the method tries to define the type of variables (predictors and responses): ordered or categorical. If a value of the variable is not numerical (except for the label for a missing value), the type of the variable is set to ``CV_VAR_CATEGORICAL``. If all existing values of the variable are numerical, the type of the variable is set to ``CV_VAR_ORDERED``. So, the default definition of variables types works correctly for all cases except the case of a categorical variable with numerical class labeles. In this case, the type ``CV_VAR_ORDERED`` is set. You should change the type to ``CV_VAR_CATEGORICAL`` using the method :ocv:func:`CvMLData::change_var_type`. For categorical variables, a common map is built to convert a string class label to the numerical class label. Use :ocv:func:`CvMLData::get_class_labels_map` to obtain this map.
Also, when reading the data, the method constructs the mask of missing values. For example, values are egual to `'?'`.
CvMLData::get_values
--------------------
Returns a pointer to the matrix of predictors and response values
.. ocv:function:: const CvMat* CvMLData::get_values() const;
Returns the pointer to the predictors and responses ``values`` matrix or ``0`` if data has not been loaded from file yet. This matrix has rows count equal to samples count, columns count equal to predictors ``+ 1`` for response (if exist) count (i.e. each row of matrix is values of one sample predictors and response) and type ``CV_32FC1``.
The method returns a pointer to the matrix of predictor and response ``values`` or ``0`` if the data has not been loaded from the file yet.
The row count of this matrix equals the sample count. The column count equals predictors ``+ 1`` for the response (if exists) count. This means that each row of the matrix contains values of one sample predictor and response. The matrix type is ``CV_32FC1``.
CvMLData::get_responses
-----------------------
Returns a pointer to the matrix of response values
.. ocv:function:: const CvMat* CvMLData::get_responses();
Returns the pointer to the responses values matrix or throw exception if data has not been loaded from file yet. This matrix has rows count equal to samples count, one column and type ``CV_32FC1``.
The method returns a pointer to the matrix of response values or throws an exception if the data has not been loaded from the file yet.
This is a single-column matrix of the type ``CV_32FC1``. Its row count is equal to the sample count, one column and .
CvMLData::get_missing
---------------------
Returns a pointer to the mask matrix of missing values
.. ocv:function:: const CvMat* CvMLData::get_missing() const;
Returns the pointer to the missing values mask matrix or throw exception if data has not been loaded from file yet. This matrix has the same size as ``values`` matrix (see :ocv:func:`CvMLData::get_values`) and type ``CV_8UC1``.
The method returns a pointer to the mask matrix of missing values or throws an exception if the data has not been loaded from the file yet.
This matrix has the same size as the ``values`` matrix (see :ocv:func:`CvMLData::get_values`) and the type ``CV_8UC1``.
CvMLData::set_response_idx
--------------------------
Specifies index of response column in the data matrix
.. ocv:function:: void CvMLData::set_response_idx( int idx );
Sets index of response column in ``values`` matrix (see :ocv:func:`CvMLData::get_values`) or throw exception if data has not been loaded from file yet. The old response column become pridictors. If ``idx < 0`` there will be no response.
The method sets the index of a response column in the ``values`` matrix (see :ocv:func:`CvMLData::get_values`) or throws an exception if the data has not been loaded from the file yet.
The old response columns become predictors. If ``idx < 0``, there is no response.
CvMLData::get_response_idx
----------
--------------------------
Returns index of the response column in the loaded data matrix
.. ocv:function:: int CvMLData::get_response_idx() const;
Gets response column index in ``values`` matrix (see :ocv:func:`CvMLData::get_values`), negative value there is no response or throw exception if data has not been loaded from file yet.
The method returns the index of a response column in the ``values`` matrix (see :ocv:func:`CvMLData::get_values`) or throws an exception if the data has not been loaded from the file yet.
If ``idx < 0``, there is no response.
CvMLData::set_train_test_split
------------------------------
Divides the read data set into two disjoint training and test subsets.
.. ocv:function:: void CvMLData::set_train_test_split( const CvTrainTestSplit * spl );
For different purposes it can be useful to devide the read data set into two disjoint subsets: training and test ones. This method sets parametes for such split (using ``spl``, see :ocv:class:`CvTrainTestSplit`) and make the data split or throw exception if data has not been loaded from file yet.
This method sets parameters for such a split using ``spl`` (see :ocv:class:`CvTrainTestSplit`) or throws an exception if the data has not been loaded from the file yet.
CvMLData::get_train_sample_idx
------------------------------
Returns the matrix of sample indices for a training subset
.. ocv:function:: const CvMat* CvMLData::get_train_sample_idx() const;
The read data set can be devided on training and test data subsets by setting split (see :ocv:func:`CvMLData::set_train_test_split`). Current method returns the matrix of samples indices for training subset (this matrix has one row and type ``CV_32SC1``). If data split is not set then the method returns ``0``. If data has not been loaded from file yet an exception is thrown.
The method returns the matrix of sample indices for a training subset. This is a single-row matrix of the type ``CV_32SC1``. If data split is not set, the method returns ``0``. If the data has not been loaded from the file yet, an exception is thrown.
CvMLData::get_test_sample_idx
-----------------------------
Returns the matrix of sample indices for a testing subset
.. ocv:function:: const CvMat* CvMLData::get_test_sample_idx() const;
Analogically with :ocv:func:`CvMLData::get_train_sample_idx`, but for test subset.
CvMLData::mix_train_and_test_idx
--------------------------------
Mixes the indices of training and test samples
.. ocv:function:: void CvMLData::mix_train_and_test_idx();
Mixes the indices of training and test samples preserving sizes of training and test subsets (if data split is set by :ocv:func:`CvMLData::get_values`). If data has not been loaded from file yet an exception is thrown.
The method shuffles the indices of training and test samples preserving sizes of training and test subsets if the data split is set by :ocv:func:`CvMLData::get_values`. If the data has not been loaded from the file yet, an exception is thrown.
CvMLData::get_var_idx
---------------------
Returns the indices of the active variables in the data matrix
.. ocv:function:: const CvMat* CvMLData::get_var_idx();
Returns used variables (columns) indices in the ``values`` matrix (see :ocv:func:`CvMLData::get_values`), ``0`` if used subset is not set or throw exception if data has not been loaded from file yet. Returned matrix has one row, columns count equel to used variable subset size and type ``CV_32SC1``.
The method returns the indices of variables (columns) used in the ``values`` matrix (see :ocv:func:`CvMLData::get_values`).
It returns ``0`` if the used subset is not set. It throws an exception if the data has not been loaded from the file yet. Returned matrix is a single-row matrix of the type ``CV_32SC1``. Its column count is equal to the size of the used variable subset.
CvMLData::chahge_var_idx
------------------------
Enables or disables particular variable in the loaded data
.. ocv:function:: void CvMLData::chahge_var_idx( int vi, bool state );
By default after reading the data set all variables in ``values`` matrix (see :ocv:func:`CvMLData::get_values`) are used. But the user may want to use only subset of variables and can include on/off (depends on ``state`` value) a variable with ``vi`` index from used subset. If data has not been loaded from file yet an exception is thrown.
By default, after reading the data set all variables in the ``values`` matrix (see :ocv:func:`CvMLData::get_values`) are used. But you may want to use only a subset of variables and include/exclude (depending on ``state`` value) a variable with the ``vi`` index from the used subset. If the data has not been loaded from the file yet, an exception is thrown.
CvMLData::get_var_types
-----------------------
Returns a matrix of the variable types.
.. ocv:function:: const CvMat* CvMLData::get_var_types();
Returns matrix of used variable types. The matrix has one row, column count equel to used variables count and type ``CV_8UC1``. If data has not been loaded from file yet an exception is thrown.
The function returns a single-row matrix of the type ``CV_8UC1``, where each element is set to either ``CV_VAR_ORDERED`` or ``CV_VAR_CATEGORICAL``. The number of columns is equal to the number of variables. If data has not been loaded from file yet an exception is thrown.
CvMLData::set_var_types
-----------------------
Sets the variables types in the loaded data.
.. ocv:function:: void CvMLData::set_var_types( const char* str );
Sets variables types according to given string ``str``. The better description of the supporting string format is several examples of it: ``"ord[0-17],cat[18]"``, ``"ord[0,2,4,10-12], cat[1,3,5-9,13,14]"``, ``"cat"`` (all variables are categorical), ``"ord"`` (all variables are ordered). That is after the variable type a list of such type variables indices is followed.
In the string, a variable type is followed by a list of variables indices. For example: ``"ord[0-17],cat[18]"``, ``"ord[0,2,4,10-12], cat[1,3,5-9,13,14]"``, ``"cat"`` (all variables are categorical), ``"ord"`` (all variables are ordered).
CvMLData::get_var_type
----------------------
Returns type of the specified variable
.. ocv:function:: int CvMLData::get_var_type( int var_idx ) const;
Returns type of variable by index ``var_idx`` ( ``CV_VAR_ORDERED`` or ``CV_VAR_CATEGORICAL``).
The method returns the type of a variable by the index ``var_idx`` ( ``CV_VAR_ORDERED`` or ``CV_VAR_CATEGORICAL``).
CvMLData::change_var_type
-------------------------
Changes type of the specified variable
.. ocv:function:: void CvMLData::change_var_type( int var_idx, int type);
Changes type of variable with index ``var_idx`` from existing type to ``type`` ( ``CV_VAR_ORDERED`` or ``CV_VAR_CATEGORICAL``).
The method changes type of variable with index ``var_idx`` from existing type to ``type`` ( ``CV_VAR_ORDERED`` or ``CV_VAR_CATEGORICAL``).
CvMLData::set_delimiter
-----------------------
Sets the delimiter in the file used to separate input numbers
.. ocv:function:: void CvMLData::set_delimiter( char ch );
Sets the delimiter for the variable values in file. E.g. ``','`` (default), ``';'``, ``' '`` (space) or other character (exapt float separator ``'.'``).
The method sets the delimiter for variables in a file. For example: ``','`` (default), ``';'``, ``' '`` (space), or other characters. The floating-point separator ``'.'`` is not allowed.
CvMLData::get_delimiter
-----------------------
Returns the currently used delimiter character.
.. ocv:function:: char CvMLData::get_delimiter() const;
Gets the set delimiter charecter.
CvMLData::set_miss_ch
---------------------
Sets the character used to specify missing values
.. ocv:function:: void CvMLData::set_miss_ch( char ch );
Sets the character denoting the missing of value. E.g. ``'?'`` (default), ``'-'``, etc (exapt float separator ``'.'``).
The method sets the character used to specify missing values. For example: ``'?'`` (default), ``'-'``. The floating-point separator ``'.'`` is not allowed.
CvMLData::get_miss_ch
---------------------
.. ocv:function:: char CvMLData::get_miss_ch() const;
Returns the currently used missing value character.
Gets the character denoting the missing value.
.. ocv:function:: char CvMLData::get_miss_ch() const;
CvMLData::get_class_labels_map
-------------------------------
Returns a map that converts strings to labels.
.. ocv:function:: const std::map<std::string, int>& CvMLData::get_class_labels_map() const;
Returns map that converts string class labels to the numerical class labels. It can be used to get original class label (as in file).
The method returns a map that converts string class labels to the numerical class labels. It can be used to get an original class label as in a file.
CvTrainTestSplit
----------------
.. ocv:class:: CvTrainTestSplit
The structure to set split of data set read by :ocv:class:`CvMLData`.
Structure setting the split of a data set read by :ocv:class:`CvMLData`.
::
struct CvTrainTestSplit
@@ -203,4 +258,8 @@ The structure to set split of data set read by :ocv:class:`CvMLData`.
bool mix;
};
There are two ways to construct split. The first is by setting training sample count (subset size) ``train_sample_count``; other existing samples will be in test subset. The second is by setting training sample portion in ``[0,..1]``. The flag ``mix`` is used to mix training and test samples indices when split will be set, otherwise the data set will be devided in the storing order (first part of samples of given size is the training subset, other part is the test one).
There are two ways to construct a split:
* Set the training sample count (subset size) ``train_sample_count``. Other existing samples are located in a test subset.
* Set a training sample portion in ``[0,..1]``. The flag ``mix`` is used to mix training and test samples indices when the split is set. Otherwise, the data set is split in the storing order: the first part of samples of a given size is a training subset, the second part is a test subset.
modules/ml/doc/neural_networks.rst
+155 -148
View File
@@ -1,7 +1,9 @@
Neural Networks
===============
ML implements feed-forward artificial neural networks, more particularly, multi-layer perceptrons (MLP), the most commonly used type of neural networks. MLP consists of the input layer, output layer, and one or more hidden layers. Each layer of MLP includes one or more neurons that are directionally linked with the neurons from the previous and the next layer. The example below represents a 3-layer perceptron with three inputs, two outputs, and the hidden layer including five neurons:
.. highlight:: cpp
ML implements feed-forward artificial neural networks or, more particularly, multi-layer perceptrons (MLP), the most commonly used type of neural networks. MLP consists of the input layer, output layer, and one or more hidden layers. Each layer of MLP includes one or more neurons directionally linked with the neurons from the previous and the next layer. The example below represents a 3-layer perceptron with three inputs, two outputs, and the hidden layer including five neurons:
.. image:: pics/mlp.png
@@ -45,10 +47,13 @@ In ML, all the neurons have the same activation functions, with the same free pa
So, the whole trained network works as follows:
#. It takes the feature vector as input. The vector size is equal to the size of the input layer.
#. Values are passed as input to the first hidden layer.
#. Outputs of the hidden layer are computed using the weights and the activation functions.
#. Outputs are passed further downstream until you compute the output layer.
#. Take the feature vector as input. The vector size is equal to the size of the input layer.
#. Pass values as input to the first hidden layer.
#. Compute outputs of the hidden layer using the weights and the activation functions.
#. Pass outputs further downstream until you compute the output layer.
So, to compute the network, you need to know all the
weights
@@ -66,10 +71,10 @@ so the error on the test set usually starts increasing after the network
size reaches a limit. Besides, the larger networks are trained much
longer than the smaller ones, so it is reasonable to pre-process the data,
using
:ref:`PCA::operator ()` or similar technique, and train a smaller network
:ocv:func:`PCA::operator ()` or similar technique, and train a smaller network
on only essential features.
Another feature of MLP's is their inability to handle categorical
Another MPL feature is an inability to handle categorical
data as is. However, there is a workaround. If a certain feature in the
input or output (in case of ``n`` -class classifier for
:math:`n>2` ) layer is categorical and can take
@@ -83,187 +88,189 @@ ML implements two algorithms for training MLP's. The first algorithm is a classi
random sequential back-propagation algorithm.
The second (default) one is a batch RPROP algorithm.
References:
.. [BackPropWikipedia] http://en.wikipedia.org/wiki/Backpropagation. Wikipedia article about the back-propagation algorithm.
*
http://en.wikipedia.org/wiki/Backpropagation
. Wikipedia article about the back-propagation algorithm.
.. [LeCun98] Y. LeCun, L. Bottou, G.B. Orr and K.-R. Muller, *Efficient backprop*, in Neural Networks---Tricks of the Trade, Springer Lecture Notes in Computer Sciences 1524, pp.5-50, 1998.
*
Y. LeCun, L. Bottou, G.B. Orr and K.-R. Muller, *Efficient backprop*, in Neural Networks---Tricks of the Trade, Springer Lecture Notes in Computer Sciences 1524, pp.5-50, 1998.
*
M. Riedmiller and H. Braun, *A Direct Adaptive Method for Faster Backpropagation Learning: The RPROP Algorithm*, Proc. ICNN, San Francisco (1993).
.. index:: CvANN_MLP_TrainParams
.. _CvANN_MLP_TrainParams:
.. [RPROP93] M. Riedmiller and H. Braun, *A Direct Adaptive Method for Faster Backpropagation Learning: The RPROP Algorithm*, Proc. ICNN, San Francisco (1993).
CvANN_MLP_TrainParams
---------------------
.. c:type:: CvANN_MLP_TrainParams
.. ocv:class:: CvANN_MLP_TrainParams
Parameters of the MLP training algorithm ::
Parameters of the MLP training algorithm. You can initialize the structure by a constructor or the individual parameters can be adjusted after the structure is created.
struct CvANN_MLP_TrainParams
The back-propagation algorithm parameters:
.. ocv:member:: double bp_dw_scale
Strength of the weight gradient term. The recommended value is about 0.1.
.. ocv:member:: double bp_moment_scale
Strength of the momentum term (the difference between weights on the 2 previous iterations). This parameter provides some inertia to smooth the random fluctuations of the weights. It can vary from 0 (the feature is disabled) to 1 and beyond. The value 0.1 or so is good enough
The RPROP algorithm parameters (see [RPROP93]_ for details):
.. ocv:member:: double rp_dw0
Initial value :math:`\Delta_0` of update-values :math:`\Delta_{ij}`.
.. ocv:member:: double rp_dw_plus
Increase factor :math:`\eta^+`. It must be >1.
.. ocv:member:: double rp_dw_minus
Decrease factor :math:`\eta^-`. It must be <1.
.. ocv:member:: double rp_dw_min
Update-values lower limit :math:`\Delta_{min}`. It must be positive.
.. ocv:member:: double rp_dw_max
Update-values upper limit :math:`\Delta_{max}`. It must be >1.
CvANN_MLP_TrainParams::CvANN_MLP_TrainParams
--------------------------------------------
The constructors.
.. ocv:function:: CvANN_MLP_TrainParams::CvANN_MLP_TrainParams()
.. ocv:function:: CvANN_MLP_TrainParams::CvANN_MLP_TrainParams( CvTermCriteria term_crit, int train_method, double param1, double param2=0 )
:param term_crit: Termination criteria of the training algorithm. You can specify the maximum number of iterations (``max_iter``) and/or how much the error could change between the iterations to make the algorithm continue (``epsilon``).
:param train_method: Training method of the MLP. Possible values are:
* **CvANN_MLP_TrainParams::BACKPROP** The back-propagation algorithm.
* **CvANN_MLP_TrainParams::RPROP** The RPROP algorithm.
:param param1: Parameter of the training method. It is ``rp_dw0`` for ``RPROP`` and ``bp_dw_scale`` for ``BACKPROP``.
:param param2: Parameter of the training method. It is ``rp_dw_min`` for ``RPROP`` and ``bp_moment_scale`` for ``BACKPROP``.
By default the RPROP algorithm is used:
::
CvANN_MLP_TrainParams::CvANN_MLP_TrainParams()
{
CvANN_MLP_TrainParams();
CvANN_MLP_TrainParams( CvTermCriteria term_crit, int train_method,
double param1, double param2=0 );
~CvANN_MLP_TrainParams();
enum { BACKPROP=0, RPROP=1 };
CvTermCriteria term_crit;
int train_method;
// back-propagation parameters
double bp_dw_scale, bp_moment_scale;
// rprop parameters
double rp_dw0, rp_dw_plus, rp_dw_minus, rp_dw_min, rp_dw_max;
};
The structure has a default constructor that initializes parameters for the ``RPROP`` algorithm. There is also a more advanced constructor to customize the parameters and/or choose the back-propagation algorithm. Finally, the individual parameters can be adjusted after the structure is created.
.. index:: CvANN_MLP
.. _CvANN_MLP:
term_crit = cvTermCriteria( CV_TERMCRIT_ITER + CV_TERMCRIT_EPS, 1000, 0.01 );
train_method = RPROP;
bp_dw_scale = bp_moment_scale = 0.1;
rp_dw0 = 0.1; rp_dw_plus = 1.2; rp_dw_minus = 0.5;
rp_dw_min = FLT_EPSILON; rp_dw_max = 50.;
}
CvANN_MLP
---------
.. c:type:: CvANN_MLP
.. ocv:class:: CvANN_MLP
MLP model ::
MLP model.
class CvANN_MLP : public CvStatModel
{
public:
CvANN_MLP();
CvANN_MLP( const Mat& _layer_sizes,
int _activ_func=SIGMOID_SYM,
double _f_param1=0, double _f_param2=0 );
virtual ~CvANN_MLP();
virtual void create( const Mat& _layer_sizes,
int _activ_func=SIGMOID_SYM,
double _f_param1=0, double _f_param2=0 );
virtual int train( const Mat& _inputs, const Mat& _outputs,
const Mat& _sample_weights,
const Mat& _sample_idx=Mat(),
CvANN_MLP_TrainParams _params = CvANN_MLP_TrainParams(),
int flags=0 );
virtual float predict( const Mat& _inputs,
Mat& _outputs ) const;
virtual void clear();
// possible activation functions
enum { IDENTITY = 0, SIGMOID_SYM = 1, GAUSSIAN = 2 };
// available training flags
enum { UPDATE_WEIGHTS = 1, NO_INPUT_SCALE = 2, NO_OUTPUT_SCALE = 4 };
virtual void read( CvFileStorage* fs, CvFileNode* node );
virtual void write( CvFileStorage* storage, const char* name );
int get_layer_count() { return layer_sizes ? layer_sizes->cols : 0; }
const Mat& get_layer_sizes() { return layer_sizes; }
protected:
virtual bool prepare_to_train( const Mat& _inputs, const Mat& _outputs,
const Mat& _sample_weights, const Mat& _sample_idx,
CvANN_MLP_TrainParams _params,
CvVectors* _ivecs, CvVectors* _ovecs, double** _sw, int _flags );
// sequential random backpropagation
virtual int train_backprop( CvVectors _ivecs, CvVectors _ovecs,
const double* _sw );
// RPROP algorithm
virtual int train_rprop( CvVectors _ivecs, CvVectors _ovecs,
const double* _sw );
virtual void calc_activ_func( Mat& xf, const double* bias ) const;
virtual void calc_activ_func_deriv( Mat& xf, Mat& deriv,
const double* bias ) const;
virtual void set_activ_func( int _activ_func=SIGMOID_SYM,
double _f_param1=0, double _f_param2=0 );
virtual void init_weights();
virtual void scale_input( const Mat& _src, Mat& _dst ) const;
virtual void scale_output( const Mat& _src, Mat& _dst ) const;
virtual void calc_input_scale( const CvVectors* vecs, int flags );
virtual void calc_output_scale( const CvVectors* vecs, int flags );
virtual void write_params( CvFileStorage* fs );
virtual void read_params( CvFileStorage* fs, CvFileNode* node );
Mat& layer_sizes;
Mat& wbuf;
Mat& sample_weights;
double** weights;
double f_param1, f_param2;
double min_val, max_val, min_val1, max_val1;
int activ_func;
int max_count, max_buf_sz;
CvANN_MLP_TrainParams params;
CvRNG rng;
};
Unlike many other models in ML that are constructed and trained at once, in the MLP model these steps are separated. First, a network with the specified topology is created using the non-default constructor or the method :ocv:func:`CvANN_MLP::create`. All the weights are set to zeros. Then, the network is trained using a set of input and output vectors. The training procedure can be repeated more than once, that is, the weights can be adjusted based on the new training data.
Unlike many other models in ML that are constructed and trained at once, in the MLP model these steps are separated. First, a network with the specified topology is created using the non-default constructor or the method ``create`` . All the weights are set to zeros. Then, the network is trained using a set of input and output vectors. The training procedure can be repeated more than once, that is, the weights can be adjusted based on the new training data.
CvANN_MLP::CvANN_MLP
--------------------
The constructors.
.. index:: CvANN_MLP::create
.. ocv:function:: CvANN_MLP::CvANN_MLP()
.. _CvANN_MLP::create:
.. ocv:function::CvANN_MLP::CvANN_MLP( const CvMat* layerSizes, int activateFunc=CvANN_MLP::SIGMOID_SYM, double fparam1=0, double fparam2=0 )
.. ocv:pyfunction:: cv2.ANN_MLP(layerSizes[, activateFunc[, fparam1[, fparam2]]]) -> <ANN_MLP object>
The advanced constructor allows to create MLP with the specified topology. See :ocv:func:`CvANN_MLP::create` for details.
CvANN_MLP::create
-----------------
.. ocv:function:: void CvANN_MLP::create( const Mat& _layer_sizes, int _activ_func=SIGMOID_SYM, double _f_param1=0, double _f_param2=0 )
Constructs MLP with the specified topology.
Constructs MLP with the specified topology.
.. ocv:function:: void CvANN_MLP::create( const Mat& layerSizes, int activateFunc=CvANN_MLP::SIGMOID_SYM, double fparam1=0, double fparam2=0 )
:param _layer_sizes: Integer vector specifying the number of neurons in each layer including the input and output layers.
.. ocv:function::void CvANN_MLP::create( const CvMat* layerSizes, int activateFunc=CvANN_MLP::SIGMOID_SYM, double fparam1=0, double fparam2=0 )
:param _activ_func: Parameter specifying the activation function for each neuron: one of ``CvANN_MLP::IDENTITY`` , ``CvANN_MLP::SIGMOID_SYM`` , and ``CvANN_MLP::GAUSSIAN`` .
.. ocv:pyfunction:: cv2.ANN_MLP.create(layerSizes[, activateFunc[, fparam1[, fparam2]]]) -> None
:param _f_param1,_f_param2: Free parameters of the activation function, :math:`\alpha` and :math:`\beta` , respectively. See the formulas in the introduction section.
:param layerSizes: Integer vector specifying the number of neurons in each layer including the input and output layers.
:param activateFunc: Parameter specifying the activation function for each neuron: one of ``CvANN_MLP::IDENTITY``, ``CvANN_MLP::SIGMOID_SYM``, and ``CvANN_MLP::GAUSSIAN``.
:param fparam1/fparam2: Free parameters of the activation function, :math:`\alpha` and :math:`\beta`, respectively. See the formulas in the introduction section.
The method creates an MLP network with the specified topology and assigns the same activation function to all the neurons.
.. index:: CvANN_MLP::train
.. _CvANN_MLP::train:
CvANN_MLP::train
----------------
.. ocv:function:: int CvANN_MLP::train( const Mat& _inputs, const Mat& _outputs, const Mat& _sample_weights, const Mat& _sample_idx=Mat(), CvANN_MLP_TrainParams _params = CvANN_MLP_TrainParams(), int flags=0 )
Trains/updates MLP.
Trains/updates MLP.
.. ocv:function:: int CvANN_MLP::train( const Mat& inputs, const Mat& outputs, const Mat& sampleWeights, const Mat& sampleIdx=Mat(), CvANN_MLP_TrainParams params = CvANN_MLP_TrainParams(), int flags=0 )
:param _inputs: Floating-point matrix of input vectors, one vector per row.
.. ocv:function::int CvANN_MLP::train( const CvMat* inputs, const CvMat* outputs, const CvMat* sampleWeights, const CvMat* sampleIdx=0, CvANN_MLP_TrainParams params = CvANN_MLP_TrainParams(), int flags=0 )
:param _outputs: Floating-point matrix of the corresponding output vectors, one vector per row.
.. ocv:pyfunction:: cv2.ANN_MLP.train(inputs, outputs, sampleWeights[, sampleIdx[, params[, flags]]]) -> niterations
:param _sample_weights: (RPROP only) Optional floating-point vector of weights for each sample. Some samples may be more important than others for training. You may want to raise the weight of certain classes to find the right balance between hit-rate and false-alarm rate, and so on.
:param inputs: Floating-point matrix of input vectors, one vector per row.
:param _sample_idx: Optional integer vector indicating the samples (rows of ``_inputs`` and ``_outputs`` ) that are taken into account.
:param outputs: Floating-point matrix of the corresponding output vectors, one vector per row.
:param _params: Training parameters. See the ``CvANN_MLP_TrainParams`` description.
:param sampleWeights: (RPROP only) Optional floating-point vector of weights for each sample. Some samples may be more important than others for training. You may want to raise the weight of certain classes to find the right balance between hit-rate and false-alarm rate, and so on.
:param _flags: Various parameters to control the training algorithm. A combination of the following parameters is possible:
:param sampleIdx: Optional integer vector indicating the samples (rows of ``inputs`` and ``outputs``) that are taken into account.
* **UPDATE_WEIGHTS = 1** Algorithm updates the network weights, rather than computes them from scratch (in the latter case the weights are initialized using the Nguyen-Widrow algorithm).
:param params: Training parameters. See the :ocv:class:`CvANN_MLP_TrainParams` description.
* **NO_INPUT_SCALE** Algorithm does not normalize the input vectors. If this flag is not set, the training algorithm normalizes each input feature independently, shifting its mean value to 0 and making the standard deviation =1. If the network is assumed to be updated frequently, the new training data could be much different from original one. In this case, you should take care of proper normalization.
:param flags: Various parameters to control the training algorithm. A combination of the following parameters is possible:
* **UPDATE_WEIGHTS** Algorithm updates the network weights, rather than computes them from scratch. In the latter case the weights are initialized using the Nguyen-Widrow algorithm.
* **NO_INPUT_SCALE** Algorithm does not normalize the input vectors. If this flag is not set, the training algorithm normalizes each input feature independently, shifting its mean value to 0 and making the standard deviation equal to 1. If the network is assumed to be updated frequently, the new training data could be much different from original one. In this case, you should take care of proper normalization.
* **NO_OUTPUT_SCALE** Algorithm does not normalize the output vectors. If the flag is not set, the training algorithm normalizes each output feature independently, by transforming it to the certain range depending on the used activation function.
This method applies the specified training algorithm to computing/adjusting the network weights. It returns the number of done iterations.
CvANN_MLP::predict
------------------
Predicts responses for input samples.
.. ocv:function:: float CvANN_MLP::predict( const Mat& inputs, Mat& outputs ) const
.. ocv:function::float CvANN_MLP::predict( const CvMat* inputs, CvMat* outputs ) const
.. ocv:pyfunction:: cv2.ANN_MLP.predict(inputs, outputs) -> retval
:param inputs: Input samples.
:param outputs: Predicted responses for corresponding samples.
The method returns a dummy value which should be ignored.
CvANN_MLP::get_layer_count
--------------------------
Returns the number of layers in the MLP.
.. ocv:function:: int CvANN_MLP::get_layer_count()
CvANN_MLP::get_layer_sizes
--------------------------
Returns numbers of neurons in each layer of the MLP.
.. ocv:function::const CvMat* CvANN_MLP::get_layer_sizes()
The method returns the integer vector specifying the number of neurons in each layer including the input and output layers of the MLP.
CvANN_MLP::get_weights
----------------------
Returns neurons weights of the particular layer.
.. ocv:function:: double* CvANN_MLP::get_weights(int layer)
:param layer: Index of the particular layer.
modules/ml/doc/normal_bayes_classifier.rst
+29 -40
View File
@@ -3,71 +3,60 @@
Normal Bayes Classifier
=======================
This is a simple classification model assuming that feature vectors from each class are normally distributed (though, not necessarily independently distributed). So, the whole data distribution function is assumed to be a Gaussian mixture, one component per class. Using the training data the algorithm estimates mean vectors and covariance matrices for every class, and then it uses them for prediction.
.. highlight:: cpp
[Fukunaga90] K. Fukunaga. *Introduction to Statistical Pattern Recognition*. second ed., New York: Academic Press, 1990.
This simple classification model assumes that feature vectors from each class are normally distributed (though, not necessarily independently distributed). So, the whole data distribution function is assumed to be a Gaussian mixture, one component per class. Using the training data the algorithm estimates mean vectors and covariance matrices for every class, and then it uses them for prediction.
.. index:: CvNormalBayesClassifier
.. [Fukunaga90] K. Fukunaga. *Introduction to Statistical Pattern Recognition*. second ed., New York: Academic Press, 1990.
CvNormalBayesClassifier
-----------------------
.. c:type:: CvNormalBayesClassifier
.. ocv:class:: CvNormalBayesClassifier
Bayes classifier for normally distributed data ::
Bayes classifier for normally distributed data.
class CvNormalBayesClassifier : public CvStatModel
{
public:
CvNormalBayesClassifier();
virtual ~CvNormalBayesClassifier();
CvNormalBayesClassifier::CvNormalBayesClassifier
------------------------------------------------
Default and training constructors.
CvNormalBayesClassifier( const Mat& _train_data, const Mat& _responses,
const Mat& _var_idx=Mat(), const Mat& _sample_idx=Mat() );
.. ocv:function:: CvNormalBayesClassifier::CvNormalBayesClassifier()
virtual bool train( const Mat& _train_data, const Mat& _responses,
const Mat& _var_idx=Mat(), const Mat& _sample_idx=Mat(), bool update=false );
.. ocv:function:: CvNormalBayesClassifier::CvNormalBayesClassifier( const Mat& trainData, const Mat& responses, const Mat& varIdx=Mat(), const Mat& sampleIdx=Mat() )
virtual float predict( const Mat& _samples, Mat* results=0 ) const;
virtual void clear();
.. ocv:function::CvNormalBayesClassifier::CvNormalBayesClassifier( const CvMat* trainData, const CvMat* responses, const CvMat* varIdx=0, const CvMat* sampleIdx=0 )
virtual void save( const char* filename, const char* name=0 );
virtual void load( const char* filename, const char* name=0 );
.. ocv:pyfunction:: cv2.NormalBayesClassifier(trainData, responses[, varIdx[, sampleIdx]]) -> <NormalBayesClassifier object>
virtual void write( CvFileStorage* storage, const char* name );
virtual void read( CvFileStorage* storage, CvFileNode* node );
protected:
...
};
.. index:: CvNormalBayesClassifier::train
.. _CvNormalBayesClassifier::train:
The constructors follow conventions of :ocv:func:`CvStatModel::CvStatModel`. See :ocv:func:`CvStatModel::train` for parameters descriptions.
CvNormalBayesClassifier::train
------------------------------
.. ocv:function:: bool CvNormalBayesClassifier::train( const Mat& _train_data, const Mat& _responses, const Mat& _var_idx =Mat(), const Mat& _sample_idx=Mat(), bool update=false )
Trains the model.
Trains the model.
.. ocv:function:: bool CvNormalBayesClassifier::train( const Mat& trainData, const Mat& responses, const Mat& varIdx = Mat(), const Mat& sampleIdx=Mat(), bool update=false )
The method trains the Normal Bayes classifier. It follows the conventions of the generic ``train`` "method" with the following limitations:
.. ocv:function::bool CvNormalBayesClassifier::train( const CvMat* trainData, const CvMat* responses, const CvMat* varIdx = 0, const CvMat* sampleIdx=0, bool update=false )
.. ocv:pyfunction:: cv2.NormalBayesClassifier.train(trainData, responses[, varIdx[, sampleIdx[, update]]]) -> retval
:param update: Identifies whether the model should be trained from scratch (``update=false``) or should be updated using the new training data (``update=true``).
The method trains the Normal Bayes classifier. It follows the conventions of the generic :ocv:func:`CvStatModel::train` approach with the following limitations:
* Only ``CV_ROW_SAMPLE`` data layout is supported.
* Input variables are all ordered.
* Output variable is categorical , which means that elements of ``_responses`` must be integer numbers, though the vector may have the ``CV_32FC1`` type.
* Output variable is categorical , which means that elements of ``responses`` must be integer numbers, though the vector may have the ``CV_32FC1`` type.
* Missing measurements are not supported.
In addition, there is an ``update`` flag that identifies whether the model should be trained from scratch ( ``update=false`` ) or should be updated using the new training data ( ``update=true`` ).
.. index:: CvNormalBayesClassifier::predict
.. _CvNormalBayesClassifier::predict:
CvNormalBayesClassifier::predict
--------------------------------
Predicts the response for sample(s).
.. ocv:function:: float CvNormalBayesClassifier::predict( const Mat& samples, Mat* results=0 ) const
Predicts the response for sample(s).
.. ocv:function::float CvNormalBayesClassifier::predict( const CvMat* samples, CvMat* results=0 ) const
The method ``predict`` estimates the most probable classes for input vectors. Input vectors (one or more) are stored as rows of the matrix ``samples`` . In case of multiple input vectors, there should be one output vector ``results`` . The predicted class for a single input vector is returned by the method.
.. ocv:pyfunction:: cv2.NormalBayesClassifier.predict(samples) -> retval, results
The method estimates the most probable classes for input vectors. Input vectors (one or more) are stored as rows of the matrix ``samples``. In case of multiple input vectors, there should be one output vector ``results``. The predicted class for a single input vector is returned by the method.
modules/ml/doc/random_trees.rst
+142 -102
View File
@@ -3,13 +3,15 @@
Random Trees
============
.. highlight:: cpp
Random trees have been introduced by Leo Breiman and Adele Cutler:
http://www.stat.berkeley.edu/users/breiman/RandomForests/
. The algorithm can deal with both classification and regression problems. Random trees is a collection (ensemble) of tree predictors that is called
*forest*
further in this section (the term has been also introduced by L. Breiman). The classification works as follows: the random trees classifier takes the input feature vector, classifies it with every tree in the forest, and outputs the class label that recieved the majority of "votes". In case of regression, the classifier response is the average of the responses over all the trees in the forest.
further in this section (the term has been also introduced by L. Breiman). The classification works as follows: the random trees classifier takes the input feature vector, classifies it with every tree in the forest, and outputs the class label that recieved the majority of "votes". In case of a regression, the classifier response is the average of the responses over all the trees in the forest.
All the trees are trained with the same parameters but on different training sets that are generated from the original training set using the bootstrap procedure: for each training set, you randomly select the same number of vectors as in the original set ( ``=N`` ). The vectors are chosen with replacement. That is, some vectors will occur more than once and some will be absent. At each node of each trained tree, not all the variables are used to find the best split, rather than a random subset of them. With each node a new subset is generated. However, its size is fixed for all the nodes and all the trees. It is a training parameter set to
All the trees are trained with the same parameters but on different training sets. These sets are generated from the original training set using the bootstrap procedure: for each training set, you randomly select the same number of vectors as in the original set ( ``=N`` ). The vectors are chosen with replacement. That is, some vectors will occur more than once and some will be absent. At each node of each trained tree, not all the variables are used to find the best split, but a random subset of them. With each node a new subset is generated. However, its size is fixed for all the nodes and all the trees. It is a training parameter set to
:math:`\sqrt{number\_of\_variables}` by default. None of the built trees are pruned.
In random trees there is no need for any accuracy estimation procedures, such as cross-validation or bootstrap, or a separate test set to get an estimate of the training error. The error is estimated internally during the training. When the training set for the current tree is drawn by sampling with replacement, some vectors are left out (so-called
@@ -20,25 +22,28 @@ In random trees there is no need for any accuracy estimation procedures, such as
Get a prediction for each vector, which is oob relative to the i-th tree, using the very i-th tree.
#.
After all the trees have been trained, for each vector that has ever been oob, find the class-"winner" for it (the class that has got the majority of votes in the trees where the vector was oob) and compare it to the ground-truth response.
After all the trees have been trained, for each vector that has ever been oob, find the class-*winner* for it (the class that has got the majority of votes in the trees where the vector was oob) and compare it to the ground-truth response.
#.
Compute the classification error estimate as ratio of the number of misclassified oob vectors to all the vectors in the original data. In case of regression, the oob-error is computed as the squared error for oob vectors difference divided by the total number of vectors.
Compute the classification error estimate as a ratio of the number of misclassified oob vectors to all the vectors in the original data. In case of regression, the oob-error is computed as the squared error for oob vectors difference divided by the total number of vectors.
For the random trees usage example, please, see letter_recog.cpp sample in OpenCV distribution.
**References:**
*
Machine Learning, Wald I, July 2002.
*Machine Learning*, Wald I, July 2002.
http://stat-www.berkeley.edu/users/breiman/wald2002-1.pdf
*
Looking Inside the Black Box, Wald II, July 2002.
*Looking Inside the Black Box*, Wald II, July 2002.
http://stat-www.berkeley.edu/users/breiman/wald2002-2.pdf
*
Software for the Masses, Wald III, July 2002.
*Software for the Masses*, Wald III, July 2002.
http://stat-www.berkeley.edu/users/breiman/wald2002-3.pdf
@@ -47,136 +52,171 @@ In random trees there is no need for any accuracy estimation procedures, such as
http://www.stat.berkeley.edu/users/breiman/RandomForests/cc_home.htm
.
.. index:: CvRTParams
.. _CvRTParams:
CvRTParams
----------
.. c:type:: CvRTParams
Training parameters of random trees ::
struct CvRTParams : public CvDTreeParams
{
bool calc_var_importance;
int nactive_vars;
CvTermCriteria term_crit;
CvRTParams() : CvDTreeParams( 5, 10, 0, false, 10, 0, false, false, 0 ),
calc_var_importance(false), nactive_vars(0)
{
term_crit = cvTermCriteria( CV_TERMCRIT_ITER+CV_TERMCRIT_EPS, 50, 0.1 );
}
CvRTParams( int _max_depth, int _min_sample_count,
float _regression_accuracy, bool _use_surrogates,
int _max_categories, const float* _priors,
bool _calc_var_importance,
int _nactive_vars, int max_tree_count,
float forest_accuracy, int termcrit_type );
};
.. ocv:class:: CvRTParams
Training parameters of random trees.
The set of training parameters for the forest is a superset of the training parameters for a single tree. However, random trees do not need all the functionality/features of decision trees. Most noticeably, the trees are not pruned, so the cross-validation parameters are not used.
.. index:: CvRTrees
.. _CvRTrees:
CvRTParams::CvRTParams:
-----------------------
The constructors.
.. ocv:function:: CvRTParams::CvRTParams()
.. ocv:function:: CvRTParams::CvRTParams( int max_depth, int min_sample_count, float regression_accuracy, bool use_surrogates, int max_categories, const float* priors, bool calc_var_importance, int nactive_vars, int max_num_of_trees_in_the_forest, float forest_accuracy, int termcrit_type )
:param calc_var_importance: If true then variable importance will be calculated and then it can be retrieved by :ocv:func:`CvRTrees::get_var_importance`.
:param nactive_vars: The size of the randomly selected subset of features at each tree node and that are used to find the best split(s). If you set it to 0 then the size will be set to the square root of the total number of features.
:param max_num_of_trees_in_the_forest: The maximum number of trees in the forest (suprise, suprise).
:param forest_accuracy: Sufficient accuracy (OOB error).
:param termcrit_type: The type of the termination criteria:
* **CV_TERMCRIT_ITER** Terminate learning by the ``max_num_of_trees_in_the_forest``;
* **CV_TERMCRIT_EPS** Terminate learning by the ``forest_accuracy``;
* **CV_TERMCRIT_ITER | CV_TERMCRIT_EPS** Use both termination criterias.
For meaning of other parameters see :ocv:func:`CvDTreeParams::CvDTreeParams`.
The default constructor sets all parameters to default values which are different from default values of :ocv:class:`CvDTreeParams`:
::
CvRTParams::CvRTParams() : CvDTreeParams( 5, 10, 0, false, 10, 0, false, false, 0 ),
calc_var_importance(false), nactive_vars(0)
{
term_crit = cvTermCriteria( CV_TERMCRIT_ITER+CV_TERMCRIT_EPS, 50, 0.1 );
}
CvRTrees
--------
.. c:type:: CvRTrees
.. ocv:class:: CvRTrees
Random trees ::
class CvRTrees : public CvStatModel
{
public:
CvRTrees();
virtual ~CvRTrees();
virtual bool train( const Mat& _train_data, int _tflag,
const Mat& _responses, const Mat& _var_idx=Mat(),
const Mat& _sample_idx=Mat(), const Mat& _var_type=Mat(),
const Mat& _missing_mask=Mat(),
CvRTParams params=CvRTParams() );
virtual float predict( const Mat& sample, const Mat& missing = 0 )
const;
virtual void clear();
virtual const Mat& get_var_importance();
virtual float get_proximity( const Mat& sample_1, const Mat& sample_2 )
const;
virtual void read( CvFileStorage* fs, CvFileNode* node );
virtual void write( CvFileStorage* fs, const char* name );
Mat& get_active_var_mask();
CvRNG* get_rng();
int get_tree_count() const;
CvForestTree* get_tree(int i) const;
protected:
bool grow_forest( const CvTermCriteria term_crit );
// array of the trees of the forest
CvForestTree** trees;
CvDTreeTrainData* data;
int ntrees;
int nclasses;
...
};
.. index:: CvRTrees::train
.. _CvRTrees::train:
The class implements the random forest predictor as described in the beginning of this section.
CvRTrees::train
---------------
.. ocv:function:: bool CvRTrees::train( const Mat& train_data, int tflag, const Mat& responses, const Mat& comp_idx=Mat(), const Mat& sample_idx=Mat(), const Mat& var_type=Mat(), const Mat& missing_mask=Mat(), CvRTParams params=CvRTParams() )
Trains the Random Trees model.
Trains the Random Tree model.
.. ocv:function:: bool CvRTrees::train( const Mat& trainData, int tflag, const Mat& responses, const Mat& varIdx=Mat(), const Mat& sampleIdx=Mat(), const Mat& varType=Mat(), const Mat& missingDataMask=Mat(), CvRTParams params=CvRTParams() )
The method ``CvRTrees::train`` is very similar to the first form of ``CvDTree::train`` () and follows the generic method ``CvStatModel::train`` conventions. All the parameters specific to the algorithm training are passed as a
:ref:`CvRTParams` instance. The estimate of the training error ( ``oob-error`` ) is stored in the protected class member ``oob_error`` .
.. ocv:function::bool CvRTrees::train( const CvMat* trainData, int tflag, const CvMat* responses, const CvMat* varIdx=0, const CvMat* sampleIdx=0, const CvMat* varType=0, const CvMat* missingDataMask=0, CvRTParams params=CvRTParams() )
.. index:: CvRTrees::predict
.. ocv:function::bool CvRTrees::train( CvMLData* data, CvRTParams params=CvRTParams() )
.. _CvRTrees::predict:
.. ocv:pyfunction:: cv2.RTrees.train(trainData, tflag, responses[, varIdx[, sampleIdx[, varType[, missingDataMask[, params]]]]]) -> retval
The method :ocv:func:`CvRTrees::train` is very similar to the method :ocv:func:`CvDTree::train` and follows the generic method :ocv:func:`CvStatModel::train` conventions. All the parameters specific to the algorithm training are passed as a :ocv:class:`CvRTParams` instance. The estimate of the training error (``oob-error``) is stored in the protected class member ``oob_error``.
CvRTrees::predict
-----------------
Predicts the output for an input sample.
.. ocv:function:: double CvRTrees::predict( const Mat& sample, const Mat& missing=Mat() ) const
Predicts the output for an input sample.
.. ocv:function::float CvRTrees::predict( const CvMat* sample, const CvMat* missing = 0 ) const
The input parameters of the prediction method are the same as in ``CvDTree::predict`` but the return value type is different. This method returns the cumulative result from all the trees in the forest (the class that receives the majority of voices, or the mean of the regression function estimates).
.. ocv:pyfunction:: cv2.RTrees.predict(sample[, missing]) -> retval
.. index:: CvRTrees::get_var_importance
:param sample: Sample for classification.
.. _CvRTrees::get_var_importance:
:param missing: Optional missing measurement mask of the sample.
CvRTrees::get_var_importance
The input parameters of the prediction method are the same as in :ocv:func:`CvDTree::predict` but the return value type is different. This method returns the cumulative result from all the trees in the forest (the class that receives the majority of voices, or the mean of the regression function estimates).
CvRTrees::predict_prob
----------------------
Returns a fuzzy-predicted class label.
.. ocv:function:: float CvRTrees::predict_prob( const cv::Mat& sample, const cv::Mat& missing = cv::Mat() ) const
.. ocv:function::float CvRTrees::predict_prob( const CvMat* sample, const CvMat* missing = 0 ) const
.. ocv:pyfunction:: cv2.RTrees.predict_prob(sample[, missing]) -> retval
:param sample: Sample for classification.
:param missing: Optional missing measurement mask of the sample.
The function works for binary classification problems only. It returns the number between 0 and 1. This number represents probability or confidence of the sample belonging to the second class. It is calculated as the proportion of decision trees that classified the sample to the second class.
CvRTrees::getVarImportance
----------------------------
.. ocv:function:: const Mat& CvRTrees::get_var_importance() const
Returns the variable importance array.
Retrieves the variable importance array.
.. ocv:function:: Mat CvRTrees::getVarImportance()
The method returns the variable importance vector, computed at the training stage when ``:ref:`CvRTParams`::calc_var_importance`` is set. If the training flag is not set, the ``NULL`` pointer is returned. This differs from the decision trees where variable importance can be computed anytime after the training.
.. ocv:function::const CvMat* CvRTrees::get_var_importance()
.. index:: CvRTrees::get_proximity
.. ocv:pyfunction:: cv2.RTrees.getVarImportance() -> importanceVector
The method returns the variable importance vector, computed at the training stage when ``CvRTParams::calc_var_importance`` is set to true. If this flag was set to false, the ``NULL`` pointer is returned. This differs from the decision trees where variable importance can be computed anytime after the training.
.. _CvRTrees::get_proximity:
CvRTrees::get_proximity
-----------------------
.. ocv:function:: float CvRTrees::get_proximity( const Mat& sample_1, const Mat& sample_2 ) const
Retrieves the proximity measure between two training samples.
Retrieves the proximity measure between two training samples.
.. ocv:function::float CvRTrees::get_proximity( const CvMat* sample1, const CvMat* sample2, const CvMat* missing1 = 0, const CvMat* missing2 = 0 ) const
The method returns proximity measure between any two samples, which is the ratio of those trees in the ensemble, in which the samples fall into the same leaf node, to the total number of the trees.
:param sample_1: The first sample.
For the random trees usage example, please, see letter_recog.cpp sample in OpenCV distribution.
:param sample_2: The second sample.
:param missing1: Optional missing measurement mask of the first sample.
:param missing2: Optional missing measurement mask of the second sample.
The method returns proximity measure between any two samples. This is a ratio of those trees in the ensemble, in which the samples fall into the same leaf node, to the total number of the trees.
CvRTrees::calc_error
--------------------
Returns error of the random forest.
.. ocv:function::float CvRTrees::calc_error( CvMLData* data, int type, std::vector<float> *resp = 0 )
The method is identical to :ocv:func:`CvDTree::calc_error` but uses the random forest as predictor.
CvRTrees::get_train_error
-------------------------
Returns the train error.
.. ocv:function:: float CvRTrees::get_train_error()
The method works for classification problems only. It returns the proportion of incorrectly classified train samples.
CvRTrees::get_rng
-----------------
Returns the state of the used random number generator.
.. ocv:function::CvRNG* CvRTrees::get_rng()
CvRTrees::get_tree_count
------------------------
Returns the number of trees in the constructed random forest.
.. ocv:function:: int CvRTrees::get_tree_count() const
CvRTrees::get_tree
------------------
Returns the specific decision tree in the constructed random forest.
.. ocv:function:: CvForestTree* CvRTrees::get_tree(int i) const
:param i: Index of the decision tree.
modules/ml/doc/statistical_models.rst
+38 -73
View File
@@ -7,9 +7,9 @@ Statistical Models
CvStatModel
-----------
.. c:type:: CvStatModel
.. ocv:class:: CvStatModel
Base class for statistical models in ML ::
Base class for statistical models in ML. ::
class CvStatModel
{
@@ -40,39 +40,27 @@ Base class for statistical models in ML ::
In this declaration, some methods are commented off. These are methods for which there is no unified API (with the exception of the default constructor). However, there are many similarities in the syntax and semantics that are briefly described below in this section, as if they are part of the base class.
.. index:: CvStatModel::CvStatModel
.. _CvStatModel::CvStatModel:
CvStatModel::CvStatModel
------------------------
The default constuctor.
.. ocv:function:: CvStatModel::CvStatModel()
Serves as a default constructor.
Each statistical model class in ML has a default constructor without parameters. This constructor is useful for a 2-stage model construction, when the default constructor is followed by ``train()`` or ``load()`` .
.. index:: CvStatModel::CvStatModel(...)
.. _CvStatModel::CvStatModel(...):
Each statistical model class in ML has a default constructor without parameters. This constructor is useful for a two-stage model construction, when the default constructor is followed by :ocv:func:`CvStatModel::train` or :ocv:func:`CvStatModel::load`.
CvStatModel::CvStatModel(...)
-----------------------------
The training constructor.
.. ocv:function:: CvStatModel::CvStatModel( const Mat& train_data ... )
Serves as a training constructor.
Most ML classes provide a single-step constructor and train constructors. This constructor is equivalent to the default constructor, followed by the ``train()`` method with the parameters that are passed to the constructor.
.. index:: CvStatModel::~CvStatModel
.. _CvStatModel::~CvStatModel:
Most ML classes provide a single-step constructor and train constructors. This constructor is equivalent to the default constructor, followed by the :ocv:func:`CvStatModel::train` method with the parameters that are passed to the constructor.
CvStatModel::~CvStatModel
-------------------------
.. ocv:function:: CvStatModel::~CvStatModel()
The virtual destructor.
Serves as a virtual destructor.
.. ocv:function:: CvStatModel::~CvStatModel()
The destructor of the base class is declared as virtual. So, it is safe to write the following code: ::
@@ -85,81 +73,62 @@ The destructor of the base class is declared as virtual. So, it is safe to write
delete model;
Normally, the destructor of each derived class does nothing. But in this instance, it calls the overridden method ``clear()`` that deallocates all the memory.
.. index:: CvStatModel::clear
.. _CvStatModel::clear:
Normally, the destructor of each derived class does nothing. But in this instance, it calls the overridden method :ocv:func:`CvStatModel::clear` that deallocates all the memory.
CvStatModel::clear
------------------
Deallocates memory and resets the model state.
.. ocv:function:: void CvStatModel::clear()
Deallocates memory and resets the model state.
The method ``clear`` does the same job as the destructor: it deallocates all the memory occupied by the class members. But the object itself is not destructed and can be reused further. This method is called from the destructor, from the ``train`` methods of the derived classes, from the methods ``load()``,``read()`` , or even explicitly by the user.
.. index:: CvStatModel::save
.. _CvStatModel::save:
The method ``clear`` does the same job as the destructor: it deallocates all the memory occupied by the class members. But the object itself is not destructed and can be reused further. This method is called from the destructor, from the :ocv:func:`CvStatModel::train` methods of the derived classes, from the methods :ocv:func:`CvStatModel::load`, :ocv:func:`CvStatModel::read()`, or even explicitly by the user.
CvStatModel::save
-----------------
Saves the model to a file.
.. ocv:function:: void CvStatModel::save( const char* filename, const char* name=0 )
Saves the model to a file.
.. ocv:pyfunction:: cv2.StatModel.save(filename[, name]) -> None
The method ``save`` saves the complete model state to the specified XML or YAML file with the specified name or default name (which depends on a particular class). *Data persistence* functionality from ``CxCore`` is used.
.. index:: CvStatModel::load
.. _CvStatModel::load:
CvStatModel::load
-----------------
Loads the model from a file.
.. ocv:function:: void CvStatModel::load( const char* filename, const char* name=0 )
Loads the model from a file.
.. ocv:pyfunction:: cv2.StatModel.load(filename[, name]) -> None
The method ``load`` loads the complete model state with the specified name (or default model-dependent name) from the specified XML or YAML file. The previous model state is cleared by ``clear()`` .
The method ``load`` loads the complete model state with the specified name (or default model-dependent name) from the specified XML or YAML file. The previous model state is cleared by :ocv:func:`CvStatModel::clear`.
.. index:: CvStatModel::write
.. _CvStatModel::write:
CvStatModel::write
------------------
Writes the model to the file storage.
.. ocv:function:: void CvStatModel::write( CvFileStorage* storage, const char* name )
Writes the model to the file storage.
The method ``write`` stores the complete model state in the file storage with the specified name or default name (which depends on the particular class). The method is called by :ocv:func:`CvStatModel::save`.
The method ``write`` stores the complete model state in the file storage with the specified name or default name (which depends on the particular class). The method is called by ``save()`` .
.. index:: CvStatModel::read
.. _CvStatModel::read:
CvStatModel::read
-----------------
.. ocv:function:: void CvStatMode::read( CvFileStorage* storage, CvFileNode* node )
Reads the model from the file storage.
Reads the model from the file storage.
.. ocv:function:: void CvStatModel::read( CvFileStorage* storage, CvFileNode* node )
The method ``read`` restores the complete model state from the specified node of the file storage. Use the function
:ref:`GetFileNodeByName` to locate the node.
:ocv:func:`GetFileNodeByName` to locate the node.
The previous model state is cleared by ``clear()`` .
.. index:: CvStatModel::train
.. _CvStatModel::train:
The previous model state is cleared by :ocv:func:`CvStatModel::clear`.
CvStatModel::train
------------------
.. ocv:function:: bool CvStatMode::train( const Mat& train_data, [int tflag,] ..., const Mat& responses, ..., [const Mat& var_idx,] ..., [const Mat& sample_idx,] ... [const Mat& var_type,] ..., [const Mat& missing_mask,] <misc_training_alg_params> ... )
Trains the model.
Trains the model.
.. ocv:function:: bool CvStatModel::train( const Mat& train_data, [int tflag,] ..., const Mat& responses, ..., [const Mat& var_idx,] ..., [const Mat& sample_idx,] ... [const Mat& var_type,] ..., [const Mat& missing_mask,] <misc_training_alg_params> ... )
The method trains the statistical model using a set of input feature vectors and the corresponding output values (responses). Both input and output vectors/values are passed as matrices. By default, the input feature vectors are stored as ``train_data`` rows, that is, all the components (features) of a training vector are stored continuously. However, some algorithms can handle the transposed representation when all values of each particular feature (component/input variable) over the whole input set are stored continuously. If both layouts are supported, the method includes the ``tflag`` parameter that specifies the orientation as follows:
@@ -167,7 +136,7 @@ The method trains the statistical model using a set of input feature vectors and
* ``tflag=CV_COL_SAMPLE`` The feature vectors are stored as columns.
The ``train_data`` must have the ``CV_32FC1`` (32-bit floating-point, single-channel) format. Responses are usually stored in the 1D vector (a row or a column) of ``CV_32SC1`` (only in the classification problem) or ``CV_32FC1`` format, one value per input vector. Although, some algorithms, like various flavors of neural nets, take vector responses.
The ``train_data`` must have the ``CV_32FC1`` (32-bit floating-point, single-channel) format. Responses are usually stored in a 1D vector (a row or a column) of ``CV_32SC1`` (only in the classification problem) or ``CV_32FC1`` format, one value per input vector. Although, some algorithms, like various flavors of neural nets, take vector responses.
For classification problems, the responses are discrete class labels. For regression problems, the responses are values of the function to be approximated. Some algorithms can deal only with classification problems, some - only with regression problems, and some can deal with both problems. In the latter case, the type of output variable is either passed as a separate parameter or as the last element of the ``var_type`` vector:
@@ -175,25 +144,21 @@ For classification problems, the responses are discrete class labels. For regres
* ``CV_VAR_ORDERED(=CV_VAR_NUMERICAL)`` The output values are ordered. This means that two different values can be compared as numbers, and this is a regression problem.
Types of input variables can be also specified using ``var_type`` . Most algorithms can handle only ordered input variables.
Types of input variables can be also specified using ``var_type``. Most algorithms can handle only ordered input variables.
Many models in the ML may be trained on a selected feature subset, and/or on a selected sample subset of the training set. To make it easier for you, the method ``train`` usually includes the ``var_idx`` and ``sample_idx`` parameters. The former parameter identifies variables (features) of interest, and the latter one identifies samples of interest. Both vectors are either integer ( ``CV_32SC1`` ) vectors (lists of 0-based indices) or 8-bit ( ``CV_8UC1`` ) masks of active variables/samples. You may pass ``NULL`` pointers instead of either of the arguments, meaning that all of the variables/samples are used for training.
Many ML models may be trained on a selected feature subset, and/or on a selected sample subset of the training set. To make it easier for you, the method ``train`` usually includes the ``var_idx`` and ``sample_idx`` parameters. The former parameter identifies variables (features) of interest, and the latter one identifies samples of interest. Both vectors are either integer (``CV_32SC1``) vectors (lists of 0-based indices) or 8-bit (``CV_8UC1``) masks of active variables/samples. You may pass ``NULL`` pointers instead of either of the arguments, meaning that all of the variables/samples are used for training.
Additionally, some algorithms can handle missing measurements, that is, when certain features of certain training samples have unknown values (for example, they forgot to measure a temperature of patient A on Monday). The parameter ``missing_mask`` , an 8-bit matrix of the same size as ``train_data`` , is used to mark the missed values (non-zero elements of the mask).
Additionally, some algorithms can handle missing measurements, that is, when certain features of certain training samples have unknown values (for example, they forgot to measure a temperature of patient A on Monday). The parameter ``missing_mask``, an 8-bit matrix of the same size as ``train_data``, is used to mark the missed values (non-zero elements of the mask).
Usually, the previous model state is cleared by ``clear()`` before running the training procedure. However, some algorithms may optionally update the model state with the new training data, instead of resetting it.
.. index:: CvStatModel::predict
.. _CvStatModel::predict:
Usually, the previous model state is cleared by :ocv:func:`CvStatModel::clear` before running the training procedure. However, some algorithms may optionally update the model state with the new training data, instead of resetting it.
CvStatModel::predict
--------------------
.. ocv:function:: float CvStatMode::predict( const Mat& sample[, <prediction_params>] ) const
Predicts the response for a sample.
Predicts the response for a sample.
.. ocv:function:: float CvStatModel::predict( const Mat& sample[, <prediction_params>] ) const
The method is used to predict the response for a new sample. In case of a classification, the method returns the class label. In case of a regression, the method returns the output function value. The input sample must have as many components as the ``train_data`` passed to ``train`` contains. If the ``var_idx`` parameter is passed to ``train`` , it is remembered and then is used to extract only the necessary components from the input sample in the method ``predict`` .
The method is used to predict the response for a new sample. In case of a classification, the method returns the class label. In case of a regression, the method returns the output function value. The input sample must have as many components as the ``train_data`` passed to ``train`` contains. If the ``var_idx`` parameter is passed to ``train``, it is remembered and then is used to extract only the necessary components from the input sample in the method ``predict``.
The suffix ``const`` means that prediction does not affect the internal model state, so the method can be safely called from within different threads.
modules/ml/doc/support_vector_machines.rst
+231 -175
View File
@@ -3,197 +3,250 @@ Support Vector Machines
.. highlight:: cpp
Originally, support vector machines (SVM) was a technique for building an optimal binary (2-class) classifier. Later the technique has been extended to regression and clustering problems. SVM is a partial case of kernel-based methods. It maps feature vectors into a higher-dimensional space using a kernel function and builds an optimal linear discriminating function in this space or an optimal hyper-plane that fits into the training data. In case of SVM, the kernel is not defined explicitly. Instead, a distance between any 2 points in the hyper-space needs to be defined.
Originally, support vector machines (SVM) was a technique for building an optimal binary (2-class) classifier. Later the technique was extended to regression and clustering problems. SVM is a partial case of kernel-based methods. It maps feature vectors into a higher-dimensional space using a kernel function and builds an optimal linear discriminating function in this space or an optimal hyper-plane that fits into the training data. In case of SVM, the kernel is not defined explicitly. Instead, a distance between any 2 points in the hyper-space needs to be defined.
The solution is optimal, which means that the margin between the separating hyper-plane and the nearest feature vectors from both classes (in case of 2-class classifier) is maximal. The feature vectors that are the closest to the hyper-plane are called "support vectors", which means that the position of other vectors does not affect the hyper-plane (the decision function).
The solution is optimal, which means that the margin between the separating hyper-plane and the nearest feature vectors from both classes (in case of 2-class classifier) is maximal. The feature vectors that are the closest to the hyper-plane are called *support vectors*, which means that the position of other vectors does not affect the hyper-plane (the decision function).
There are a lot of good references on SVM. You may consider starting with the following:
SVM implementation in OpenCV is based on [LibSVM]_.
*
[Burges98] C. Burges. *A tutorial on support vector machines for pattern recognition*, Knowledge Discovery and Data Mining 2(2), 1998.
(available online at
http://citeseer.ist.psu.edu/burges98tutorial.html
).
.. [Burges98] C. Burges. *A tutorial on support vector machines for pattern recognition*, Knowledge Discovery and Data Mining 2(2), 1998 (available online at http://citeseer.ist.psu.edu/burges98tutorial.html)
*
Chih-Chung Chang and Chih-Jen Lin. *LIBSVM - A Library for Support Vector Machines*
(
http://www.csie.ntu.edu.tw/~cjlin/libsvm/
)
.. index:: CvSVM
.. _CvSVM:
CvSVM
-----
.. c:type:: CvSVM
Support Vector Machines ::
class CvSVM : public CvStatModel
{
public:
// SVM type
enum { C_SVC=100, NU_SVC=101, ONE_CLASS=102, EPS_SVR=103, NU_SVR=104 };
// SVM kernel type
enum { LINEAR=0, POLY=1, RBF=2, SIGMOID=3 };
// SVM params type
enum { C=0, GAMMA=1, P=2, NU=3, COEF=4, DEGREE=5 };
CvSVM();
virtual ~CvSVM();
CvSVM( const Mat& _train_data, const Mat& _responses,
const Mat& _var_idx=Mat(), const Mat& _sample_idx=Mat(),
CvSVMParams _params=CvSVMParams() );
virtual bool train( const Mat& _train_data, const Mat& _responses,
const Mat& _var_idx=Mat(), const Mat& _sample_idx=Mat(),
CvSVMParams _params=CvSVMParams() );
virtual bool train_auto( const Mat& _train_data, const Mat& _responses,
const Mat& _var_idx, const Mat& _sample_idx, CvSVMParams _params,
int k_fold = 10,
CvParamGrid C_grid = get_default_grid(CvSVM::C),
CvParamGrid gamma_grid = get_default_grid(CvSVM::GAMMA),
CvParamGrid p_grid = get_default_grid(CvSVM::P),
CvParamGrid nu_grid = get_default_grid(CvSVM::NU),
CvParamGrid coef_grid = get_default_grid(CvSVM::COEF),
CvParamGrid degree_grid = get_default_grid(CvSVM::DEGREE) );
virtual float predict( const Mat& _sample ) const;
virtual int get_support_vector_count() const;
virtual const float* get_support_vector(int i) const;
virtual CvSVMParams get_params() const { return params; };
virtual void clear();
static CvParamGrid get_default_grid( int param_id );
virtual void save( const char* filename, const char* name=0 );
virtual void load( const char* filename, const char* name=0 );
virtual void write( CvFileStorage* storage, const char* name );
virtual void read( CvFileStorage* storage, CvFileNode* node );
int get_var_count() const { return var_idx ? var_idx->cols : var_all; }
protected:
...
};
.. [LibSVM] C.-C. Chang and C.-J. Lin. *LIBSVM: a library for support vector machines*, ACM Transactions on Intelligent Systems and Technology, 2:27:1--27:27, 2011. (http://www.csie.ntu.edu.tw/~cjlin/papers/libsvm.pdf)
.. index:: CvSVMParams
.. _CvSVMParams:
CvSVMParams
CvParamGrid
-----------
.. c:type:: CvSVMParams
.. ocv:class:: CvParamGrid
SVM training parameters ::
The structure represents the logarithmic grid range of statmodel parameters. It is used for optimizing statmodel accuracy by varying model parameters, the accuracy estimate being computed by cross-validation.
struct CvSVMParams
{
CvSVMParams();
CvSVMParams( int _svm_type, int _kernel_type,
double _degree, double _gamma, double _coef0,
double _C, double _nu, double _p,
const CvMat* _class_weights, CvTermCriteria _term_crit );
.. ocv:member:: double CvParamGrid::min_val
int svm_type;
int kernel_type;
double degree; // for poly
double gamma; // for poly/rbf/sigmoid
double coef0; // for poly/sigmoid
Minimum value of the statmodel parameter.
double C; // for CV_SVM_C_SVC, CV_SVM_EPS_SVR and CV_SVM_NU_SVR
double nu; // for CV_SVM_NU_SVC, CV_SVM_ONE_CLASS, and CV_SVM_NU_SVR
double p; // for CV_SVM_EPS_SVR
CvMat* class_weights; // for CV_SVM_C_SVC
CvTermCriteria term_crit; // termination criteria
};
.. ocv:member:: double CvParamGrid::max_val
Maximum value of the statmodel parameter.
The structure must be initialized and passed to the training method of
:ref:`CvSVM` .
.. ocv:member:: double CvParamGrid::step
.. index:: CvSVM::train
Logarithmic step for iterating the statmodel parameter.
.. _CvSVM::train:
CvSVM::train
------------
.. ocv:function:: bool CvSVM::train( const Mat& _train_data, const Mat& _responses, const Mat& _var_idx=Mat(), const Mat& _sample_idx=Mat(), CvSVMParams _params=CvSVMParams() )
Trains SVM.
The method trains the SVM model. It follows the conventions of the generic ``train`` "method" with the following limitations:
* Only the ``CV_ROW_SAMPLE`` data layout is supported.
* Input variables are all ordered.
* Output variables can be either categorical ( ``_params.svm_type=CvSVM::C_SVC`` or ``_params.svm_type=CvSVM::NU_SVC`` ), or ordered ( ``_params.svm_type=CvSVM::EPS_SVR`` or ``_params.svm_type=CvSVM::NU_SVR`` ), or not required at all ( ``_params.svm_type=CvSVM::ONE_CLASS`` ).
* Missing measurements are not supported.
All the other parameters are gathered in the
:ref:`CvSVMParams` structure.
.. index:: CvSVM::train_auto
.. _CvSVM::train_auto:
CvSVM::train_auto
-----------------
.. ocv:function:: train_auto( const Mat& _train_data, const Mat& _responses, const Mat& _var_idx, const Mat& _sample_idx, CvSVMParams params, int k_fold = 10, CvParamGrid C_grid = get_default_grid(CvSVM::C), CvParamGrid gamma_grid = get_default_grid(CvSVM::GAMMA), CvParamGrid p_grid = get_default_grid(CvSVM::P), CvParamGrid nu_grid = get_default_grid(CvSVM::NU), CvParamGrid coef_grid = get_default_grid(CvSVM::COEF), CvParamGrid degree_grid = get_default_grid(CvSVM::DEGREE) )
Trains SVM with optimal parameters.
:param k_fold: Cross-validation parameter. The training set is divided into ``k_fold`` subsets. One subset is used to train the model, the others form the test set. So, the SVM algorithm is executed ``k_fold`` times.
The method trains the SVM model automatically by choosing the optimal
parameters ``C`` , ``gamma`` , ``p`` , ``nu`` , ``coef0`` , ``degree`` from
:ref:`CvSVMParams`. Parameters are considered optimal
when the cross-validation estimate of the test set error
is minimal. The parameters are iterated by a logarithmic grid, for
example, the parameter ``gamma`` takes the values in the set
(
:math:`min`,
:math:`min*step`,
:math:`min*{step}^2` , ...
:math:`min*{step}^n` )
where
:math:`min` is ``gamma_grid.min_val`` ,
:math:`step` is ``gamma_grid.step`` , and
:math:`n` is the maximal index such that
The grid determines the following iteration sequence of the statmodel parameter values:
.. math::
\texttt{gamma\_grid.min\_val} * \texttt{gamma\_grid.step} ^n < \texttt{gamma\_grid.max\_val}
(min\_val, min\_val*step, min\_val*{step}^2, \dots, min\_val*{step}^n),
So ``step`` must always be greater than 1.
where :math:`n` is the maximal index satisfying
If there is no need to optimize a parameter, the corresponding grid step should be set to any value less or equal to 1. For example, to avoid optimization in ``gamma`` , set ``gamma_grid.step = 0`` , ``gamma_grid.min_val`` , ``gamma_grid.max_val`` as arbitrary numbers. In this case, the value ``params.gamma`` is taken for ``gamma`` .
.. math::
\texttt{min\_val} * \texttt{step} ^n < \texttt{max\_val}
The grid is logarithmic, so ``step`` must always be greater then 1.
CvParamGrid::CvParamGrid
------------------------
The constructors.
.. ocv:function:: CvParamGrid::CvParamGrid()
.. ocv:function:: CvParamGrid::CvParamGrid( double min_val, double max_val, double log_step )
The full constructor initializes corresponding members. The default constructor creates a dummy grid:
::
CvParamGrid::CvParamGrid()
{
min_val = max_val = step = 0;
}
CvParamGrid::check
------------------
Checks validness of the grid.
.. ocv:function:: bool CvParamGrid::check()
Returns ``true`` if the grid is valid and ``false`` otherwise. The grid is valid if and only if:
* Lower bound of the grid is less then the upper one.
* Lower bound of the grid is positive.
* Grid step is greater then 1.
CvSVMParams
-----------
.. ocv:class:: CvSVMParams
SVM training parameters.
The structure must be initialized and passed to the training method of :ocv:class:`CvSVM`.
CvSVMParams::CvSVMParams
------------------------
The constructors.
.. ocv:function:: CvSVMParams::CvSVMParams()
.. ocv:function:: CvSVMParams::CvSVMParams( int svm_type, int kernel_type, double degree, double gamma, double coef0, double Cvalue, double nu, double p, CvMat* class_weights, CvTermCriteria term_crit )
:param svm_type: Type of a SVM formulation. Possible values are:
* **CvSVM::C_SVC** C-Support Vector Classification. ``n``-class classification (``n`` :math:`\geq` 2), allows imperfect separation of classes with penalty multiplier ``C`` for outliers.
* **CvSVM::NU_SVC** :math:`\nu`-Support Vector Classification. ``n``-class classification with possible imperfect separation. Parameter :math:`\nu` (in the range 0..1, the larger the value, the smoother the decision boundary) is used instead of ``C``.
* **CvSVM::ONE_CLASS** Distribution Estimation (One-class SVM). All the training data are from the same class, SVM builds a boundary that separates the class from the rest of the feature space.
* **CvSVM::EPS_SVR** :math:`\epsilon`-Support Vector Regression. The distance between feature vectors from the training set and the fitting hyper-plane must be less than ``p``. For outliers the penalty multiplier ``C`` is used.
* **CvSVM::NU_SVR** :math:`\nu`-Support Vector Regression. :math:`\nu` is used instead of ``p``.
See [LibSVM]_ for details.
:param kernel_type: Type of a SVM kernel. Possible values are:
* **CvSVM::LINEAR** Linear kernel. No mapping is done, linear discrimination (or regression) is done in the original feature space. It is the fastest option. :math:`K(x_i, x_j) = x_i^T x_j`.
* **CvSVM::POLY** Polynomial kernel: :math:`K(x_i, x_j) = (\gamma x_i^T x_j + coef0)^{degree}, \gamma > 0`.
* **CvSVM::RBF** Radial basis function (RBF), a good choice in most cases. :math:`K(x_i, x_j) = e^{-\gamma ||x_i - x_j||^2}, \gamma > 0`.
* **CvSVM::SIGMOID** Sigmoid kernel: :math:`K(x_i, x_j) = \tanh(\gamma x_i^T x_j + coef0)`.
:param degree: Parameter ``degree`` of a kernel function (POLY).
:param gamma: Parameter :math:`\gamma` of a kernel function (POLY / RBF / SIGMOID).
:param coef0: Parameter ``coef0`` of a kernel function (POLY / SIGMOID).
:param Cvalue: Parameter ``C`` of a SVM optimiazation problem (C_SVC / EPS_SVR / NU_SVR).
:param nu: Parameter :math:`\nu` of a SVM optimization problem (NU_SVC / ONE_CLASS / NU_SVR).
:param p: Parameter :math:`\epsilon` of a SVM optimization problem (EPS_SVR).
:param class_weights: Optional weights in the C_SVC problem , assigned to particular classes. They are multiplied by ``C`` so the parameter ``C`` of class ``#i`` becomes :math:`class\_weights_i * C`. Thus these weights affect the misclassification penalty for different classes. The larger weight, the larger penalty on misclassification of data from the corresponding class.
:param term_crit: Termination criteria of the iterative SVM training procedure which solves a partial case of constrained quadratic optimization problem. You can specify tolerance and/or the maximum number of iterations.
The default constructor initialize the structure with following values:
::
CvSVMParams::CvSVMParams() :
svm_type(CvSVM::C_SVC), kernel_type(CvSVM::RBF), degree(0),
gamma(1), coef0(0), C(1), nu(0), p(0), class_weights(0)
{
term_crit = cvTermCriteria( CV_TERMCRIT_ITER+CV_TERMCRIT_EPS, 1000, FLT_EPSILON );
}
CvSVM
-----
.. ocv:class:: CvSVM
Support Vector Machines.
CvSVM::CvSVM
------------
Default and training constructors.
.. ocv:function:: CvSVM::CvSVM()
.. ocv:function:: CvSVM::CvSVM( const Mat& trainData, const Mat& responses, const Mat& varIdx=Mat(), const Mat& sampleIdx=Mat(), CvSVMParams params=CvSVMParams() )
.. ocv:function::CvSVM::CvSVM( const CvMat* trainData, const CvMat* responses, const CvMat* varIdx=0, const CvMat* sampleIdx=0, CvSVMParams params=CvSVMParams() )
.. ocv:pyfunction:: cv2.SVM(trainData, responses[, varIdx[, sampleIdx[, params]]]) -> <SVM object>
The constructors follow conventions of :ocv:func:`CvStatModel::CvStatModel`. See :ocv:func:`CvStatModel::train` for parameters descriptions.
CvSVM::train
------------
Trains an SVM.
.. ocv:function:: bool CvSVM::train( const Mat& trainData, const Mat& responses, const Mat& varIdx=Mat(), const Mat& sampleIdx=Mat(), CvSVMParams params=CvSVMParams() )
.. ocv:function::bool CvSVM::train( const CvMat* trainData, const CvMat* responses, const CvMat* varIdx=0, const CvMat* sampleIdx=0, CvSVMParams params=CvSVMParams() )
.. ocv:pyfunction:: cv2.SVM.train(trainData, responses[, varIdx[, sampleIdx[, params]]]) -> retval
The method trains the SVM model. It follows the conventions of the generic :ocv:func:`CvStatModel::train` approach with the following limitations:
* Only the ``CV_ROW_SAMPLE`` data layout is supported.
* Input variables are all ordered.
* Output variables can be either categorical (``params.svm_type=CvSVM::C_SVC`` or ``params.svm_type=CvSVM::NU_SVC``), or ordered (``params.svm_type=CvSVM::EPS_SVR`` or ``params.svm_type=CvSVM::NU_SVR``), or not required at all (``params.svm_type=CvSVM::ONE_CLASS``).
* Missing measurements are not supported.
All the other parameters are gathered in the
:ocv:class:`CvSVMParams` structure.
CvSVM::train_auto
-----------------
Trains an SVM with optimal parameters.
.. ocv:function:: bool CvSVM::train_auto( const Mat& trainData, const Mat& responses, const Mat& varIdx, const Mat& sampleIdx, CvSVMParams params, int k_fold = 10, CvParamGrid Cgrid = CvSVM::get_default_grid(CvSVM::C), CvParamGrid gammaGrid = CvSVM::get_default_grid(CvSVM::GAMMA), CvParamGrid pGrid = CvSVM::get_default_grid(CvSVM::P), CvParamGrid nuGrid = CvSVM::get_default_grid(CvSVM::NU), CvParamGrid coeffGrid = CvSVM::get_default_grid(CvSVM::COEF), CvParamGrid degreeGrid = CvSVM::get_default_grid(CvSVM::DEGREE), bool balanced=false)
.. ocv:function::bool CvSVM::train_auto( const CvMat* trainData, const CvMat* responses, const CvMat* varIdx, const CvMat* sampleIdx, CvSVMParams params, int kfold = 10, CvParamGrid Cgrid = get_default_grid(CvSVM::C), CvParamGrid gammaGrid = get_default_grid(CvSVM::GAMMA), CvParamGrid pGrid = get_default_grid(CvSVM::P), CvParamGrid nuGrid = get_default_grid(CvSVM::NU), CvParamGrid coeffGrid = get_default_grid(CvSVM::COEF), CvParamGrid degreeGrid = get_default_grid(CvSVM::DEGREE), bool balanced=false )
.. ocv:pyfunction:: cv2.SVM.train_auto(trainData, responses, varIdx, sampleIdx, params[, k_fold[, Cgrid[, gammaGrid[, pGrid[, nuGrid[, coeffGrid[, degreeGrid[, balanced]]]]]]]]) -> retval
:param k_fold: Cross-validation parameter. The training set is divided into ``k_fold`` subsets. One subset is used to train the model, the others form the test set. So, the SVM algorithm is executed ``k_fold`` times.
:param \*Grid: Iteration grid for the corresponding SVM parameter.
:param balanced: If ``true`` and the problem is 2-class classification then the method creates more balanced cross-validation subsets that is proportions between classes in subsets are close to such proportion in the whole train dataset.
The method trains the SVM model automatically by choosing the optimal
parameters ``C``, ``gamma``, ``p``, ``nu``, ``coef0``, ``degree`` from
:ocv:class:`CvSVMParams`. Parameters are considered optimal
when the cross-validation estimate of the test set error
is minimal.
If there is no need to optimize a parameter, the corresponding grid step should be set to any value less than or equal to 1. For example, to avoid optimization in ``gamma``, set ``gamma_grid.step = 0``, ``gamma_grid.min_val``, ``gamma_grid.max_val`` as arbitrary numbers. In this case, the value ``params.gamma`` is taken for ``gamma``.
And, finally, if the optimization in a parameter is required but
the corresponding grid is unknown, you may call the function ``CvSVM::get_default_grid`` . To generate a grid, for example, for ``gamma`` , call ``CvSVM::get_default_grid(CvSVM::GAMMA)`` .
the corresponding grid is unknown, you may call the function :ocv:func:`CvSVM::get_default_grid`. To generate a grid, for example, for ``gamma``, call ``CvSVM::get_default_grid(CvSVM::GAMMA)``.
This function works for the case of classification
( ``params.svm_type=CvSVM::C_SVC`` or ``params.svm_type=CvSVM::NU_SVC`` )
This function works for the classification
(``params.svm_type=CvSVM::C_SVC`` or ``params.svm_type=CvSVM::NU_SVC``)
as well as for the regression
( ``params.svm_type=CvSVM::EPS_SVR`` or ``params.svm_type=CvSVM::NU_SVR`` ). If ``params.svm_type=CvSVM::ONE_CLASS`` , no optimization is made and the usual SVM with parameters specified in ``params`` is executed.
(``params.svm_type=CvSVM::EPS_SVR`` or ``params.svm_type=CvSVM::NU_SVR``). If ``params.svm_type=CvSVM::ONE_CLASS``, no optimization is made and the usual SVM with parameters specified in ``params`` is executed.
.. index:: CvSVM::get_default_grid
CvSVM::predict
--------------
Predicts the response for input sample(s).
.. _CvSVM::get_default_grid:
.. ocv:function:: float CvSVM::predict( const Mat& sample, bool returnDFVal=false ) const
.. ocv:function::float CvSVM::predict( const CvMat* sample, bool returnDFVal=false ) const
.. ocv:function::float CvSVM::predict( const CvMat* samples, CvMat* results ) const
.. ocv:pyfunction:: cv2.SVM.predict(sample[, returnDFVal]) -> retval
:param sample(s): Input sample(s) for prediction.
:param returnDFVal: Specifies a type of the return value. If ``true`` and the problem is 2-class classification then the method returns the decision function value that is signed distance to the margin, else the function returns a class label (classification) or estimated function value (regression).
:param results: Output prediction responses for corresponding samples.
If you pass one sample then prediction result is returned. If you want to get responses for several samples then you should pass the ``results`` matrix where prediction results will be stored.
CvSVM::get_default_grid
-----------------------
Generates a grid for SVM parameters.
.. ocv:function:: CvParamGrid CvSVM::get_default_grid( int param_id )
Generates a grid for SVM parameters.
:param param_id: SVN parameters IDs that must be one of the following:
:param param_id: SVM parameters IDs that must be one of the following:
* **CvSVM::C**
@@ -207,33 +260,36 @@ CvSVM::get_default_grid
* **CvSVM::DEGREE**
The grid will be generated for the parameter with this ID.
The grid is generated for the parameter with this ID.
The function generates a grid for the specified parameter of the SVM algorithm. The grid may be passed to the function ``CvSVM::train_auto`` .
.. index:: CvSVM::get_params
.. _CvSVM::get_params:
The function generates a grid for the specified parameter of the SVM algorithm. The grid may be passed to the function :ocv:func:`CvSVM::train_auto`.
CvSVM::get_params
-----------------
Returns the current SVM parameters.
.. ocv:function:: CvSVMParams CvSVM::get_params() const
Returns the current SVM parameters.
This function may be used to get the optimal parameters obtained while automatically training :ocv:func:`CvSVM::train_auto`.
This function may be used to get the optimal parameters obtained while automatically training ``CvSVM::train_auto`` .
.. index:: CvSVM::get_support_vector*
.. _CvSVM::get_support_vector*:
CvSVM::get_support_vector*
CvSVM::get_support_vector
--------------------------
Retrieves a number of support vectors and the particular vector.
.. ocv:function:: int CvSVM::get_support_vector_count() const
.. ocv:function:: const float* CvSVM::get_support_vector(int i) const
Retrieves a number of support vectors and the particular vector.
.. ocv:pyfunction:: cv2.SVM.get_support_vector_count() -> nsupportVectors
:param i: Index of the particular support vector.
The methods can be used to retrieve a set of support vectors.
CvSVM::get_var_count
--------------------
Returns the number of used features (variables count).
.. ocv:function:: int CvSVM::get_var_count() const
.. ocv:pyfunction:: cv2.SVM.get_var_count() -> nvars