potok/opencv
1
0
Fork 0
mirror of https://github.com/opencv/opencv.git synced 2026-07-30 15:53:03 +04:00
Code Packages Releases Wiki Activity

Merge pull request #28289 from JonasPerolini:pr-aruco-identification

Browse Source 
Identify ArUco markers based on threshold to reduce false positives #28289

**Goal:** parametrize the current marker identification process (pixel-based majority count) to reduce the number of false positives while maintaining high recall. Useful in high risk scenarios in which false positives are not acceptable. 

**Context:** This PR builds on top of https://github.com/opencv/opencv/pull/23190 in which we've introduced a pixel-based confidence in the marker detection.

**Solution:** Include a new parameter: `validBitIdThreshold` used to identify markers based on the pixel count of each cell. Set the parameter default either to 50% which is equivalent to the current majority count implementation or to 49% which already singnificantly reduces the number of false positives (see details below). 

**Test coverage:** 
- Unit tests: `CV_ArucoDetectionThreshold`, `CV_InvertedArucoDetectionThreshold`
- The impact of `validBitIdThreshold` on false positives was also tested using the benchmark dataset: `MIRFLICKR-25k` https://www.kaggle.com/datasets/skfrost19/mirflickr25k which contains random images without any markers. Every marker detection is a false positive. 

Example of images in the dataset:

![im2048](https://github.com/user-attachments/assets/3e38796b-67ce-44be-a91d-2fd268414515)

![im17627](https://github.com/user-attachments/assets/37253b9f-829d-4bac-b9fc-c844d16f546e)

**Results:** A threshold of 49% already allows to significantly reduce the number of false positives for the dict `DICT_4X4_1000`: 
- `5942` false positives for `validBitIdThreshold = 0.5`
- `629` false positives for `validBitIdThreshold = 0.49` and `0.46` 
   - number of false positives divided by `9.5` when compared to `validBitIdThreshold = 0.5`
- `139` false positives for `validBitIdThreshold = 0.43` and `0.4` 
   - number of false positives divided by `42` when compared to `validBitIdThreshold = 0.5`

Dicts with a higher number of cells are not as impacted since it's much harder to obtain false positives. However, the less cells in a marker the further away it can be reliably detected, so the dict `DICT_4X4_1000` is commonly used.

<img width="1280" height="800" alt="false_positive_image_rate" src="https://github.com/user-attachments/assets/1a0ee16a-221d-443e-835b-022ed6dea6b0" />

In the image attached, the values of `validBitIdThreshold` tested are:  `0.10f, 0.20f, 0.30f, 0.40f, 0.43f, 0.46f, 0.49f, 0.50f, 0.53f, 0.56f, 0.60f, 0.70f, 0.80f, 0.90f`

Summary of the results: [summary.csv](https://github.com/user-attachments/files/24315662/summary.csv)

Note that we can also analyse the number of false positives per marker `id`. For example, here's the histogram for the dict `DICT_4X4_1000`. (The CSV attached contains all the results)

<img width="1440" height="640" alt="false_positive_ids_DICT_4X4_1000_thr0 50" src="https://github.com/user-attachments/assets/af4f3ff8-9b8f-4682-9d51-a090c2610d8c" />

For example, the marker id 17 is detected 252 times with  `validBitIdThreshold = 0.5` and only 34 times with `validBitIdThreshold = 0.49`. Looking at marker 17 (see below), we understand that this simple pattern randomly occurs in images.

<img width="447" height="441" alt="Marker17" src="https://github.com/user-attachments/assets/f5d09227-b39b-4598-94f9-b529f8300703" />

Results for every dict and every `validBitIdThreshold` [per_id.csv](https://github.com/user-attachments/files/24315667/per_id.csv)

**Missing coverage:** there is no labeled dataset with images containing markers to analyse the impact of on the recall  (i.e. look at the true positive rate). For my specific use case (drones) any threshold above `0.4` allows to maintain a high recall in all conditions.

### Pull Request Readiness Checklist


- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [ ] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
This commit is contained in:
Jonas Perolini
2026-03-04 06:44:28 +01:00
committed by GitHub
parent 91c78f5064
commit 5e91b461bc
8 changed files with 259 additions and 66 deletions
Download Patch File Download Diff File Expand all files Collapse all files
modules/objdetect/src/aruco/aruco_detector.cpp
+44 -51
View File
@@ -310,12 +310,11 @@ static void _detectInitialCandidates(const Mat &grey, vector<vector<Point2f> > &
/**
* @brief Given an input image and candidate corners, extract the bits of the candidate, including
* @brief Given an input image and candidate corners, extract the cell pixel ratio of the candidate, including
* the border bits
*/
static Mat _extractBits(InputArray _image, const vector<Point2f>& corners, int markerSize,
int markerBorderBits, int cellSize, double cellMarginRate, double minStdDevOtsu,
OutputArray _cellPixelRatio = noArray()) {
static Mat _extractCellPixelRatio(InputArray _image, const vector<Point2f>& corners, int markerSize,
int markerBorderBits, int cellSize, double cellMarginRate, double minStdDevOtsu) {
CV_Assert(_image.getMat().channels() == 1);
CV_Assert(corners.size() == 4ull);
CV_Assert(markerBorderBits > 0 && cellSize > 0 && cellMarginRate >= 0 && cellMarginRate <= 0.5);
@@ -339,11 +338,11 @@ static Mat _extractBits(InputArray _image, const vector<Point2f>& corners, int m
warpPerspective(_image, resultImg, transformation, Size(resultImgSize, resultImgSize),
INTER_NEAREST);
// output image containing the bits
Mat bits(markerSizeWithBorders, markerSizeWithBorders, CV_8UC1, Scalar::all(0));
// output image containing the ratio of white pixels in each cell
Mat cellPixelRatio(markerSizeWithBorders, markerSizeWithBorders, CV_32FC1, Scalar::all(0));
// check if standard deviation is enough to apply Otsu
// if not enough, it probably means all bits are the same color (black or white)
// if not enough, it probably means all pixels are the same color (black or white)
Mat mean, stddev;
// Remove some border just to avoid border noise from perspective transformation
Mat innerRegion = resultImg.colRange(cellSize / 2, resultImg.cols - cellSize / 2)
@@ -351,18 +350,13 @@ static Mat _extractBits(InputArray _image, const vector<Point2f>& corners, int m
meanStdDev(innerRegion, mean, stddev);
if(stddev.ptr< double >(0)[0] < minStdDevOtsu) {
// all black or all white, depending on mean value
if(mean.ptr< double >(0)[0] > 127)
bits.setTo(1);
else
bits.setTo(0);
if(_cellPixelRatio.needed()) bits.convertTo(_cellPixelRatio, CV_32F);
return bits;
}
if(mean.ptr< double >(0)[0] > 127){
cellPixelRatio.setTo(1);
} else {
cellPixelRatio.setTo(0);
}
Mat cellPixelRatio;
if (_cellPixelRatio.needed()) {
_cellPixelRatio.create(markerSizeWithBorders, markerSizeWithBorders, CV_32FC1);
cellPixelRatio = _cellPixelRatio.getMatRef();
return cellPixelRatio;
}
// now extract code, first threshold using Otsu
@@ -377,38 +371,40 @@ static Mat _extractBits(InputArray _image, const vector<Point2f>& corners, int m
cellSize - 2 * cellMarginPixels));
// count white pixels on each cell to assign its value
size_t nZ = (size_t) countNonZero(square);
if(nZ > square.total() / 2) bits.at<unsigned char>(y, x) = 1;
// define the cell pixel ratio as the ratio of the white pixels. For inverted markers, the ratio will be inverted.
if(_cellPixelRatio.needed()) cellPixelRatio.at<float>(y, x) = (nZ / (float)square.total());
cellPixelRatio.at<float>(y, x) = (nZ / (float)square.total());
}
}
return bits;
return cellPixelRatio;
}
/**
* @brief Return number of erroneous bits in border, i.e. number of white bits in border.
* @brief Return number of erroneous bits in border, i.e. bits for which pixel ratio > validBitIdThreshold.
*/
static int _getBorderErrors(const Mat &bits, int markerSize, int borderSize) {
static int _getBorderErrors(const Mat &cellPixelRatio, int markerSize, int borderSize, float validBitIdThreshold) {
int sizeWithBorders = markerSize + 2 * borderSize;
CV_Assert(markerSize > 0 && bits.cols == sizeWithBorders && bits.rows == sizeWithBorders);
CV_Assert(markerSize > 0 && cellPixelRatio.cols == sizeWithBorders && cellPixelRatio.rows == sizeWithBorders);
// Get border error. cellPixelRatio has the opposite color as the borders.
int totalErrors = 0;
for(int y = 0; y < sizeWithBorders; y++) {
for(int k = 0; k < borderSize; k++) {
if(bits.ptr<unsigned char>(y)[k] != 0) totalErrors++;
if(bits.ptr<unsigned char>(y)[sizeWithBorders - 1 - k] != 0) totalErrors++;
// Left and right vertical sides
if(cellPixelRatio.ptr<float>(y)[k] > validBitIdThreshold) totalErrors++;
if(cellPixelRatio.ptr<float>(y)[sizeWithBorders - 1 - k] > validBitIdThreshold) totalErrors++;
}
}
for(int x = borderSize; x < sizeWithBorders - borderSize; x++) {
for(int k = 0; k < borderSize; k++) {
if(bits.ptr<unsigned char>(k)[x] != 0) totalErrors++;
if(bits.ptr<unsigned char>(sizeWithBorders - 1 - k)[x] != 0) totalErrors++;
// Top and bottom horizontal sides
if(cellPixelRatio.ptr<float>(k)[x] > validBitIdThreshold) totalErrors++;
if(cellPixelRatio.ptr<float>(sizeWithBorders - 1 - k)[x] > validBitIdThreshold) totalErrors++;
}
}
return totalErrors;
@@ -482,49 +478,43 @@ static uint8_t _identifyOneCandidate(const Dictionary& dictionary, const Mat& _i
scaled_corners[i].y = _corners[i].y * scale;
}
Mat cellPixelRatio;
Mat candidateBits =
_extractBits(_image, scaled_corners, dictionary.markerSize, params.markerBorderBits,
params.perspectiveRemovePixelPerCell,
params.perspectiveRemoveIgnoredMarginPerCell, params.minOtsuStdDev,
cellPixelRatio);
Mat cellPixelRatio =
_extractCellPixelRatio(_image, scaled_corners, dictionary.markerSize, params.markerBorderBits,
params.perspectiveRemovePixelPerCell,
params.perspectiveRemoveIgnoredMarginPerCell, params.minOtsuStdDev);
// analyze border bits
int maximumErrorsInBorder =
int(dictionary.markerSize * dictionary.markerSize * params.maxErroneousBitsInBorderRate);
int(dictionary.markerSize * dictionary.markerSize * params.maxErroneousBitsInBorderRate);
int borderErrors =
_getBorderErrors(candidateBits, dictionary.markerSize, params.markerBorderBits);
_getBorderErrors(cellPixelRatio, dictionary.markerSize, params.markerBorderBits, params.validBitIdThreshold);
// check if it is a white marker
if(params.detectInvertedMarker){
// to get from 255 to 1
Mat invertedImg = ~candidateBits-254;
int invBError = _getBorderErrors(invertedImg, dictionary.markerSize, params.markerBorderBits);
Mat invCellPixelRatio = 1.f - cellPixelRatio;
int invBError = _getBorderErrors(invCellPixelRatio, dictionary.markerSize, params.markerBorderBits, params.validBitIdThreshold);
// white marker
if(invBError<borderErrors){
cellPixelRatio = 1.f - cellPixelRatio;
borderErrors = invBError;
invertedImg.copyTo(candidateBits);
invCellPixelRatio.copyTo(cellPixelRatio);
typ=2;
}
}
if(borderErrors > maximumErrorsInBorder) return 0; // border is wrong
// take only inner bits
Mat onlyBits =
candidateBits.rowRange(params.markerBorderBits,
candidateBits.rows - params.markerBorderBits)
.colRange(params.markerBorderBits, candidateBits.cols - params.markerBorderBits);
Mat onlyCellPixelRatio =
cellPixelRatio.rowRange(params.markerBorderBits,
cellPixelRatio.rows - params.markerBorderBits)
.colRange(params.markerBorderBits, cellPixelRatio.cols - params.markerBorderBits);
// try to indentify the marker
if(!dictionary.identify(onlyBits, idx, rotation, params.errorCorrectionRate))
// try to identify the marker
if(!dictionary.identify(onlyCellPixelRatio, idx, rotation, params.errorCorrectionRate, params.validBitIdThreshold))
return 0;
// compute the candidate's confidence
if(confidenceNeeded) {
Mat groundTruthbits;
Mat bitsUints = dictionary.getBitsFromByteList(dictionary.bytesList.rowRange(idx, idx + 1), dictionary.markerSize, rotation);
bitsUints.convertTo(groundTruthbits, CV_32F);
Mat groundTruthbits = dictionary.getMarkerBits(idx, rotation);
markerConfidence = _getMarkerConfidence(groundTruthbits, cellPixelRatio, dictionary.markerSize, params.markerBorderBits);
}
@@ -1403,11 +1393,14 @@ void ArucoDetector::refineDetectedMarkers(InputArray _image, const Board& _board
if(refineParams.errorCorrectionRate >= 0) {
// extract bits
Mat bits = _extractBits(
Mat cellPixelRatio = _extractCellPixelRatio(
grey, rotatedMarker, dictionary.markerSize, detectorParams.markerBorderBits,
detectorParams.perspectiveRemovePixelPerCell,
detectorParams.perspectiveRemoveIgnoredMarginPerCell, detectorParams.minOtsuStdDev);
Mat bits;
cellPixelRatio.convertTo(bits, CV_8UC1);
Mat onlyBits =
bits.rowRange(detectorParams.markerBorderBits, bits.rows - detectorParams.markerBorderBits)
.colRange(detectorParams.markerBorderBits, bits.rows - detectorParams.markerBorderBits);
modules/objdetect/src/aruco/aruco_dictionary.cpp
+46 -12
View File
@@ -73,25 +73,32 @@ void Dictionary::writeDictionary(FileStorage& fs, const String &name)
}
bool Dictionary::identify(const Mat &onlyBits, int &idx, int &rotation, double maxCorrectionRate) const {
CV_Assert(onlyBits.rows == markerSize && onlyBits.cols == markerSize);
bool Dictionary::identify(const Mat &onlyCellPixelRatio, CV_OUT int &idx, CV_OUT int &rotation, double maxCorrectionRate, float validBitIdThreshold) const {
CV_Assert(onlyCellPixelRatio.rows == markerSize && onlyCellPixelRatio.cols == markerSize);
int maxCorrectionRecalculed = int(double(maxCorrectionBits) * maxCorrectionRate);
// get as a byte list
Mat candidateBytes = getByteListFromBits(onlyBits);
idx = -1; // by default, not found
// search closest marker in dict
for(int m = 0; m < bytesList.rows; m++) {
int currentMinDistance = markerSize * markerSize + 1;
int currentRotation = -1;
for(unsigned int r = 0; r < 4; r++) {
int currentHamming = cv::hal::normHamming(
bytesList.ptr(m)+r*candidateBytes.cols,
candidateBytes.ptr(),
candidateBytes.cols);
for(int r = 0; r < 4; r++) {
Mat bitsRot = getBitsFromByteList(bytesList.rowRange(m, m + 1), markerSize, r);
bitsRot.convertTo(bitsRot, CV_32F);
// Loop over all bits dictBitsList [m, markerSize * markerSize, 4]; onlyCellPixelRatio [markerSize, markerSize]
int currentHamming = 0;
for(int i = 0; i < markerSize; i++) {
for(int j = 0; j < markerSize; j++) {
// If detected bit is too far from the ground truth, consider it false.
if(fabs(onlyCellPixelRatio.at<float>(i, j) - static_cast<float>(bitsRot.at<float>(i, j))) > validBitIdThreshold){
currentHamming++;
}
}
}
if(currentHamming < currentMinDistance) {
currentMinDistance = currentHamming;
@@ -111,6 +118,16 @@ bool Dictionary::identify(const Mat &onlyBits, int &idx, int &rotation, double m
}
bool Dictionary::identify(const Mat &onlyBits, CV_OUT int &idx, CV_OUT int &rotation, double maxCorrectionRate) const {
CV_Assert(onlyBits.rows == markerSize && onlyBits.cols == markerSize);
Mat candidateBitRatio;
onlyBits.convertTo(candidateBitRatio, CV_32F);
const float validBitIdThreshold = DEFAULT_VALID_BIT_ID_THRESHOLD;
return identify(candidateBitRatio, idx, rotation, maxCorrectionRate, validBitIdThreshold);
}
int Dictionary::getDistanceToId(InputArray bits, int id, bool allRotations) const {
CV_Assert(id >= 0 && id < bytesList.rows);
@@ -147,7 +164,8 @@ void Dictionary::generateImageMarker(int id, int sidePixels, OutputArray _img, i
Mat innerRegion = tinyMarker.rowRange(borderBits, tinyMarker.rows - borderBits)
.colRange(borderBits, tinyMarker.cols - borderBits);
// put inner bits
Mat bits = 255 * getBitsFromByteList(bytesList.rowRange(id, id + 1), markerSize);
Mat bits = getMarkerBits(id);
bits.convertTo(bits, CV_8U, 255.0);
CV_Assert(innerRegion.total() == bits.total());
bits.copyTo(innerRegion);
@@ -194,12 +212,28 @@ Mat Dictionary::getByteListFromBits(const Mat &bits) {
}
Mat Dictionary::getMarkerBits(int markerId, int rotationId) const {
const int nbRotations = 4;
CV_Assert(markerId < bytesList.rows);
CV_Assert(rotationId < nbRotations);
Mat bits(markerSize, markerSize, CV_32F, Scalar::all(0));
Mat bitsUints = getBitsFromByteList(bytesList.rowRange(markerId, markerId + 1), markerSize, rotationId);
bitsUints.convertTo(bits, CV_32F);
CV_Assert(bits.rows == markerSize && bits.cols == markerSize);
return bits;
}
Mat Dictionary::getBitsFromByteList(const Mat &byteList, int markerSize, int rotationId) {
CV_Assert(byteList.total() > 0 &&
byteList.total() >= (unsigned int)markerSize * markerSize / 8 &&
byteList.total() <= (unsigned int)markerSize * markerSize / 8 + 1);
CV_Assert(rotationId >=0 && rotationId < 4);
CV_Assert(byteList.channels() >= 4);
CV_Assert(rotationId >= 0 && rotationId < 4);
Mat bits(markerSize, markerSize, CV_8UC1, Scalar::all(0));