1 ArUco marker
ArUco marker是由S.Garrido-Jurado等人在2014年提出的，全称是Augmented Reality University of Cordoba，详见他们的论文《Automatic generation and detection of highly reliable fiducial markers under occlusion》。
它类似于二维码，通过预先标定的相机参数和捕获到的aruco标记图像，就可以求解出marker与相机的相对坐标关系。实现增强现实大致包括以下几个步骤：
- 相机标定，包括相机的内参、畸变系数等；
- 利用相机参数和预先设定好的marker参数（包括边长、角点等）求解相机3D坐标系到marker所在3D坐标系（z=0）的空间变换关系T；
- 将T作用于虚拟物体计算其在相机成像平面的投影。
2 Opencv中实现一个简单的AR应用
2.1 编译aruco模块
aruco在opencv的extra_moduls（opencv_contrib）里，需要额外编译才能使用。
- （不推荐）编译全部opencv_contrib并引用，参考链接；
- （推荐）只编译某些模块，把这些模块从“opencv_contrib-x.x.x/modules”拷贝到“opencv/modules”，然后执行opencv编译即可。
注意一定要下载与opencv版本相对应的opencv_contrib，可从国内镜像（链接）快速下载。
2.2 生成aruco marker
aruco字典中定义了不同大小的marker（4x4、5x5、6x6、7x7），每个词典中包含了固定数量的标记（50、100、250、1000），详细可以参考opencv里面的定义：
/**
* @brief Predefined markers dictionaries/sets
* Each dictionary indicates the number of bits and the number of markers contained
* - DICT_ARUCO_ORIGINAL: standard ArUco Library Markers. 1024 markers, 5x5 bits, 0 minimum
distance
*/
enum CV_EXPORTS_W_SIMPLE PREDEFINED_DICTIONARY_NAME {
DICT_4X4_50 = 0,
DICT_4X4_100,
DICT_4X4_250,
DICT_4X4_1000,
DICT_5X5_50,
DICT_5X5_100,
DICT_5X5_250,
DICT_5X5_1000,
DICT_6X6_50,
DICT_6X6_100,
DICT_6X6_250,
DICT_6X6_1000,
DICT_7X7_50,
DICT_7X7_100,
DICT_7X7_250,
DICT_7X7_1000,
DICT_ARUCO_ORIGINAL
};
通过以下C++代码可以创建一个marker图像，生成的图像大小为固定的200x200像素：
// 获取大小为6x6、包含1000个marker的字典
cv::Ptr<cv::aruco::Dictionary> dictionary = cv::aruco::getPredefinedDictionary(cv::aruco::DICT_6X6_1000);
// 绘制marker图像到markerImage，且在边缘添加1格黑色边界
cv::Mat markerImage;
cv::aruco::drawMarker(dictionary, 23, 200, markerImage, 1);
2.3 检测aruco marker
将打印出来的marker放置在环境中，我们就可以通过相机和以下代码对其进行实时检测：
// 检测到的marker的id
std::vector<int> ids;
// 检测到的marker的角点列表
std::vector<std::vector<cv::Point2f>> corners;
// 执行检测
cv::aruco::detectMarkers(image, dictionary, corners, ids, params);
为了提高检测精度，我们可以通过detectMarkers的cv::aruco::DetectorParameters参数（即上一段代码中的params）来设置一些参数。首先让我们先看一下可以设置的参数列表：
/**
* @brief Parameters for the detectMarker process:
* - adaptiveThreshWinSizeMin: minimum window size for adaptive thresholding before finding
* contours (default 3).
* - adaptiveThreshWinSizeMax: maximum window size for adaptive thresholding before finding
* contours (default 23).
* - adaptiveThreshWinSizeStep: increments from adaptiveThreshWinSizeMin to adaptiveThreshWinSizeMax
* during the thresholding (default 10).
* - adaptiveThreshConstant: constant for adaptive thresholding before finding contours (default 7)
* - minMarkerPerimeterRate: determine minimum perimeter for marker contour to be detected. This
* is defined as a rate respect to the maximum dimension of the input image (default 0.03).
* - maxMarkerPerimeterRate: determine maximum perimeter for marker contour to be detected. This
* is defined as a rate respect to the maximum dimension of the input image (default 4.0).
* - polygonalApproxAccuracyRate: minimum accuracy during the polygonal approximation process to
* determine which contours are squares.
* - minCornerDistanceRate: minimum distance between corners for detected markers relative to its
* perimeter (default 0.05)
* - minDistanceToBorder: minimum distance of any corner to the image border for detected markers
* (in pixels) (default 3)
* - minMarkerDistanceRate: minimum mean distance beetween two marker corners to be considered
* similar, so that the smaller one is removed. The rate is relative to the smaller perimeter
* of the two markers (default 0.05).
* - cornerRefinementMethod: corner refinement method. (CORNER_REFINE_NONE, no refinement.
* CORNER_REFINE_SUBPIX, do subpixel refinement. CORNER_REFINE_CONTOUR use contour-Points)
* - cornerRefinementWinSize: window size for the corner refinement process (in pixels) (default 5).
* - cornerRefinementMaxIterations: maximum number of iterations for stop criteria of the corner
* refinement process (default 30).
* - cornerRefinementMinAccuracy: minimum error for the stop cristeria of the corner refinement
* process (default: 0.1)
* - markerBorderBits: number of bits of the marker border, i.e. marker border width (default 1).
* - perpectiveRemovePixelPerCell: number of bits (per dimension) for each cell of the marker
* when removing the perspective (default 8).
* - perspectiveRemoveIgnoredMarginPerCell: width of the margin of pixels on each cell not
* considered for the determination of the cell bit. Represents the rate respect to the total
* size of the cell, i.e. perpectiveRemovePixelPerCell (default 0.13)
* - maxErroneousBitsInBorderRate: maximum number of accepted erroneous bits in the border (i.e.
* number of allowed white bits in the border). Represented as a rate respect to the total
* number of bits per marker (default 0.35).
* - minOtsuStdDev: minimun standard deviation in pixels values during the decodification step to
* apply Otsu thresholding (otherwise, all the bits are set to 0 or 1 depending on mean higher
* than 128 or not) (default 5.0)
* - errorCorrectionRate error correction rate respect to the maximun error correction capability
* for each dictionary. (default 0.6).
*/
struct CV_EXPORTS_W DetectorParameters {
DetectorParameters();
CV_WRAP static Ptr<DetectorParameters> create();
CV_PROP_RW int adaptiveThreshWinSizeMin;
CV_PROP_RW int adaptiveThreshWinSizeMax;
CV_PROP_RW int adaptiveThreshWinSizeStep;
CV_PROP_RW double adaptiveThreshConstant;
CV_PROP_RW double minMarkerPerimeterRate;
CV_PROP_RW double maxMarkerPerimeterRate;
CV_PROP_RW double polygonalApproxAccuracyRate;
CV_PROP_RW double minCornerDistanceRate;
CV_PROP_RW int minDistanceToBorder;
CV_PROP_RW double minMarkerDistanceRate;
CV_PROP_RW int cornerRefinementMethod;
CV_PROP_RW int cornerRefinementWinSize;
CV_PROP_RW int cornerRefinementMaxIterations;
CV_PROP_RW double cornerRefinementMinAccuracy;
CV_PROP_RW int markerBorderBits;
CV_PROP_RW int perspectiveRemovePixelPerCell;
CV_PROP_RW double perspectiveRemoveIgnoredMarginPerCell;
CV_PROP_RW double maxErroneousBitsInBorderRate;
CV_PROP_RW double minOtsuStdDev;
CV_PROP_RW double errorCorrectionRate;
};
这里设置挑几个参数进行设置，其他参数可根据应用场景自行添加：
cv::Ptr<cv::aruco::DetectorParameters> params = cv::aruco::DetectorParameters::create();
params->adaptiveThreshWinSizeMin = 7;
params->cornerRefinementWinSize = 5;
params->cornerRefinementMaxIterations = 100;
params->cornerRefinementMinAccuracy = 0.1;
params->cornerRefinementMethod = cv::aruco::CORNER_REFINE_CONTOUR;
2.4 绘制检测结果及AR显示
利用下面的代码可将检测到的marker边界及角点绘制在图像上：
// 绘制检测到的marker的框
cv::aruco::drawDetectedMarkers(image, corners, ids);
利用检测到的角点（corners）以及下面代码中出现的相机参数（cameraMatrix、distCoeffs）、marker尺寸（markerLength）来计算marker与相机坐标系之间的三维空间变换关系，分别得到旋转分量rvecs和平移分量tvecs。其中rvecs可以通过罗德里格斯变换计算对应的旋转矩阵，tvecs的单位与markerLength的单位一致。
// 估计第一个marker的位姿矩阵
double markerLength = 50;
std::vector<cv::Vec3d> rvecs, tvecs;
cv::aruco::estimatePoseSingleMarkers(corners, markerLength, cameraMatrix, distCoeffs, rvecs, tvecs);
cv::aruco::drawAxis(image, cameraMatrix, distCoeffs, rvecs, tvecs, 0.1);
如果想绘制其他的虚拟物体，将前面得到的rvecs和tvecs应用到该虚拟物体的变换中即可。