Summary

This chapter introduced AR, along with a robust set of approaches to the problem of tracking an image in 3D space.

We began by learning the concept of 6DOF tracking and different ways of representing rotations. We recognized that familiar tools such as ORB descriptors, FLANN-based matching, and Kalman filtering are useful in this kind of tracking, but that we also needed to work with camera and lens parameters in order to solve the PnP problem.

Next, we addressed practical considerations of how best to represent a reference object (such as a book cover or a photo print) in the form of a grayscale image, a set of 2D keypoints, and a set of 3D keypoints.

We proceeded to implement a class that encapsulated a demo of image tracking in 3D space, with a 3D highlighting effect as a basic form of AR. Our implementation dealt with real-time considerations, such as the need to update the Kalman filter's transition matrix based on fluctuations in the frame rate.

Finally, we considered...