Understanding face tracking

Let's start with some background on face tracking and the technology that makes it work. Face tracking is a kind of Augmented Reality that (usually) uses the front-facing camera on your mobile device. Apps such as Snapchat, Instagram, and Animoji have popularized face filter technology, and it has now become mainstream on mobile devices. It makes for highly entertaining and creative experiences. The technology detects facial features and expressions, and Unity AR Foundation enables you to write applications for attaching 3D objects to specific facial features that are tracked.

Face tracking begins with a frame of the video from your device's camera. It analyzes the pixels, looking for patterns that represent a face – for example, the bridge of the nose is lighter than the pixels surrounding it, and the eyes are darker than the forehead. Key points and regions are recognized and used to construct a 3D mesh, like a mask, representing the...