You've probably realized that, although we're using AR to position the game objects, the video feed is little more than a background. For example, the ball bounces or rolls on our game court floor, but otherwise isn't realistically interacting with our table top surroundings.
To be a better augmented reality, the app needs to see and understand the table top environment. Different AR toolkit SDKs approach this problem differently, if at all. The Vuforia toolkit, for example, provides a feature called Smart Terrain that builds a surface mesh from your device's video feed. Similarly (but quite differently), the Microsoft MixedRealityToolkit builds a spatial mesh using its depth sensors and HPU chip and then provides Unity components for surface and smart object understanding. Presently, we will use the Vuforia tools.
...
