In this section, we do a brief survey of some of the tricky issues that AR researchers have struggled with in the past, and present and are likely to struggle with in the future.
Technical issues in relation to augmented reality
Field of view
In a theatre, on a computer screen, in a handheld mobile device, or in an AR headset, the angle from one edge of the view area to the opposite is commonly referred to as the angle of view or field of view (FOV). For example, a typical movie theatre screen is about 54 degrees; an IMAX theatre screen is 70 degrees. The human eye has about 160 degrees horizontal (monocular) or 200 degrees combined binocular FOV. The HTC Vive VR headset has about 100 degrees. Note that sometimes FOV is reported as separate horizontal and vertical; other times, it's a (better-sounding) diagonal measure.
Although not commonly discussed this way, when you hold a mobile device in front of you, it does have a field of view that is measured by the size of the screen and how far away you're holding it. So, an arm's length, about 18 inches away, is just 10 degrees or so. This is why you often see people preferring a large screen tablet for mobile AR rather than a phone.
When it comes to wearables, the expectations are greater. The Microsoft HoloLens FOV is only about 35 degrees, equivalent to holding your smartphone about 6 to 8 inches in front of your face or using a 15-inch computer monitor on your desk. Fortunately, despite the limitation, users report that you get used to it and learn to move your head instead of your eyes to discover and follow objects of interest in AR. Metavision Meta 2 does better; it's FOV is 90 degrees (diagonal).
The following image illustrates the effect of FOV when you wear a HoloLens device (image by Adam Dachis/NextReality):
Visual perception
The rendered image needs to satisfy the expectations of our visual perceptions to the extent that the goal of AR is to display virtual objects so they could realistically seem to reside in our physical environment. If the AR is just an overlay or annotation of the real world, then this may not be as important.
When rendering objects for 3D view, the views from the left and right eyes are offset slightly, based on your interpupillary distance (distance between the eyes), called parallax. This is not a problem and is handled in every VR and wearable AR device, but it's still worth mentioning.
Virtual AR objects coexisting in the real world that are in front of real objects should hide the objects behind them. That's easy; just draw the object on top. The opposite is not as simple. When the virtual object is behind a real-world one, say your virtual pet runs under a table or behind a sofa, it should be partially or completely hidden. This requires a spatial map of the environment; its mesh is used to occlude the computer graphics when rendering the scene.
An even more difficult problem comes up with photorealistic rendering of virtual objects. Ideally, you'd want the lighting on the object to match the lighting in the room itself. Suppose in the real world, the only light is a lamp in the corner of the room, but your AR object is lit from the opposite side. That would be conspicuously inconsistent and artificial. Apple ARKit and Google ARCore address this issue by capturing the ambient light color, intensity, and direction and then adjusting the virtual scene lighting accordingly, even offering the ability to calculate shadows from your virtual objects. This provides a more realistic render of your objects in the real world.
Focus
Photographers have known about depth of field since the beginning of photography. When a lens is focused on an object, things more in the foreground or further away may be out of focus; that range is called the depth of field. The iris in your eye is a lens too, and it adjusts to focus on near versus far objects, called accommodation. We can actually feel our iris changing its focus, and this oculomotor cue of stretching and relaxing also contributes to our depth perception.
However, using near-eye displays (in VR as well as AR), all the rendered objects are in focus, regardless of their distance perceived via parallax. Furthermore, the angle between your eyes changes when you're focused on something close up versus something further away, called vergence. So, we get mixed signals, focus (accommodation) on one distance and vergence at another. This results in what is called an accommodation-vergence conflict. This disparity can become tiring, at best, and inhibits the illusion of realism. This is a problem with both wearable AR and VR devices.
Potential solutions may emerge using eye tracking to adjust the rendered image according to your vergence. There is also the promise of advanced light-field technology that more accurately merges computer-generated graphics with real-world light patterns (see Magic Leap at https://www.magicleap.com).
Resolution and refresh rate
No discussion of AR displays would be complete without mentioning pixels, those tiny colored dots that make up the image on the display screen. The more, the better. It is measured in terms of resolution. What's more important, perhaps, is pixel density, pixels per inch, as well as the color depth. Higher density displays produce a crisper image. Greater color depth, such as HDR displays (high dynamic range), provides more bits per pixel so there could be a more natural and linear range of brightness.
We also talk about motion-to-photon latency. This is the time it takes for the AR device to detect changes in the location and orientation and have that represented on the screen. A lagged latency is not just unrealistic and feels sloppy, it can result in motion sickness, especially in wearable displays. Depending on the device, the screen may refresh in cycles of 60 frames per second or more. All the sensor readings, analysis, geometry, and graphics rendering must be performed with that tiny timeframe or the user will experience latency.
Ergonomics
Finally, the look and feel and comfort of devices is critical to their market acceptance and usefulness in practical situations. Handheld devices continue to get thinner and more lightweight. That's awesome, provided they continue to be important for AR.
Most of us agree that eventually all of this will move into wearable eyewear. Unless you're in an industrial hardhat environment, we all look forward to the day when AR eyewear becomes as lightweight and comfortable as a pair of sunglasses. Then, we'll wish for AR contact lenses (then, retinal implants?).
In 2009, Rolf Hainich described the ultimate display in his book The End of Hardware: Augmented Reality and Beyond as follows:
a nonintrusive, comfortable, high-resolution, wide-FOV, near-eye display with high dynamic range and perfect tracking.
