What can we do in VR?

So, what can we do with VR? Let's explore this, but before we begin, it's worth it to point out that this medium is still in its infancy. At the time of this writing, we're on the first generation of consumer VR hardware and the vast majority of the population hasn't even seen a VR headset yet, much less experienced it. Try this: the next time you're in a restaurant or a public space, ask yourself how many of the people around you have likely ever seen a VR headset—a handful at best. Now, how many of them have watched a movie (a century-old medium), watched television (three-quarters of a century), or played a video game (just shy of half a century)? VR is that new. We haven't come close to discovering everything we can do with it.

With that in mind then, use these ideas as a map of the current state of things and some fodder for ideas, but realize that there's much much more that we haven't even thought of yet. Why shouldn't you be the one to discover something new?

Games in VR

As we discussed a moment ago, VR at its core creates an experience of presence. If you're developing a game for VR, this means that designs that focus on giving the player an experience of being in a place are good candidates for the medium. Skyrim VR and Fallout 4 VR do a fantastic job of making players feel as though they're really in these expansive worlds. Myst-like games that put the player into a space they can explore and manipulate work well too.

The addition of motion controllers to simulate hands, such as those supplied with the HTC Vive, Oculus Rift, and Oculus Quest, enable developers to create simulations with complex interactions, such as Job Simulator and Vinyl Reality, which wouldn't be possible using traditional game controllers. Tender Claws' Virtual Virtual Reality provides a great example, meanwhile, of achieving 6DoF-like control with the Oculus Go's 3DoF controller.

The immersive aspect of VR means that games that surround you with the experience, such as Space Pirate Trainer, work well because the player can interact with actors all around them and not just what's in front. This need to watch all around you can be a focus of your design.

The sensation of motion VR evoked in players turns fast-moving games such as Thumper and Ludicrous Speed into physically-engaging experiences, and games such as Beat Saber capitalize on the player's physical movements to turn the game into a fitness tool as well.

Games in VR present a few challenges too, though. This same experience of presence and physical movement that makes the experience so engaging can mean that not every game design is a great candidate for VR. Simply porting a 2D game into VR isn't likely to work. A Heads-Up Display (commonly abbreviated as HUD) placed over the scene in 2D space won't work in VR, as there's no 2D plane to put it on. Fast movements that could be perfectly fine in 2D may make players motion-sick in VR. The decision to make a game for VR needs to be a conscious choice, and you'll need to design with the medium's strengths and challenges in mind.

Note

When thinking about moving a game or a game design from 2D into VR, there are a few specific areas that need to be considered: will the movement scheme work in VR? How can the UI be designed to fit into the world in VR? Will the game fit within the performance constraints of VR? Does putting this game into VR improve the experience of playing it? We'll address all of these considerations—movement, UI, and performance, in later chapters.

 

 

Interactive VR

Interactive VR experiences aren't just limited to games. 3D painting applications such as Tilt Brush allow users to sculpt and paint in room-scale 3D and share their creations with other users. Google Earth VR allows users to explore the earth, much of it in 3D. Interactive storytelling experiences such as Colosse, Allumette, Coco VR, and others immerse users in a story and allow them to interact with the world and characters. Interactive VR applications and experiences can be built for productivity or entertainment and can take almost any form imaginable.

It's worthwhile to keep a few considerations in mind when thinking about creating an interactive VR application. The mouse and keyboard aren't generally available to users in VR—they can't see these devices to use them, so interactions are usually best designed around the controllers provided with the VR system. Text can be difficult to read in VR—display resolutions are improving, but they're still low enough that small text may not be readable. The lack of a 2D HUD means that traditional menus don't work easily—usually, these need to be built into the world or attached to the player's virtual hands (see Tilt Brush for an excellent example of this.)

Note

Input and output are the main considerations for interactive VR—how will the user communicate input to the system, and how do they get information back out of it? In both cases, you have to design around the strengths and weaknesses of the system. You don't have a 2D HUD or a mouse, but you do have objects that can be moved and manipulated in space. VR displays can't yet approach the resolution of a desktop monitor, so reading a lot of text may not work. Successful designs in VR take these factors into account and turn them into deliberate design choices.

Interactive VR offers incredible possibilities for entirely new ways of exploring and interacting, and it's likely that we haven't even begun to see the full range of possibilities yet.

 

VR cinema – movies, documentary, and journalism

The same experience of presence that makes VR so well-suited for certain types of games makes it a powerful medium for documentary and journalism applications. VR is able to immerse users in a circumstance or environment and can evoke empathy by allowing viewers to share an experience deeply. Chris Milk, a pioneering VR filmmaker, has referred to VR as the "ultimate empathy machine," and we think that's a fair description. Alejandro Iñárritu's CARNE y ARENA was awarded a special Oscar by the Academy of Motion Picture Arts and Sciences in 2017 to recognize its powerful use of the medium to tell a story with deep empathy. VR's capacity to create presence through immersion makes things possible that simply can't be done on a flat screen.

A player experiencing Alejandro Iñárritu's CARNE y ARENA at Los Angeles County Museum of Art

Film and video in VR can be presented in several ways, which generally boil down to the shape of the virtual screen on which the images are presented and whether those images are presented in monoscopic 2D or stereo 3D. Flat or curved surfaces are generally used to present media carried over from traditional film or television, while domes, panoramas, or spheres can be used to surround the viewer with a more immersive 2D or 3D experience.

Mono 360° video surrounds the viewer but lacks depth—it's simply mapped onto a sphere surrounding the player. This has the advantage of being easier to produce and requires far less storage and less expensive equipment, and for many scenes, the difference between this and true stereo may be difficult to detect. Most early VR videos were produced this way. Stereo 360° video is similarly mapped to a sphere around the player (we'll learn how to do this in a later chapter), but displays a different image to each eye for true stereo depth. (We'll learn how to do this too.) New approaches to volumetric video that use light fields, Light Detection And Ranging (LIDAR) and photogrammetry to map real environments into genuine 3D virtual environments are beginning to appear and will likely become more prevalent as technology matures and processing power increases. As of this writing, they're still fairly new, often expensive, and still largely confined to the realms of high-end professionals and academics.

Documentary and journalism pieces are most often filmed as live-action video shot on a 360° camera or rig, in mono or increasingly in stereo, allowing the viewer to look around and become immersed in a seamless sensory environment. 360° cinema is generally intended to be a direct, immersive, and engaging experience, but is usually not interactive. The viewer is generally not able to move freely through the scene except by triggering a cut to a new scene and generally can't affect the events that go on within the scene. 

Note

In planning a cinematic VR experience, two of the primary choices to make are the following: will the experience be presented in mono or stereo, and what's the shape of the virtual screen on which it will be displayed?

Cinematic VR is another area in which simply porting the the language of the flat screen isn't enough. There's no concept of a frame in 360° film, and no concept of a shot size such as a close-up or a long shot. VR filmmakers have to be very careful about moving the camera, as it's very easy to make viewers sick with a moving or shaky camera. Film-making in VR is still in its infancy, and we're beginning to learn the ways the grammar of the language differs from traditional film or television, but still have a long way to go before we'll fully understand the language of the new medium.

This hasn't stopped filmmakers such as Alejandro Innaritu, Nonny de la Pena, Chris Milk, and Felix and Paul from creating astonishing and powerful cinematic experiences in VR, and this highlights what an exciting time it is to be participating in the creation and discovery of a powerful and entirely new art form.

 

Variants of VR cinema include the following:

• Narrative stories
• Documentaries
• Journalism
• Concerts and happenings
• Sports
• Virtual tourism

Architecture, Engineering, and Construction (AEC) and real estate

VR is ideally suited for Architecture, Engineering, and Construction (AEC) planning, as it allows designers to explore and iterate quickly on designs, and it serves as an excellent communication tool between designers and clients. VR provides an immersive experience that allows the user to explore and review the space in a real-world scale in a way that simply isn't possible through any other medium.

Note

The Architecture, Engineering, and Construction industries are often bundled under the blanket initialism AEC.

For the same reasons that VR is such a useful tool for AEC, it's equally useful for real-estate applications, providing prospective buyers an opportunity to tour a home remotely, or to experience a space before it's been built. No medium represents space and scale better than VR.

Unreal Engine, as we'll see, is particularly well-suited for architectural applications, as its physically-based workflow for materials and lighting makes it possible to create surfaces that look real and respond to light the way their real-world counterparts would.

In addition to providing a realistic lighting and shading model ideal for the realistic representation of spaces, Epic Games (the makers of Unreal Engine), also provides a suite of tools designed for non-game uses such as architectural visualization. The most important of these is a toolkit called Datasmith, which allows high-detail scenes to be imported from architectural Computer-Aided Design (CAD) and 3D packages into Unreal with little or no modification required to reproduce the object placement, lighting, and shading from the original source.

 

 

Note

Architectural visualization is often shortened to archvis or archviz.

In terms of practical workflow, engineering and architecture environments for VR usually begin in a CAD or a 3D Digital Content Creation (DCC) tool, and are then brought into Unreal either by hand or by using the Datasmith workflow, where it can be made into an environment that can be explored in VR.

For real-estate applications, the environment may be fully modeled in 3D, or may be photographed as a 360° sphere or panorama, which provides less interactivity but is much easier and less expensive to produce. Even though it limits the user's movement, 360° photography can still provide an immersive sense of the space that the user couldn't experience otherwise.

Engineering and design

As with building planning, VR can be an outstandingly effective tool for engineering and other design applications. Designs can be tested in depth and iterated rapidly without requiring physical prototypes to be built and can be placed in virtual environments that allow them to be evaluated in context. Designers can use VR to explore designs and see how parts will fit together and to communicate with stakeholders in an experience that closely replicates the experience of actually handling and interacting with the object.

Education and training

It can be argued that VR began its life in education, in 1929, when Edwin Link created the Link Trainer to train aircraft pilots using an early immersive simulator. The combination of immersion and interaction makes VR a powerful tool for education, learning, and exploration. VR, at its core, is capable of providing a much more concrete and experiential understanding of a subject than other media. Where most other media communicate ideas, VR communicates direct experience. 

Traditional education often focuses on communicating facts to students, but facts in isolation can bore or overwhelm them if they don't yet have sufficient context to know what they need them for in the first place. VR, by contrast, can be used to allow students to discover and learn concepts by working directly with materials and representations of ideas they're exploring and learning, practicing real skills and turning abstract ideas into experience. Context is a natural by-product of immersion, and VR's ability to evoke presence can be instrumental in creating a physical, social, or emotional frame for the subject being learned. This can potentially make it meaningful or understandable to the student in ways that may not otherwise have been possible, and can allow students to explore the ways a complex system's parts fit together. 

VR also can aid concentration because it isolates the student's senses from distractions that aren't part of the topic being explored and can be effective at creating virtual social learning environments, such as virtual classrooms.

Educational VR can (and should) be made easy to use, immersive and engaging, and meaningful to the student and can allow students to learn at their own pace and use its interaction to fuel their own exploration and discovery.

Commerce, advertising, and retail

VR in commerce (the nickname, v-commerce, is sometimes used to describe it) offers a range of new ways for customers to experience products and can create opportunities to connect customers with products they may not otherwise have encountered. Car buyers, for instance, can explore color choices and options in a virtual car configurator that allows them to experience what their chosen options would look and feel like around them. This experience can also be instrumental in moving an aspirational purchase out of the imagination and into the realm of something that feels real.

For retailers, VR offers a way to reach customers who are not able to visit shops, increasing accessibility and the likelihood of sales. Customers can see more clearly and in context what a product is, reducing confusion and returns. VR can give a customer a chance to try before they buy, even where the product might be too large, too far away, or too elaborate to demonstrate effectively by other means. Virtual showrooms, for example, can allow customers to place furnishings together into a virtual environment that allows them to see how the pieces would fit together and how they might fit in their own space.

VR can be used as well to facilitate an emotional connection with the brand, placing the customer into a virtual environment or experience that supports the brand's emotional space, such as a mountaintop or a fashion show.

Medicine and mental health

VR offers promising opportunities as well in psychology, medicine, neuroscience, and physical and occupational therapy. VR, for example, can be used in physical therapy by slowing time and allowing patients to perform actions slowly and repeatedly and has been used successfully for pain management. VR is also useful for providing simulated virtual patients for medical and emergency training.

In the fields of mental and behavioral health, VR has powerful applications in assessment, training, and the treatment of stress-related disorders. Patients can be exposed to complex stimuli to help to assess and rehabilitate cognitive functions for stroke, traumatic brain injury, and similar neurological disorders.

So much else

The through line through all of the uses of VR described is that VR works especially well to communicate context and create meaning through presence and to allow complex physical interactions with objects that just couldn't be done with a flat screen. Without question, there are still more valuable uses of VR that haven't yet been discovered or considered. The only limit is our own imagination.