Unreal Engine 4 Virtual Reality Projects: Build immersive, real-world VR applications using UE4, C++, and Unreal Blueprints

The most common way of immersing a user, and the one we'll be talking about in this book, is through the use of a Head-Mounted Display (HMD), often just referred to as a headset. (There are other ways of doing VR—projecting images on walls, for example, but in this book, we focus on head-mounted VR.) The user's headset displays the virtual world and tracks the movement of their head to rotate and shift the view to create the illusion that they're actually looking around and moving through physical space. Some headsets, though not all of them, include headphones to add to the illusion by enabling sounds in the environment to sound as though they're coming from their sources in the virtual world through a process called spatialized audio. 

Some headsets only track the direction the user is looking, while others can track changes to the user's position as well. If you're using a headset that tracks rotation but not position, and you lean forward to try to look more closely at an object, nothing's going to happen. The object will seem as though it's moving away from you as you try to lean in toward it. If you do this on a headset that tracks position as well, your virtual head will move closer to the object. We use the term Degrees of Freedom (DoF) to describe the ways objects can move in space. (Yes, it's OK to pronounce it doff. All of the developers do.) Take a look at the following points:

Headsets can either be tethered to a computer—as is the case with the Oculus Rift and the HTC Vive, which allows the full computing power of the attached PC to drive the visuals – or they can be self-contained devices such as the Samsung Gear, Oculus Go, Oculus Quest, and HTC Vive Focus. At the time of this writing, wireless connections between PCs and VR headsets are beginning to enter the market.

Most headsets also come paired with input devices that allow users to interact with the world, which can act as pointers or as hands. Handheld devices, as with headsets, can be tracked in three or six degrees of freedom. 3DoF devices such as the Oculus Go's controller are essentially pointers—users can aim them but can't reach out and grab something. 6DoF devices act much more like virtual hands and allow users to interact with the world in a much greater variety of ways.

One of the major mistakes many new developers make when first approaching VR is that they try to apply the traditional designs they're used to creating in 2D space to the VR space and, for the most part, this doesn't work. VR is its own medium, and it doesn't follow the same rules as the media that came before it. It's worth it to take a moment to look at what this means.

When most people first consider VR, they see the headset and assume that it's primarily a visual experience—traditional flat-screen media shown in stereo. It's understandable that it would seem this way, but their perception misses the point. Yes, the VR headset is (depending on whether or not it includes integrated audio) either primarily or entirely a display device, but the experience it creates for the user is very different than the experience created by a traditional flat screen.

Let's imagine for a minute that you're looking at a photo or a 2D video looking down over the edge of a tall building. You see the streets far below, but they don't really feel as though they're far below you. They're just small in the image. Take the same image, but now present it in stereo through a VR headset, and you'll probably experience vertigo. Why is this? Take a look at the following screenshot:

Non-immersive media, no matter how large or detailed, still leaves the viewer surrounded by reminders that the scene isn't real. Immersive media, on the other hand, seems to surround the user completely. (Scene: Soul:City Environment Pack, by Epic Games)

First, as we mentioned a moment ago, you're immersed in the experience. There's nothing else in the surrounding world to remind you that it isn't real. Let's jump back to our previous example—the building edge on your television—turn around and look behind you. Oh. You're just in your living room. Even when you look directly at it, the largest television you could possibly buy still leaves you with lots of peripheral vision to remind you that what you're seeing there isn't real. Everything on a flat screen, even a 3D screen, takes place on the other side of a window. You're watching, but you're not really there. In VR, the window is gone. When you look to the right, the world is still there. Look behind you, and you're still in it. Your perception is completely overtaken by an experience that has become an environment, not just a frame you're looking at.

Second, the stereo image creates a sense of real depth. You can see how far down the drop really goes. The cars in the street below aren't just small, they're far away. In a 6DoF headset that allows motion tracking, your movements in the real world are mirrored in the virtual world. You can lean over the edge or step back. This mixture of immersion, real depth perception, and natural response to your movement comes together to convince your body that what you're perceiving is real. We call this phenomenon presence, and it's a sensation that's mostly experienced physically.

The mechanics of immersion and the resulting experience of presence are unique to VR. No other medium does this.

While we're on the topic of presence, it's worth pointing out that it's a fragile phenomenon, and the current state of VR technology still faces a few challenges to creating a sense of presence fully and reliably. Some of these are rooted in hardware and are almost certainly going to go away as the technology advances. Users can feel the headset on their face, for example, and on wired headsets, they can feel the cable running from the headset. The current generation of headsets offers a field of view that's too narrow to provide peripheral vision. (The desktop devices offer a 110° field of view, but your eyes, meanwhile, can perceive a field twice as wide.) Display resolutions aren't yet high enough to keep users from being able to see individual pixels (VR users call this the screen door effect), and finicky optics can blur the user's vision if they're not perfectly aligned. This means, in practice, that it's hard to read small text on a VR headset, and that users are sometimes reminded of the hardware when they have to adjust it to get back into the sweet spot for the lenses.

Looking at the state of things, though, it's obvious that these hardware challenges won't last forever. Self-contained and wireless headsets are quickly entering the market, with increasingly reliable tracking that no longer relies on external equipment. Displays are getting wider, resolutions are getting higher, and optical waveguides show great promise for lighter displays with wider in-focus regions. VR works extremely well already, and it's easy to see how it's going to continue to improve.

There are a few other things that can break presence that we can't do as much about—hitting a desk accidentally with a controller, for example, or running into furniture, losing tracking, or hearing sounds from outside the experience. We can manage these when we have control over the user's space, but where we don't, there's not much we can do.

Even given these limitations, though, think about how profoundly the current generation of VR can create a sense of presence in a user, and realize that it only gets better from here. Users believe what they experience in VR to a degree that simply doesn't happen with other media. They explore and learn in ways that aren't possible in any other way. They empathize and connect with people and places more deeply than they could in any way, other than physically being there. Nothing else goes as deep. And we're only getting started.

As we discussed a moment ago, VR at its core creates an experience of presence. If you're developing a game for VR, this means that designs that focus on giving the player an experience of being in a place are good candidates for the medium. Skyrim VR and Fallout 4 VR do a fantastic job of making players feel as though they're really in these expansive worlds. Myst-like games that put the player into a space they can explore and manipulate work well too.

The addition of motion controllers to simulate hands, such as those supplied with the HTC Vive, Oculus Rift, and Oculus Quest, enable developers to create simulations with complex interactions, such as Job Simulator and Vinyl Reality, which wouldn't be possible using traditional game controllers. Tender Claws' Virtual Virtual Reality provides a great example, meanwhile, of achieving 6DoF-like control with the Oculus Go's 3DoF controller.

The immersive aspect of VR means that games that surround you with the experience, such as Space Pirate Trainer, work well because the player can interact with actors all around them and not just what's in front. This need to watch all around you can be a focus of your design.

The sensation of motion VR evoked in players turns fast-moving games such as Thumper and Ludicrous Speed into physically-engaging experiences, and games such as Beat Saber capitalize on the player's physical movements to turn the game into a fitness tool as well.

Games in VR present a few challenges too, though. This same experience of presence and physical movement that makes the experience so engaging can mean that not every game design is a great candidate for VR. Simply porting a 2D game into VR isn't likely to work. A Heads-Up Display (commonly abbreviated as HUD) placed over the scene in 2D space won't work in VR, as there's no 2D plane to put it on. Fast movements that could be perfectly fine in 2D may make players motion-sick in VR. The decision to make a game for VR needs to be a conscious choice, and you'll need to design with the medium's strengths and challenges in mind.

Interactive VR experiences aren't just limited to games. 3D painting applications such as Tilt Brush allow users to sculpt and paint in room-scale 3D and share their creations with other users. Google Earth VR allows users to explore the earth, much of it in 3D. Interactive storytelling experiences such as Colosse, Allumette, Coco VR, and others immerse users in a story and allow them to interact with the world and characters. Interactive VR applications and experiences can be built for productivity or entertainment and can take almost any form imaginable.

It's worthwhile to keep a few considerations in mind when thinking about creating an interactive VR application. The mouse and keyboard aren't generally available to users in VR—they can't see these devices to use them, so interactions are usually best designed around the controllers provided with the VR system. Text can be difficult to read in VR—display resolutions are improving, but they're still low enough that small text may not be readable. The lack of a 2D HUD means that traditional menus don't work easily—usually, these need to be built into the world or attached to the player's virtual hands (see Tilt Brush for an excellent example of this.)

Interactive VR offers incredible possibilities for entirely new ways of exploring and interacting, and it's likely that we haven't even begun to see the full range of possibilities yet.

The same experience of presence that makes VR so well-suited for certain types of games makes it a powerful medium for documentary and journalism applications. VR is able to immerse users in a circumstance or environment and can evoke empathy by allowing viewers to share an experience deeply. Chris Milk, a pioneering VR filmmaker, has referred to VR as the "ultimate empathy machine," and we think that's a fair description. Alejandro Iñárritu's CARNE y ARENA was awarded a special Oscar by the Academy of Motion Picture Arts and Sciences in 2017 to recognize its powerful use of the medium to tell a story with deep empathy. VR's capacity to create presence through immersion makes things possible that simply can't be done on a flat screen.

A player experiencing Alejandro Iñárritu's CARNE y ARENA at Los Angeles County Museum of Art

Film and video in VR can be presented in several ways, which generally boil down to the shape of the virtual screen on which the images are presented and whether those images are presented in monoscopic 2D or stereo 3D. Flat or curved surfaces are generally used to present media carried over from traditional film or television, while domes, panoramas, or spheres can be used to surround the viewer with a more immersive 2D or 3D experience.

Mono 360° video surrounds the viewer but lacks depth—it's simply mapped onto a sphere surrounding the player. This has the advantage of being easier to produce and requires far less storage and less expensive equipment, and for many scenes, the difference between this and true stereo may be difficult to detect. Most early VR videos were produced this way. Stereo 360° video is similarly mapped to a sphere around the player (we'll learn how to do this in a later chapter), but displays a different image to each eye for true stereo depth. (We'll learn how to do this too.) New approaches to volumetric video that use light fields, Light Detection And Ranging (LIDAR) and photogrammetry to map real environments into genuine 3D virtual environments are beginning to appear and will likely become more prevalent as technology matures and processing power increases. As of this writing, they're still fairly new, often expensive, and still largely confined to the realms of high-end professionals and academics.

Documentary and journalism pieces are most often filmed as live-action video shot on a 360° camera or rig, in mono or increasingly in stereo, allowing the viewer to look around and become immersed in a seamless sensory environment. 360° cinema is generally intended to be a direct, immersive, and engaging experience, but is usually not interactive. The viewer is generally not able to move freely through the scene except by triggering a cut to a new scene and generally can't affect the events that go on within the scene. 

Cinematic VR is another area in which simply porting the the language of the flat screen isn't enough. There's no concept of a frame in 360° film, and no concept of a shot size such as a close-up or a long shot. VR filmmakers have to be very careful about moving the camera, as it's very easy to make viewers sick with a moving or shaky camera. Film-making in VR is still in its infancy, and we're beginning to learn the ways the grammar of the language differs from traditional film or television, but still have a long way to go before we'll fully understand the language of the new medium.

This hasn't stopped filmmakers such as Alejandro Innaritu, Nonny de la Pena, Chris Milk, and Felix and Paul from creating astonishing and powerful cinematic experiences in VR, and this highlights what an exciting time it is to be participating in the creation and discovery of a powerful and entirely new art form.

Variants of VR cinema include the following:

VR is ideally suited for Architecture, Engineering, and Construction (AEC) planning, as it allows designers to explore and iterate quickly on designs, and it serves as an excellent communication tool between designers and clients. VR provides an immersive experience that allows the user to explore and review the space in a real-world scale in a way that simply isn't possible through any other medium.

For the same reasons that VR is such a useful tool for AEC, it's equally useful for real-estate applications, providing prospective buyers an opportunity to tour a home remotely, or to experience a space before it's been built. No medium represents space and scale better than VR.

Unreal Engine, as we'll see, is particularly well-suited for architectural applications, as its physically-based workflow for materials and lighting makes it possible to create surfaces that look real and respond to light the way their real-world counterparts would.

In addition to providing a realistic lighting and shading model ideal for the realistic representation of spaces, Epic Games (the makers of Unreal Engine), also provides a suite of tools designed for non-game uses such as architectural visualization. The most important of these is a toolkit called Datasmith, which allows high-detail scenes to be imported from architectural Computer-Aided Design (CAD) and 3D packages into Unreal with little or no modification required to reproduce the object placement, lighting, and shading from the original source.

In terms of practical workflow, engineering and architecture environments for VR usually begin in a CAD or a 3D Digital Content Creation (DCC) tool, and are then brought into Unreal either by hand or by using the Datasmith workflow, where it can be made into an environment that can be explored in VR.

For real-estate applications, the environment may be fully modeled in 3D, or may be photographed as a 360° sphere or panorama, which provides less interactivity but is much easier and less expensive to produce. Even though it limits the user's movement, 360° photography can still provide an immersive sense of the space that the user couldn't experience otherwise.

As with building planning, VR can be an outstandingly effective tool for engineering and other design applications. Designs can be tested in depth and iterated rapidly without requiring physical prototypes to be built and can be placed in virtual environments that allow them to be evaluated in context. Designers can use VR to explore designs and see how parts will fit together and to communicate with stakeholders in an experience that closely replicates the experience of actually handling and interacting with the object.

It can be argued that VR began its life in education, in 1929, when Edwin Link created the Link Trainer to train aircraft pilots using an early immersive simulator. The combination of immersion and interaction makes VR a powerful tool for education, learning, and exploration. VR, at its core, is capable of providing a much more concrete and experiential understanding of a subject than other media. Where most other media communicate ideas, VR communicates direct experience. 

Traditional education often focuses on communicating facts to students, but facts in isolation can bore or overwhelm them if they don't yet have sufficient context to know what they need them for in the first place. VR, by contrast, can be used to allow students to discover and learn concepts by working directly with materials and representations of ideas they're exploring and learning, practicing real skills and turning abstract ideas into experience. Context is a natural by-product of immersion, and VR's ability to evoke presence can be instrumental in creating a physical, social, or emotional frame for the subject being learned. This can potentially make it meaningful or understandable to the student in ways that may not otherwise have been possible, and can allow students to explore the ways a complex system's parts fit together. 

VR also can aid concentration because it isolates the student's senses from distractions that aren't part of the topic being explored and can be effective at creating virtual social learning environments, such as virtual classrooms.

Educational VR can (and should) be made easy to use, immersive and engaging, and meaningful to the student and can allow students to learn at their own pace and use its interaction to fuel their own exploration and discovery.

VR in commerce (the nickname, v-commerce, is sometimes used to describe it) offers a range of new ways for customers to experience products and can create opportunities to connect customers with products they may not otherwise have encountered. Car buyers, for instance, can explore color choices and options in a virtual car configurator that allows them to experience what their chosen options would look and feel like around them. This experience can also be instrumental in moving an aspirational purchase out of the imagination and into the realm of something that feels real.

For retailers, VR offers a way to reach customers who are not able to visit shops, increasing accessibility and the likelihood of sales. Customers can see more clearly and in context what a product is, reducing confusion and returns. VR can give a customer a chance to try before they buy, even where the product might be too large, too far away, or too elaborate to demonstrate effectively by other means. Virtual showrooms, for example, can allow customers to place furnishings together into a virtual environment that allows them to see how the pieces would fit together and how they might fit in their own space.

VR can be used as well to facilitate an emotional connection with the brand, placing the customer into a virtual environment or experience that supports the brand's emotional space, such as a mountaintop or a fashion show.

VR offers promising opportunities as well in psychology, medicine, neuroscience, and physical and occupational therapy. VR, for example, can be used in physical therapy by slowing time and allowing patients to perform actions slowly and repeatedly and has been used successfully for pain management. VR is also useful for providing simulated virtual patients for medical and emergency training.

In the fields of mental and behavioral health, VR has powerful applications in assessment, training, and the treatment of stress-related disorders. Patients can be exposed to complex stimuli to help to assess and rehabilitate cognitive functions for stroke, traumatic brain injury, and similar neurological disorders.

The through line through all of the uses of VR described is that VR works especially well to communicate context and create meaning through presence and to allow complex physical interactions with objects that just couldn't be done with a flat screen. Without question, there are still more valuable uses of VR that haven't yet been discovered or considered. The only limit is our own imagination.

When VR works, as we discussed earlier, it works through a process we call immersion, which we described earlier as a perceived experience of being physically present in a virtual world. For an experience to be immersive, a few things need to be true.

Immersion in VR serves a single goal—the creation of an experience of presence in the user. Presence, as we previously defined it, is a sensation of being in a place, and this is, in large part, a phenomenon perceived physically. Very often, they respond physically and instinctively to things in the world such as heights or objects flying toward them. The body largely believes what it perceives in VR and responds accordingly. If you think of presence primarily in physiological terms, you'll have an easier time understanding how users experience it. What does this experience make your user feel?

A key to understanding presence is to understand that VR doesn't so much work by trying to simulate an environment accurately as it does by triggering and fooling a range of systems that we use to perceive the world. This is one of the reasons why we can get away with low detail in our textures if we get the movement of the world right and keep latency low—our perceptive systems are much more aware of motion than detail. VR doesn't have to fool all the senses—just the right ones in the right ways.

Are you sensing a pattern here? You absolutely must maintain frame rate. High latency will pull the user right out of immersion, and this is a leading trigger for simulator sickness. Consider the work you're asking the renderer to do in VR, and you'll see that this is going to be a bit of a challenge. The HTC Vive Pro displays a 2,880 x 1,600 image (1,400 x 1,600 per eye), while the original Vive and the Oculus Rift display 2,160 x 1,200 (1,080 x 1,200 per eye), and all of them require this to happen 90 times per second, leaving the renderer 11 milliseconds to prepare the frame. The Oculus Go displays 2,560 x 1,440 pixels (1,280 x 1,440 per eye) 72 times per second, meaning the renderer has about 13 milliseconds to deliver the frame. The Unreal Engine renderer is blazingly fast, but, even so, this is a lot to render, and there is not a lot of time in which to get the frame drawn. You're going to have to make some compromises to reach your target. We'll talk about ways to do this throughout this book.

Here's a list of headsets currently on the market and their rendering demands.

Beyond dropping frames, the next most common cause of simulator sickness is the sensory conflict we mentioned earlier—a mismatch between motion perceived visually and motion felt in the inner ear. There are two major types of motion you're going to need to accommodate in VR:

Movement of the player's avatar is handled by the locomotion system you implement for your experience. You really don't have a choice here—you're going to have to create movement that isn't happening in real life, but there are things you can do to make this less of a problem and we'll talk about them shortly.

You should never interfere, however, with movement of the player's head.

What this means in practice is: never move the camera in a way the user didn't cause by their own actions. If you're making a game and the user's avatar dies, don't leave the camera bolted to the head as the body falls. You will almost definitely make users sick if you do this. Consider cutting to a third-person view instead or handling the action in some other way. Never move the camera to force the user to look around in a cinematic, and don't apply a walking bob or a camera shake. The user should control their head always.

This applies both ways. If the player moves their head, the camera must move, even if the game is paused or loading. Never stop tracking.

If you need to teleport the user to a new location or change cameras for any reason, consider using a fast fade to black or white to cover the transition. People instinctively blink when they turn their heads quickly, and it's a good idea to mimic this behavior.

In-game cut scenes need to be handled differently in VR than they would be on a traditional flat screen for the same reasons. Ordinarily, in authoring a cut scene, you would take control of the camera, moving and cutting from shot to shot, but you can't do this in VR. You can't control where your user is going to look, and you need to be careful moving them around. This leaves you with a few options. First, if your scenes are pre-rendered, then you really have no choice but to map them on to a screen in the virtual environment. This breaks immersion, but is no more difficult for the user than watching a movie in real life. If you're doing them in-engine, you need to think about how you're going to handle the player's point of view.

For a first-person point of view, it's probably best to stage the cinematic scene around the user and allow them to look and move freely within it. You can't cut away to another shot when doing this, and you can't guarantee that your user will be looking where you want them to look when a key moment occurs, but it's the most immersive approach you can take.

Cut scenes can also be handled in the third-person, in which you pull the user's viewpoint out of their body and allow them to view the scene unfolding, but you need to do this carefully—the out-of-body experience can be disorienting for your player and can weaken immersion and the player's identification with the character.

For film-making in VR, be very careful of the ways you move the camera. Even very small moves may induce sickness. Users will tolerate forward movement more easily than side-to-side or rotational movement and seem to tolerate movement more easily if it's justified by a visible vehicle or some other way of explaining why it's happening.

Thinking about how to use the camera in VR isn't just about managing user discomfort either. This is new territory, and the rules you learned from film and gaming work differently here. You're designing to recreate the user's eyes, not a camera, and this has far-reaching implications for your compositions. How does the user move? Do they know where you want them to look? What do they see when they look at their hands? What about a mirror? How does surrounding the user with a world (instead of making them watch it through a window) change their relationship to it? All of these factors require a conscious choice as you develop your work.

Depending on the type of application you're creating, you're probably going to need to give your users a way to change their location, either by teleporting or moving smoothly. (We'll dig into this in depth in a later chapter.) If you do choose to implement a smooth movement method though, don't accelerate or decelerate as the player starts and stops moving. Start the movement at full speed, or if you opt to smooth your starts and stops at all, keep them very short. (And, of course, never do a start-moving or stop-moving animation that takes control of the user's camera.)

We mentioned a moment ago that VR mimics the user's eyes, not a camera. For this reason, don't do things in your simulation that eyes don't do in real life. Never change the focal length of the camera lens or its depth of field. The eyes' focal lengths don't change the way cinematic zoom lenses do, and you're very likely to make your user sick if you change this.

Manipulating the depth of field isn't a good idea in current-generation VR, as we don't yet have a reliable way to know what the user is actually looking at within the view. In the future, as eye-tracking improves, this will likely change, but for now, don't make this choice for your user.

Motion blur shouldn't be applied to your camera or objects in the scene. This is an artifact of the way film photographs a static frame for a fixed period of time, smearing the motion within that frame, but that's not the way eyes work, and it will look unnatural in VR.

While we're on the topic, steer clear of other camera-mimicking effects, such as lens flares and film grain. Again, these mimic the behavior of film, not the eyes, and we're not trying to mimic film in VR. Filmic effects such as these can also cause unwanted physical side effects in the user, contributing to simulator sickness if the effects don't line up between the eyes, and they cost precious frame time to render. Don't use them.

Have you ever looked out the window of a car sitting still, and watched a large vehicle such as a truck or bus moving, and felt as if you were moving in the opposite direction instead? This phenomenon is called vection, and it refers to the illusion of self-movement produced by optical flow patterns. If a large portion of your view is moving, this can produce sensations of movement in your body, and as we discussed earlier, sensations of movement that don't match the signals from the inner ear can trigger simulator sickness.

What this means in practice is that, if a big chunk of your user's view is moving, you're at risk of inducing simulator sickness. We've talked about this already with regard to moving the user's head (don't do it), and we've touched on some of the ways we can handle this in your locomotion system, but you'll also want to be aware of other circumstances that can cause vection.

Be aware of moving patterns that fill large parts of the frame—whether or not they're part of your locomotion system, they can still create an illusion of motion, which may be a problem for your users.

Several games and applications have experimented with a tunnel vision effect to reduce vection when users need to move quickly through the environment—when the player's avatar runs, an iris closes in from the edges of the view to reduce peripheral vision.

Users seem to be much more tolerant of forward movement than they are of strafing—moving side-to-side. This may in part be because, in real life, we move forward far more than we move sideways, but it may also be because the optical flow the user sees when moving forward still has a relatively fixed point at the center, whereas in sideways movement, everything in the view moves.

Rotating the player's view is especially problematic. It moves pretty much everything in the view, and the vestibular system is especially tuned to detecting rotation. Be very careful here. Smooth rotations are generally not a good idea, but developers have found that snapping the user to a new rotation works well to reorient the user without making them sick. It turns out that the brain is very good at filling in interruptions in perception, so snapping to a new rotation or "blinking" the view during a large movement can be very effective at disrupting the perception of motion without distracting the user.

Many developers have also found that giving users a visible vehicle that moves with them, such as an aircraft cockpit, can mitigate the effect of vection when rotating. Whether this is an appropriate solution for you depends on the type of experience you're creating, but the takeaway here should be that users seem to be less prone to simulation sickness if they're given fixed points of reference in their view. Where this is appropriate, consider factoring it into your design, and where it isn't, consider other ways of breaking the optic flow if you have to do large smooth movements such as blinking or snapping.

If you're allowing your user to move smoothly through your environment, be aware that certain features of environments can provoke simulator sickness when users navigate them. Stairs are especially bad. Stairs that provide collision for every step so the view bounces when user navigates it are worse. Environment features such as these that create a sense of vertical movement when traversed can be difficult because the inner ear is very sensitive to changes in altitude.

Avoid stairs if you can. If you can't avoid them, be conscious of how steep they are and how fast you're letting your user move over them. You'll have to test a bit to get it right.

Be careful of using bright lights and strong contrasts in your scene. Bright lights contribute to simulator sickness in some users, and strong contrasts can increase the user's sense of vection as the world moves across their view. Also, with current hardware, bright lights can often create a flare on the headset's fresnel lenses, which can pull users out of immersion by reminding them of the hardware they're wearing. In general, it's recommended that you use cooler shades and dimmer lights than you normally would.

VR communicates the scale of objects in the world in ways that flat screens simply do not. Each of us sees the world in stereo vision through a pair of eyes that are a fixed distance apart. This distance, called Interpupillary Distance (IPD), contributes to our sense of how large or small objects in the world appear. Most VR headsets can be adjusted to match the interpupillary distance of their user and should be adjusted correctly to minimize eyestrain.

What this means for you as a developer is that the scale of objects in your world matters. On a flat screen, the user is limited to comparing the size of an object to another object to determine how large it is, but in VR, the user's IPD drives an absolute sense of scale. An object that's too large or too small on a flat screen will still appear normal if it's alone on the screen. The same object in VR, even if there's nothing to which it can be compared, will look wrong to a viewer in stereo 3D.

Some users may be prone to simulator sickness if the scale of the world feels wrong, and even those who aren't will still likely feel that the world feels "wrong," without necessarily knowing why.

Make sure objects in your world are scaled correctly. In Unreal, by default, one Unreal Unit (UU) is equal to one centimeter.

Your users in VR are moving around the real world wearing electric blindfolds. Respect this, and be careful what you ask them to do in VR. Take care when asking users to swing their arms, run, or strafe, as they can easily run into obstacles or walls in the real world. For headsets with cables, don't ask users to turn repeatedly in the same direction and tangle themselves in the cable. Be conscious as well of asking users to reach for objects on the floor or outside their normal reach area—this may not be easy or possible in their real-world physical environment. As mentioned earlier, avoid shifting the horizon in ways that could cause your user to lose balance. Remember that nearly all of the user's information about the world is coming from the VR simulation while they're in it—be conscious of how this information lines up with or contradicts what's in the invisible physical world around them.

The eyes use muscles to focus on objects and orient the eyes, and these muscles, like any other, can get fatigued. We call this eyestrain. Symptoms of eyestrain can include headaches, fatigue, and blurred or double vision. As a designer, there are things you can do to minimize eyestrain in your users, and understanding a little about what causes eyestrain will help you do this.

First, eyestrain can be caused by flickering. We've already talked a lot about the importance of keeping latency low—this is another reason to keep low latency a priority. Don't create purposely flickering content, as this can produce eyestrain but could also trigger photosensitive seizures.

Second, the eyes need to do some physical work to focus on an object in 3D space. They have to adjust the shape of their lenses to focus on the object (this is called accommodation), and they need to aim themselves so their lines of sight converge at the object. This is called vergence. We naturally have a reflex that correlates these two actions with each other, so the eyes naturally want to converge to a depth plane that matches the depth to which their lenses are focusing, and the lenses naturally want to focus in a way that matches where the eyes are converging. The problem comes in VR, where the actual images the eyes are seeing are a fixed distance away, but the content of those images exist at a variety of virtual depth planes, so the eyes still have to rotate so they converge at the objects they're looking at. This creates a conflict, as the focal depth the lenses are accommodating doesn't match the depth at which the eyes are converging, and it can cause eyestrain.

You can manage this when designing your world by keeping these two demands in mind. The fresnel lenses on the HMD make the headset screen appear to be about 1 m from the eyes, allowing the lenses to accommodate to a focal plane about 1 m away. The user's eyes, then, will naturally find it easier to focus on objects in the virtual world that appear to be about that far away. In practice, objects are most easily viewed at a range of 0.75 m to 3.5 m, with 1 m seeming to be ideal. Avoid making users look for long periods at objects less than half a meter away from the eye. 

Don't force your user to be an eyeball contortionist to view your user interface. Attaching a GUI to the user's face is usually a bad idea—as they turn their head to view a UI element, it appears to "run away" because it's attached to the same head that's turning to try to look at it, so users have to turn their eyeballs alone to focus on it. Don't do this to them. It's irritating to users, fatiguing, and has no real-world analogue. Put your UI in the world so your users can focus on it from comfortable viewing angles and at a comfortable distance. Attaching UI elements to the user's body, such as a wrist, can work well as it allows users to bring it into view when they want to interact with it. Putting GUI elements into a cockpit or vehicle can work well too. UI elements can be placed around the world and revealed when the user looks at them.

Keep GUI elements within the ideal range we discussed and at an angle that allows it to be read without straining, if you do wind up attaching it to the user's head. 

Try to avoid creating situations that force the user to change focal distance rapidly and often. If you're making a shooter, for example, that puts critical information on a nearby UI element while the enemies are in the distance, you may be creating a situation that will force your user to change focus frequently to check the UI and focus on enemies in the field. In a flat-screen game, this wouldn't be a problem, but in VR, it will tire them out. Design your UI in such a way that the user can get critical information without focusing on it—easy-to-read graphical elements, for example, or consider putting UI elements over the enemies' heads.

GUI elements can be occluded by objects in the world that are nearer to the camera than the UI element is. Don't try to use tricks from 2D gaming space to change this. In 2D game design, it's common to draw a UI element over a 3D element even if that element would really block the player's view of it. If you do this in VR, however, you'll create a confusing stereo image that won't be at all comfortable to look at. Accept the reality that your UI exists as a physical object in the world and follows the same rules as other physical objects.

Presence, when it's achieved in VR, produces strong reactions. It's an intimate experience, a visceral experience, and sometimes a fear-inducing experience. Be conscious of what you're doing as you craft experiences—you can easily trigger a fight-or-flight response in some users. This might be exactly what you intend, and we're not suggesting that you shy away from whatever it is you're trying to create. But be aware that you can be playing with strong stuff here and make intentional choices. VR is much more capable of triggering phobias than its flat-screen predecessors because the user is immersed in the space and not being constantly reminded by their peripheral vision that what they're seeing isn't true. Be on the lookout for circumstances that can induce vertigo, claustrophobia, fear of the dark, fear of snakes, spiders, or other phobias. Remember also that users will react more strongly to threats within their personal space.

VR puts demands on the user's body, eyes, and mind that other media don't. They're wearing a device on their head, and often standing or moving physically. Design your experience to let them exit whenever they want to or need to and resume later on. Let them take breaks as they need them.

In contrast to games and applications on flat screens, users in VR can't do anything else while they're waiting for the application to load. Optimize to keep your load times short. Remember as well that, even during a load, your application must be responding to the user's head tracking.

VR is in its infancy as a medium and an art form. It's far too early to pretend we know what its rules are really going to turn out to be. In the early days of film, actors were always filmed in full-frame, because the conventional wisdom at the time was that audiences wouldn't pay to see half an actor. Be equally willing to question the guidelines and advice you receive in VR design. These represent the current best understanding of what seems to work, but that doesn't mean that there aren't other ways to do things that haven't been tried. Be open to them. This is part of the reason why these guidelines were each presented with information about why they exist—so you can understand where they're coming from and make your own choices and try your own experiments. You're on a frontier in VR, part of the creation of an entirely new means of communication. Don't be afraid to explore.

The first thing to do in developing a design is to decide what it's for. This may sound obvious, but it happens all too often that developers jump right into a project and start building without first taking a step back and figuring out what it is that they're really trying to make and who it's for. The result, more often than not, is either an unfocused experience that doesn't really achieve what it was intended to do because the parts aren't all working together to support a common goal, or a project that takes a long time to complete. This wastes a lot of work, as developers discover things that need to change and have to throw out existing work to make the changes. By taking some time to plan before you begin building software, you can save yourself a lot of effort and make it more likely that the project will succeed.

The first thing to remember in design is that the more you build, the more difficult and expensive changes get, so try to make these decisions as early in the process as you can. The cheapest prototype you can make is in your own mind. The second cheapest is on paper. Once you start building software, start with the bare minimum you need to get your project running—a gray box environment, or a simple prototype, and test it to see how it needs to change. You're almost guaranteed to discover a few things you hadn't anticipated, and this is the time to discover these things and change what you need. Once you've gone through this process, discovered what really works and what doesn't, and adjusted your design to respond to what you learned, now you're ready to begin putting expensive art and polish into the work. Too many developers work backward and try to make the final product right out of the gate, and they get locked into decisions that could have been changed more easily if the legwork had been done first.

With that in mind, the first thing to think about is who the project is for and why you're making it for them. Is this a game or an entertainment experience? What do you want the user to feel? What will they be doing while they play or participate in the experience? The same questions apply to cinematic VR—what's this experience about? What story are you trying to tell? Take a moment to write this down.

If you're making a learning experience, what does the user need to learn? What's the best way to teach it?

If you're making an architecture or design visualization application, what's important to your end user? An architect or engineer may want to be able to look inside walls and structures to see electrical and plumbing designs, while a real-estate buyer may care more about the quality of light in the space.

Figure out who your user is, and what's important to them, and clarify what you're trying to create and what's important to you. This should be done on paper. It's very easy for vague design elements to hide in your mental model, only to reveal holes or unexpected questions when you start to write them down.

Pretty much the next thing you should do once you've clarified your design intention for a VR project is to think about how it fits in VR.

Think about your project in terms of what VR allows you to do. Does it rely on immersion and a strong sense of presence to work? Is it about making use of VR's ability to simulate the body or to give context to information? Why does your project work better in VR than it would on a flat screen? What can your user do or experience that they couldn't in traditional media?

Think about the challenges VR imposes as well. As we've seen in the best practices mentioned, VR imposes a different set of challenges than traditional media. Simulator sickness is a major concern—does your project require you to move the camera in ways that are going to be uncomfortable for your users? Does it rely on your users to move in a way that may be difficult or impossible in VR? Are you asking your users to read lots of small text that may not be legible on current headsets? Think about the practices we outlined, and evaluate whether any of them pose challenges to your design. This doesn't necessarily mean your design can't work in VR, but it does mean that you'll have to do some additional design thinking to work through those challenges.

Your choice to put your project into VR should be made deliberately. You should be able to describe why your project works better in VR than in traditional media, and how you plan to handle the challenges VR imposes. This too should happen in writing. You'll probably discover opportunities you hadn't seen, and a few challenges you hadn't realized you'd need to overcome. Writing these down will help you to understand what's important about your project and what needs to happen for it to succeed.

Now that you've clarified who your project is for, what you intend it to do, and why it makes sense to do it in VR, you're ready to begin figuring out what it's really going to take to build it. It's helpful to figure this out in terms of a Minimum Viable Product (MVP). This, simply put, is a version of the product that contains only what's needed for it to satisfy its intention. An architectural visualization project, for example, needs to put the viewer into the building at the correct scale, and give the user some way to move around and see what it looks and feels like from different perspectives. What your MVP contains is your choice as a designer, but you should be clear about whether the thing you're talking about is a thing you need or a thing you want. If the project simply isn't worth doing if you can't get a given feature into it, then it's a needed feature and should go into your MVP. If it would improve the experience but users could still get what they needed without it, it's not part of the MVP.

The contents of your MVP will differ greatly between different types of projects—the needs of a cinematic VR experience are substantially different than those of an engineering visualization application, but as a designer, you should know what they are and write them down. You don't need to write a book or an essay here—a list of bullets should be enough, but for each item on the list, ask yourself whether the project could still do what it's intended to do without it, and be clear about your answers. Wants, even strong wants, aren't needs. The point here is to know where your floor is.

Be on the lookout as well for things you missed. Imagine your user using your project—what are they trying to do, from moment to moment, from the moment they start up the application until the moment they shut it down? Use this exercise to discover items you missed, and figure out whether they're wants or needs, and get them on to the list if they belong there.

If you've gone through the preceding exercises, you should have a clear idea of what your project is for, why it works in VR, and what needs to be in it for it to work. Now you're ready to figure out how to do it.

For the items in your MVP, what do you need to make them exist? Do you need a UI element to display information to the user? Do you need a way for your user to move around? Does the user need to be able to load or save information, or connect to a server?

For each item in your list, figure out what that item really requires you to build and write it down. You should come out of this exercise with a pretty clear breakdown of the things you need to do to get your project built.

Look through this list—where are the big jobs, and where are the big risks? Can you achieve all of this with the time and resources you have? Do you need to re-evaluate your scope if it's starting to look too big? Bear in mind that it's almost always better to do fewer things well than to try to do everything and do a poor job of it. It's common, at this point, to discover that the project scope exceeds what you can realistically do well, and this is a good thing. The time to discover this is now while it's still on paper and you can reorganize the work, move items off the MVP, or change your schedule or resources. If you discover these problems near the start, you have a fighting chance of solving them, whereas if you discover them only once you're months-deep into software development, you may discover that you've painted yourself into a corner. Set yourself up for success on paper while you still have the flexibility to do so.

Some items in your breakdown will be easier to do than others, and some will be more fun. Use your judgement when you figure out the order in which you should do things. In general, it's a good idea to tackle the risky things first. If something's important enough to be in your MVP, and there's a risk that it could go long, or might not work at all, it's often smart to get it out of the way early. Doing this gives you time to iterate on a risky item while you work on other things, or in the worst-case scenario, if you discover that a thing you'd counted on just can't be done, you're still early enough in your project that you may be able to fall back to another plan. Don't leave high-risk, high-priority items to the end—you'll be in trouble if something goes wrong.

Look for dependencies between items. If something can't be done before another thing is done—a character, for example, that can't be animated until the character is built and rigged, then make sure those dependencies factor into your plan. It does you no good to plan to do a thing in a certain order and then discover that you can't because something it depends on isn't ready.

A word about project management—there's a forest of literature out there about planning and tracking projects, and discussing them in depth falls outside the scope of this book. Broadly, these fall into two major schools of thought: waterfall and agile. Waterfall project management methods lay out tasks in a rigid order that assumes that, once one task is done, the next can begin. This works well when the things you're doing are well-defined and don't entail much risk, such as painting a house, but VR design and development rarely works out like this. You simply may not know whether a feature is done until you see it running alongside other systems, and you may have to loop back at that point and change something or rework it entirely. Agile methods, such as Scrum, take this reality into account, and are intended for design and development projects where things are going to need to be revisited as the project evolves and reveals new information. In general, agile methods work much better for software development than waterfall plans.

Depending on the scope of the project, you may not need to apply a formal project management method, but even if you're planning loosely, there should still be a plan, and you should make sure the plan accommodates the reality that you're going to have to loop back and iterate on features and design, that some items are going to depend on others, and that some things are going to take longer than you thought.

Test your designs as early as you can. VR especially is a very new medium, and people respond to it in very different ways. Test with as diverse a range of subjects as you can, as early as you can, so you can spot things you need to change while they're still relatively easy to change.

Remember as well that VR developers make terrible test subjects. We use VR far, far more than other users and tend to be much more comfortable with VR interfaces and much less prone to simulator sickness than typical users. Test VR with users who are new to it as well as with users who are comfortable with the medium.

Test with as diverse a population as you can. VR embodies the user in ways that previous media don't, and this can matter to your users. Hands that look fine to you may feel alien to a user whose hands look different. Make sure your test population isn't limited to just people like you.

Look for opportunities to test as early in the process as you can make them happen. Even long before you've reached your MVP, test elements such as locomotion systems that may need design iteration. Put users into a gray-box environment and have them navigate through it, and watch what they do and where they get stuck. The more testing you do, the better your project will get, and the earlier you test, the easier it is to act on what you've learned.

Lots of people assume that finished products somehow spring fully-formed from the minds of genius designers or developers. It doesn't work this way. Anything worth making takes iteration to get there.

Prepare yourself now for the reality that the first iteration of your design isn't going to be everything you wanted it to be, and that's the point. The purpose of the first draft of anything is to show you what's really important about the thing you're building and how it needs to fit together. Plan for this. Design is a process, and time and iteration, more than anything else, are the key elements of that process.

This is why we so strongly advise designing on paper first and testing prototypes in software as early as you can. Each time you give yourself something tangible to respond to, you're going to discover something about it and probably discover a way to make it better.