How does VR work?

VR may seem difficult to comprehend but by the end of this book, this mountain will seem more like a manageable hill. Two areas of understanding are required to fully comprehend how VR works: development and engagement. The latter requires less explanation, so we will start with that first.

VR experiences require devices specifically for VR applications. They can be smartphones or HMDs. To engage in a VR experience, you can either open the application on your phone or open the application on your headset. With the application active, you can enjoy the experience until you are ready to exit.

Developing for VR is a bit more time-consuming and intricate. Before you begin developing a VR experience, it is always good to write out a roadmap. This can include features you want to integrate, assets you want to utilize, and even interactions you want to have. This can be a checklist or a narrative description.

When you are building experiences, recall the list of elements I mentioned in the Technological components of building an immersive experience section. Developing a good workflow is easier when you can formulate a plan with the elements you have access to. You will need audio, animation, and 3D models. What will those look like and sound like?

Using that as a guide, you can gather all of your assets from other sources, such as Blender, Photoshop, or even asset marketplaces.

When you have all your assets, you can open Unity and begin to build out each scene of the experience. You may combine C# code with music, 3D models, character animation, and object interactions all into one experience.

After you have a prototype, you can integrate testing into the development process. Since we experience on devices, it is best to test on those devices early and often. When you finish, you build and export the VR experience to your intended device (headset or smartphone) and enjoy the experience. We will expand on how to develop VR experiences later, but this should help establish a standard process as you gain more experience.

How do we experience VR?

The most popular way to experience VR is with a headset or HMD. The headset will render a visual in each eye separately. This is called stereoscopic rendering. This is different from seeing through an open viewport, which is a full-screen view, similar to what we see with most smartphone experiences.

Once you have the headset on, you can load an experience onto your headset, and you can enjoy the experience.

Some experiences will require you to clear out a space to move around, and others will work perfectly in a seated position. All this depends on what you intend to do in VR. The only thing hindering you is your desires.

Hardware, software, and platforms that support VR development and engagement

Even though we will be focusing on developing VR experiences in Unity using an Oculus Quest as the test device, it is important to be aware and familiar with other tools since all the stuff we will be learning can easily be applied to other tools that are mentioned. More importantly, as time goes on, higher-end devices will become more accessible, and creators will always explore various tools to meet the scope of what the client needs:

• Headsets: VR headsets allow the user to enter an immersive digital world by simply putting on a headset. There are a variety of headsets available on the market at a range of price points. Let’s discuss some of the popular headsets and what features make them unique for VR:
  • Desktop-based headsets require a computer to use. Some VR experiences require high-end graphics rendering and computer processing to play, and these headsets will allow you to plug into a USB port and engage. The limiting factor is that you must have a hardline connection to enjoy VR:
    • PlayStation VR is a console-based VR system developed by Sony to work with the Sony PlayStation (PS). It allows you to experience high-end console games through VR. To do so, you also need a PS4 or PS5, a PS camera, and a VR system (headset and handheld controllers). This is great for users who don’t have an expensive computer but have a PS4 or PS5 game console. More importantly, by utilizing PS Network, you can play games with friends fairly easily in VR.
    • HTC Vive is a VR headset developed by HTC. It uses two sensors in the corners of the room to map your surroundings and track your movements while the headset is also connected to a computer. It uses two hand controllers to track finger and hand movements. It provides a fully immersive experience but may require a computer with good processing power to run smoothly. Unlike PSVR, which uses the PS’s library of games, HTC Vive uses the SteamVR library, an online-based VR gaming software.
    • Oculus Rift S is a connected VR headset by Oculus (Meta). It provides a fully immersive experience with high-end graphics comparable to a console gaming experience. It has two touch controllers that provide hand and finger tracking. You do not need additional sensors to use the headset. The price point and the minimal setup hardware make this a solid headset for the price.
    • HP Reverb G2 is a connected VR headset by Hewlett Packard. Much like the Oculus Rift S, it provides a fully immersive experience with a headset and hand controllers that support touch input and hand tracking. This headset is capable of playing on lower-end computers, but to get the best experience, a computer with a good graphics card and processor is recommended. It has one of the best displays of any headset with a 2K (2,160 pixel x 2,160 pixel) per eye resolution. It also comes with off-ear speakers that improve the quality of the audio out of the box. For the price, this is one of the more premium VR options on the market.
    • Valve Index is a PC-powered VR headset by Valve. It is a fully immersive VR headset that requires an external base station for full tracking capabilities. It comes with two sensors you must mount to a wall for body tracking, and two controllers for touch input and hand tracking. It is one of the most expensive headsets on the market, but that comes with some perks. First, it is one of the most comfortable headsets to wear for long periods. Visually, it also provides a good pixel density at a high frame rate, allowing you to get the crispest picture possible without the lag.
  • Mobile VR is another category of VR experiences that often trade accessibility for quality. It follows the basic concept of VR as you use your phone to play the experiences with or without a case you can wear on your head. It does not provide hand or body tracking, but you can use a Bluetooth controller to interact with elements in the experience. Typically, with mobile VR, the headset case is device agnostic as it serves solely as a phone holder. In this case, you usually choose one based on the comfort and the price rather than the software features. The quality of the VR experience is usually determined based on your mobile device:
    • Android phones are the most popular phones for mobile VR. They have a lot of support behind them from Google, Samsung, and the open developer community on the internet. Because Android is open source, there are very few barriers hindering developers from developing VR experiences. This provides some of the most innovative games and applications on the market. Interestingly enough, almost all VR headsets are Android-based, so the difference in experiences is normally due to the lack of sensors and processing power of the Android smartphone.
    • iPhone’s VR is simple, but it can be limited. If you want to have a VR experience on iPhones, you simply choose the VR app you downloaded from the App Store, place the iPhone into a VR head-mounted case, and enjoy. Again, much like Android phone VR experiences, you don’t have hand or body tracking, and the processing power is not as high. You can use a Bluetooth remote to interact with objects, but you can expect these experiences to be more passive, such as watching 360 videos. Unlike Android, iPhone does not have as much developer support, and the experiences available are restricted to ones you can find on the App Store. iOS is a good introduction to VR for early adopters, but it only skims the surface of possibilities for intermediate to advanced users.
  • Standalone headsets do not require a computer connection to run, although some games may require a connected device. Standalone headsets are unique because they provide the portability of a mobile device and the quality of a desktop headset. They are the most popular category of headsets because they have the best features for the price. They require an internet connection and most of the experiences that are played are stored on the device. It is likely that if you meet someone who has a VR device, it will be a standalone headset:
    • Oculus Quest 2 is a standalone headset by Oculus (Meta). It is the most popular headset on the market at the time of writing. It does not require a PC but is capable of supporting connected experiences as a hybrid device. It has a high refresh rate for a standalone headset and supports hand tracking and gestures with two hand controllers. It is the most versatile headset because it can play a variety of experiences from a variety of platforms.
    • Lenovo Mirage VR S3 is a standalone headset developed by Lenovo. Like other standalone headsets, it does not require a PC to use. It only comes with one remote control, so you will have limited hand gestures and interactions with this headset experience. A unique feature of this device is that its memory can be expanded with a microSD card.
• Platforms: These are integral to the VR experience because they serve as hubs for navigating the virtual world and accessing games and applications. Without a VR platform, your headset is just a paperweight with futuristic sensors. Platforms provide you with access to different games and marketplaces, depending on the supported headset, although there is overlap in the content that each platform offers:
  • SteamVR is a customizable VR platform by Valve that allows you to launch apps and games and interact with objects in VR. It supports most VR headsets on the market. SteamVR is a one-stop shop for everything you would want to do with your headset.
  • Oculus VR is the default platform for Oculus headsets. Although this is specific to Oculus headsets, it is good to mention because most people who have a VR headset have an Oculus. Oculus is similar to SteamVR in terms of its customizability and access to popular marketplace applications. Oculus VR supports a hybrid mode that allows you to choose if you want to render things with your computer by plugging it in or with your headset as a standalone device.
  • Windows Mixed Reality is a VR platform for Windows operating systems. It provides VR experiences for compatible devices such as the HP Reverb. In Windows Mixed Reality, you can choose from a variety of apps and games available on your computer.
  • Open Source Virtual Reality (OSVR) is an open source VR platform that supports a variety of devices on Windows, Android, OSX, and Linux. It is more of a developer kit, but the focus of OSVR is to improve the standard for smooth VR experiences.
• Accessories: They can improve your day-to-day experience in VR. These options are normally hardware-focused and not software-focused, but since this book is about enhancing VR experiences, I figured the physical comfort should at least be covered. This will not be a comprehensive list or endorsement of particular products. My goal is to make you aware of what is available so that you can make more informed decisions about the experiences you want to have:
  • Haptic feedback gloves and suits provide extended sensory input through direct feedback. With gloves and suits, you can feel the difference between a hard, soft, and smooth virtual surface. It fills the gap of touch sensation that is left void with virtual objects. When we talk about making experiences more immersive, improving touch sensation with additional hardware can do that.
  • Sensory masks allow users in VR to smell in VR. These masks simulate natural, relaxing, and realistic smells and add them to your VR experiences. They work as attachments to your headset that you can use in your development process to provide sensations such as water mist, heat, wind, and vibration for your different VR scenarios.
  • Omnidirectional treadmills provide a unique opportunity to improve mobility within VR. One of the limiting factors of VR is the physical space available for you to navigate in VR. Even in the most robust VR experiences, because of space limitations, we are stuck with using joysticks to explore virtual worlds. You may be able to duck under objects or jump over fences, but locomotion is often hindered for logistical and safety reasons. Omnidirectional treadmills do that by giving you the means to walk in any direction for an unlimited time virtually while also having a small physical footprint. If you want to have a more realistic simulation of movement in VR, look into one of these.
  • Headphones that connect directly to the headset provide a better audio experience. It makes you feel more immersed because the audio quality is not obstructed by sounds from your environment. If you want to improve the audio quality, you can get some headphones.
  • Extended batteries. If you have a standalone headset such as an Oculus Quest, one of the biggest selling points is also its limiting factor. Yes, I am talking about battery life. Standalone headsets are great because you are not tethered to a computer. Although that can be liberating, it does mean your experiences have a time limit. With an extended battery, you can extend that time limit without compromising comfort.
• Game and rendering engines: These are the premier software for developing experiences for VR. Their ability to render elements in real time allows for interaction-based experiences that react and respond to user engagement:
  • Unity is one of the most popular engines for AR, VR, and mobile gaming. It is very versatile and can be used to create applications for a variety of industries. We will be focusing on VR development in Unity for this book primarily.
  • Unreal Engine is another popular game engine known for rendering high-fidelity experiences in real time. It supports a wide range of VR platforms and provides avenues for developers to use Python and C++ in their development processes.
  • Godot is an open source game engine that uses modules to expand the features of the platform. It has a huge library of tools to build games and other experiences for a range of platforms. The Godot gaming community is growing, and more support is available as a result.
  • CryEngine is a reliable second-tier engine that’s used for games and provides a great toolset to build VR experiences. It offers great visuals and a sandbox of options that are relevant to VR development. It does not have as much support as Unity or Unreal, but it’s open source and can fulfill all the needs you may have to get started.
  • AppGameKit is a great game engine for beginners and hobbyists that provides developers with the tools needed to learn game development. You can expand its features by installing the VR extension. Here, you can develop VR games with a library of common interactions you will find in most games. Compared to other engines where you can make VR applications, AppGameKit focuses primarily on game development.
  • Blender can support VR development. Blender is great for content creation, and the latest version allows you to connect to an Oculus Quest. With Blender, you can view your model creation in a virtual space and traverse the elements. You can also model objects in VR, but there are not as many tools to build interactive games and applications as there are for most engines.
  • Amazon Sumerian is a lesser-known software developed by Amazon that allows you to render VR experiences with Amazon Web Services (AWS). It is a popular platform for early developers because it does not require programming or 3D skills to build experiences. More importantly, these experiences can work with a variety of devices, from smartphones to high-end headsets.
  • Sketchup Studio is a 3D modeling tool that allows you to create models specific to construction and architecture. It has a VR feature where you can create those models in VR. If you are familiar with CAD and construction-specific tools, Sketchup may be a good avenue for exploring VR.
• Software development kits (SDKs): These are a set of tools, sample scenes, and code examples that allow developers to create games and applications for specific platforms and hardware. In most cases, you download the SDK from a reputable website and utilize it within a compatible software, such as Unity:
  • Steam VR Plugin allows you to develop VR experiences in a game engine such as Unity and Unreal and distribute those experiences on the Steam VR marketplace – one of the largest marketplaces in the world. This SDK also allows you to add Steam-specific features to your experiences, such as achievements and matchmaking.
  • Open VR SDK is a developer kit by Valve that is used to make VR experiences for a variety of hardware. It builds off of the foundation that SteamVR provides to give developers more features for their hardware-specific interactions.
  • Oculus Integration SDK allows you to develop VR experiences for Oculus in the Unity game engine. It has all the tools needed to go from concept to finished project and make it available for Oculus users.
  • Virtual Reality Toolkit is a collection of scripts and interactions that work for a variety of VR experiences. At the time of writing, it is only compatible with the Unity game engine.
  • Dev Kit allows you to develop experiences for PSVR. Compared to other SDKs, where you can download them and start developing very quickly, with PSVR, you must become a registered developer first.
  • Cardboard SDK (formerly Google VR) is a developer kit that allows you to make VR experiences for Android and iOS. It supports motion tracking, stereoscopic rendering, and user interaction for mobile devices out of the box to let you create engaging VR experiences on a smartphone.
  • PICO Unity XR SDK is used to develop experiences for the Lenovo Mirage. It supports games and applications built on the Android platform.
• Content creation: These tools allow creators to create 3D objects and animated assets for their VR experiences. Game engines are perfect for rendering elements in real time, but those objects need to be created and optimized for rendering. That includes having a low poly count, textures, materials, and even the right real-world scale. You can do all of that in content creation software:
  • Maya is an industry standard for game development and animation. Most 3D characters and animated assets for games made in Unity and Unreal are made with Maya. If you need a tool that does everything you want for VR, Maya is a safe choice that has applications in other industries.
  • Cinema 4D is a 3D animation powerhouse. It is used by a variety of entertainment industries due to its ability to provide high-quality renders and simulations. Although the simulations and renders do not translate to Unity, Cinema 4D has superb modeling, animation, and polygon reduction tools to make your VR scene more performant.
  • Blender is one of the most versatile tools a content creator can have. It can support 3D modeling, 3D animation, 2D animation, texturing, 3D mesh optimization, and 3D world-building. Pretty much anything you can think of, Blender can do. You can create your content here and export it to an engine to render it.
  • ZBrush is a 3D sculpting program that gives you the ability to create 3D models and objects as if you are sculpting with clay. It has a variety of tools to streamline your process and give you the highest quality creations.
  • Photoshop is a dependable 2D content creation tool. It is great for prototyping user interfaces, designing interactive UI elements, creating decals, and 2D animation. Game engines are not the best at creating professional-looking UI themes, but Photoshop can be used to address that.
  • Oculus Medium is a tool that allows you to create content in VR using the Oculus VR headset. You can create 3D models using sculpting and traditional modeling techniques. Medium has a robust array of painting tools to texture your models to your liking.

We know what VR is, we are familiar with all the hardware and software (Figure 1.3), and we know what applications can be made. With this foundation, we can begin to explore VR development. The next section will cover ways to approach VR development. Instead of diving deep into Unity and building prototypes, it is wise to expand on the workflow I mentioned earlier in this chapter so that you can approach development with confidence and efficiency:

Figure 1.3 – Core elements of VR development