Using Unity3D to create great holographic apps

Microsoft says the best way to build HoloLens apps is to use Unity3D. Unity3D, or Unity as some people call it, is a cross-platform game engine. This application was first used to create OS X applications in 2005, but has grown to support more than 15 platforms at the moment. Since one of those platforms is Direct3D on Windows, this was an obvious choice for the HoloLens team as the way to build 3D worlds.

Next to being a game engine, it also is a development environment for this engine, making it a great tool with which to create HoloLens apps. Unity3D natively supports HoloLens as a platform, so you do not need to install plugins.

Unity is not free. However, the Unity Personal license is all you need if you want to develop HoloLens apps and is free if you fulfill the requirements. I suggest that you go to their website to look up the exact license agreement, but basically it says that if the company using Unity has less than $100,000 in annual gross revenue, you are free to use the tool.

If you are unfamiliar with Unity I can reassure you that there is nothing magical about it and the tool is not that hard to learn. In this and in the following chapters, I will show you all you need to know to work with it.

When you install Unity from their website, you also need to install the Unity plugin for Visual Studio. There is a very good reason for this. Unity allows you to deploy to all their platforms from within the Unity editor itself, with one notable exception--the HoloLens. The final compile and build and the deployment still need to be done through Visual Studio.

Next to the Build and Deploy scenarios, we will also need Visual Studio to write scripts. Although you can use MonoDevelop, a free independent development tool based on the Mono framework, I still recommend using Visual Studio. You need Visual Studio anyway, so why not take advantage of the power of this IDE?

One word of warning--you should make sure that you have enough memory in your machine to have two instances of Visual Studio running at the same time. You will use one instance to edit your scripts, and the other to do the building and deploying of the UWP application.

Scripts in Unity are pieces of code that enhance or change the behavior of objects or add new behavior to them. Unity itself is more a 3D design environment and leaves the writing of the scripts to other tools. So you will find yourself switching between the Unity editor and the Visual Studio editor quite a lot.

Our first HoloApp in Unity3D

When you have all the tools installed, it is time to start up Unity. Immediately, you will notice a big difference between Unity projects and Visual Studio projects. The latter has solution files and project files that determine what goes together to create a project. Unity, however, uses a folder to determine what a project is.

When you start up Unity, you will be greeted by a screen that gives you the option to open a previous project or to create a new one. When you select New Project, you will get this:

The Unity new project screen

Unity wants you to give the project a name, a folder where the files will be stored, and an organization to which the license is assigned.

We can choose to have a 3D or 2D application, which we will leave to the default 3D option. The Add Asset Package gives you the option to add additional packages, but we will not use that here for now. The Analytics option is quite handy to debug your app.

Press Create project to have Unity create the necessary folders and files for you.

The default Unity screen

The screen we see is what the game looks like. For now, this is an empty screen with some sort of ground and sky above it. In the left-most panel, you see the objects currently available. Unity maintains a hierarchy of objects, and this is where you can see them. In the lower part of the screen, you see the Assets we have in our project. Currently, we do not have any assets, so this is empty, but we will later add some.

The two items we do have are a camera and a light source. The light is important--without light there is nothing to see in our virtual world. The camera is usually the way we look at the scene, but in the HoloLens this works slightly different--we need two cameras. One for each eye, remember? Fortunately, the tool takes care of this for us, so we do not need to worry about it.

Fixing the camera

We need to fix the camera a bit though. Remember when we started up the emulator and everything was black? The reason for this is that anything that is rendered black in the device is going to be transparent. The reason for this is quite straightforward--the device adds light points to the real world and black would mean it removes light points from the real world. This is unfortunately physically impossible. So anything that is black will not get any light point and is, therefore, transparent. We need to change this here as well. Select the Main Camera in the left side and see how the properties appear on the right-hand side in the inspector . This is where we can make changes to our assets, in this case, our camera.

Changing the position

First, we need to change the position. The default camera is placed at the coordinates {0, 1, -10}. In HoloLens, however, we are the camera. We are at the center of what is going on, so we need to change these to camera coordinates to {0,0,0}. You enter these values manually or you can click the cogwheel in the top-right corner and click on Reset. This will reset all values to the default which in our case is {0,0,0}.

Changing the background color

Another thing we need to change is the color of the virtual world. We do this by changing Clear flags. This is the color that is being used when no pixels need to be drawn for our scene. In a game, it would be nice to have a default background such as the one we see now, but in HoloLens, we want the default to be transparent and thus black.

Change Clear Flags from Skybox to Solid color and change the background color underneath this to black (RGB--0,0,0).

Every now and then, Unity might show Skybox again when we select other objects, but this is something we can ignore.

The camera itself has a "MainCamera" tag. This means that the SDK will take this camera and use it as the point of view. You can have multiple cameras in a scene, but only one camera can be the main camera. By default, this tag is already assigned, so we do not need to change anything here.

The properties of the main camera

Adding holographic objects

Our scene looks quite dull. We have nothing in our world besides a black background. So let's add something a bit more interesting to our world.

We will add a sphere in our world. To do this, take the following steps:

1. Make sure that the camera is not selected anymore--click anywhere in the hierarchy panel and verify that the inspector window is empty.
2. Click on the Create button at the top of the hierarchy window.
3. Select Sphere. You will note that the sphere is added to the hierarchy, although it is not visible. The reason for this is that, in order to make the whole system performant, the insides of our 3D objects are not rendered. Since the sphere is placed at location {0,0,0}, the camera is inside the sphere and we cannot see it.
4. Move the sphere by selecting it in the hierarchy panel and change its location in the inspector window. Move it about three meters away from the user and move it half a meter below the head of the user. Since the HoloLens is located at {0,0,0}, this means we have to place the sphere at {0,-0.5, 3}.
5. Scale the sphere (a sphere with a diameter of 1 meter is quite large) and make it 25 centimeters.

You will end up with something like this:

Our scene with the newly added sphere

Create a sample script

Like I have said before, scripting, an important part of Unity projects, is done in C#. I want to add an empty script here, just to show you how its done and what happens if you do so.

In the pane at the bottom, named Assets, right click and create a new folder named Scripts. Although this is not required, it is always a good idea to organize your project in logical folders. This way you can always find your components when you need them. Double-click on that folder to open it and see its contents. It should be empty. Right-click again and select Create -> C# Script. Name it PointerHelper. Unity will add the .cs extension to the file, so you should not do that.

Once you have added this empty script, take a look at the inspector pane on the right side. You will see it is not empty at all--the script contains a class named PointerHelper, derived from MonoBehaviour with two methods in it--Start and Update.

We will dive into this later on, but for now I will tell you that this is common with most scripts. The Start() method is called when the script is first loaded, and the Update() method is called each frame. We will discuss frames later on.

When you have created this script, we need to attach it to an object or an asset in our project. This particular script will later on allow us to create an object that shows the user where they are looking. Therefore, it makes sense to attach it to the camera. To do this, drag the script upward to the hierarchy pane and hover right above Main Camera. If you do this right, you will see that the Camera is selected in a light blue color. If it is a darker shade of blue, you will also see a line underneath the Camera object, stating that this script will be a child item of Camera instead of being a part of it. It needs to be a part of it, not a child.

You can always check whether you attached it correctly by selecting the camera and looking at the inspector pane. At the bottom of that pane, you will see the PointerHelper (Script) component being added to the cameras properties.

We will come back to this script later on.

Building the project

Of course, before building your project, it is a good idea to save it. If you click on Save or use the Ctrl+S key combination, Unity will ask you to save your scene. A scene is a collection of objects laid out in a 3D world. In our case, this is the collection containing the light source, the camera, and our sphere. Just give your scene a name, I used main and placed that in the subfolder Scenes under Assets (which is the default).

When you have saved your project, we can start to build the code. However, before we do that, we have to tell Unity that we are working on a Windows Holographic application. To do that, do the following:

1. Go the File | Build Settings or press Ctrl + Shift + B.
2. In the dialog you see now, press the Add open scenes button. This makes sure that our current scene is part of the package.
3. In the platform selection box, we can choose for which platform we would like to build our app. As you can see, there is quite a large choice of platforms, but we will choose Windows Store and then click on the Switch Platform button. You can tell we now have this platform as the default by the appearance of the Unity logo behind the Windows Store option.
4. On the right-hand side, we can set specific Windows Store options.
5. SDK: Windows 10.
6. UWP Build Type: D3D (meaning DirectX 3D).
7. Build and Run does not matter here; we can leave that.
8. Check Unity C# Projects.
9. Check Development Build.

Unity Build Settings for HoloLens

10. Now, before we press Build, we need to do one more thing--click on the Player Settings... button. This is where you set the properties for the Unity Player, thus the application that loads the scenes and performs the animations, interactions, and so on. We, however, do not use the Unity Player but use our own. Still, we need to set one very important property:
11. Click on Player Settings, and see that the Inspector changes.
12. Click on Other Settings.
13. Check Virtual Reality Supported and verify that the Virtual Reality SDK contains Windows Holographic. This makes sure that we have two cameras when we run the app.
14. Now, we can press Build. Unity will ask us for a place to put the C# and Visual Studio files. I usually create a folder in the current folder with the name App. Press Select Folder and Unity will build the code for us.

The first time you do this, you will notice this takes some time. It has to generate quite a lot of files, and it needs to package up all our assets. The next time only the changes need to be processed, so it will be much quicker.

When the building is done, Unity will open an Explorer window where our project is. You will see the newly created App folder. If you open that, you will find the .sln file, in my case HelloHoloOnUnity.sln. This is a Visual Studio solution file we can open in Visual Studio. Let's do this!

Continuing in Visual Studio after Unity

When you open the solution in Visual Studio, you will see three different projects. If you only see one, you have opened the .sln file in the root folder. Trust me, this will happen quite often. The names are the same and the folders look quite similar. However, the one with the three projects is the one we can use to build and deploy; so, open that one. Again, this is found in the App folder you created and pointed at in the last step of the Build in Unity.

The structure of a Unity HoloLens UWP app solution

As I said before, the solution contains three different projects:

• Assembly-CSharp (Universal Windows)
• Assembly-CSharp-firstpass (Universal Windows)
• HelloHoloOnUnity (Universal Windows)

The last project is our final project, the one we will deploy to the device. The first is a placeholder containing our scripts. If you look at it, you will see in the Scripts folder our PointerHelper.cs file:

The solution structure for a Unity project

If you make changes to the scripts here, they will also be visible in Unity. So this is the place to write the specific code for your application.

The second project is a helper-like project that ties the projects together. We do not have to worry about that now.

The actual project itself, in our case, HelloHoloOnUnity, is nothing more than a loader. It loads the Unity3D player and launches our scenes. The player is deployed to the device and that takes care of running the application. There is not much we can do in this code base now.

Build configurations with Unity

Take a look at the configurations in the Build or Deploy drop-down. We have three configurations now, but although some names might seem familiar, they are not quite what you are used to. We have the following options:

Configuration name	Optimizations	Profiler enabled	Usage
Debug	No	Yes	Debug your scripts
Master	Yes	No	Deploy to the Store
Release	Yes	Yes	Test the application and test performance.

Also, note that the processor architecture is ARM, by default. This is a left-over from the Phone tooling on which the HoloLens SDK is based. However, the HoloLens uses a x86 processor, so you need to set this to x86 as well. I suggest that you use Debug/X86 for now. We also have the now familiar options for the deployment. You can use the emulator, the device through Wi-Fi (Remote Machine), or the device through USB (Device)--which one you choose is up to you.

Building and running your HoloLens Unity app

Build the project, and run the debugger. The build will take some time at first. It needs to get all the packages from the NuGet server. It will build all code; then deploy the whole app and all dependencies to the device or emulator; it then starts the app and attaches the debugger. This whole process can take a couple of minutes.

However, after this is done the first time, the next build and deployments will be much faster.

When it starts up, you will be greeted in the virtual world with a nice Made With Unity logo. After a little while, this goes away. In front of you, about three meters away and about half a meter below your eyes, you will see a white sphere floating in mid-air.

Walk around it! Look at it! Crawl underneath it! Try and grab it!

That last part doesn't work. The default near clipping plane in Unity is 30 centimetres, meaning that everything that is closer than that will not be rendered. Microsoft says it is best to set the clipping plane to 85 centimetre but, to be honest, 30 works just fine, and that is what they use in their own applications anyway. So I tend to keep it at that.

Of course, you can never grab the object; it is virtual. But to be honest -- you were tempted, right?