FruitWarrior MR HoloLens Workshop


Let’s do something cool with Microsoft HoloLens, the world’s first self-contained, untethered head-mounted holographic computing device for Mixed Reality!
Mixed reality blends 3D holographic content into your physical world, giving holograms real-world context and scale, allowing you to interact with both digital content and the world around you.
This guide will walk through creating Fruit Warrior MR, which definitely has no relation to the game Fruit Ninja whatsoever. This project takes concepts from Holographic Academy, mixed with assets from the HoloToolkit to teach concepts such as HoloLens Gestures, Voice Commands, Spatial Sound, and Spatial Mapping.

Check out the finished game:

Github link to the finished project here.

Before Starting

Before starting, make sure you have all the tools installed from the Installation Checklist here. This includes Visual Studio 2015 Update 3, the HoloLens emulator if you don’t have your own device, and Unity 5.5.
In addition, please download the latest HoloToolkit Unity package from their releases. At the time of writing this guide, that’s HoloToolkit-Unity-v1.5.5.0.unitypackage.

Setting Up

  1. Make a new 3D project.

    New 3D Unity Project

    New 3D Unity Project

  2. Make a folder called “Our Assets” to help organize what we make vs. what the HoloToolkit makes.

    ourassets

    “OurAssets” Folder

  3. Import HoloToolkit. Import all except HoloToolkit-Examples folder. (Or import all and delete HoloToolkit-Examples afterwards).

    Importing HoloTolkit

    Importing HoloTolkit

  4. Make folder in Our Assets named Scenes. Save Scene.

    Scenes

    Scenes

  5. Delete default Main Camera. Drag and drop HoloLens Camera prefab into your Hierarchy Assets: HoloToolkit -> Input -> Prefab -> HoloLensCamera

    HoloToolkit Camera

    HoloToolkit Camera

  6. Configure Settings. These settings will set the frame rate performance to fastest, set the build target to Windows 10, set the build type to Direct3D, and enables HoloLens VR in Unity.
    1. Menu bar: HoloToolKit -> Configure -> Apply HoloLens Project Settings

      Settings

      Settings

You should be testing your application either on the emulator or on a HoloLens device throughout this guide.
First you have to build a Visual Studio solution, which you can do from the HoloToolkit Menu Bar -> Build Window. Then check out the Build and Deploy section here.

Build

Build

Import Holograms

  1. You can feel free to use your own models. If you want to use my fruit objects (crafted by the talented Kat Park), download the finished version of this project. Scroll up to the top of this page for the link to the finished Github repo.
  2. In this order: Drag the Holograms folder (FruitWarriorMR-master/Assets/Holograms) from the completed Github repo into the “OurAssets” folder in your Unity project. Then, drag the Prefabs folder (FruitWarriorMR-master/Assets/Prefabs) from the completed Github repo into the “OurAssets” folder in your Unity project. This order matters, as the Prefabs are relying on information about the models in the Holograms folder. They won’t render properly without it.

    Hologams Folder

    Holograms Folder

Gaze

The HoloToolkit comes with a bunch of nifty things to help us with Gaze input. The InputManager prefab is a system that manages gaze and various input sources currently supported by HoloLens, such as hands and gestures. Importantly, it also includes a fake input source that allows you to simulate hand input when in the editor. By default, this can be done by holding Shift (left hand) or Space (right hand), moving the mouse to move the hand and use the left mouse button to tap.
The cursor will follow the user’s gaze, with an additional HandDetectedFeedback asset to give feedback to the user when their hand is detected in the ready state.

  1. In the Assets: HoloToolkit -> Input -> Prefab folder, drag the InputManager prefab into your Hierarchy to input your Gaze Manager.
  2. In the Assets: HoloToolkit ->Input -> Prefab -> Cursor folder, drag the CursorWithFeedback prefab into your Hierarchy to input your cursor.

    Drag and drop the InputManager

    Drag and drop the InputManager

For more about Gaze, take the time to look at the scripts, specifically GazeManager.cs and InputManager.cs. Additionally, go through the Holographic Academy workshop about Gaze.

Spatial Mapping

The HoloToolkit has prefabs that let you use the HoloLens created mesh of your physical world, in your Unity project.

  1. Add the Spatial Perception capability in your app, in Unity under Edit -> Project Settings -> Player -> Settings for Windows Store -> Publishing Settings -> Capabilities.

    Spatial Mapping Setting

    Spatial Mapping Setting

  2. Drag the SpatialMapping prefab in Assets: HoloToolkit -> SpatialMapping -> Prefabs into your Hierarchy.
  3. Click on the SpatialMapping object in your Hierarchy. In the Inspector on the right side, uncheck Draw Visual Meshes under Spatial Mapping Manager.
  4. In the Inspector, change Triangles per Cubic Meter to 1500 and Time Between Updates to 2. This will give us a denser and more frequently updated world mesh.

Spawn Fruit

Now that our world is set up, it’s time to make some falling fruit! Feel free to use the finished scripts in the Github repo, or follow along.

  1. Create a new folder in “OurAssets” called “Scripts”. In the “Scripts” folder, create a new C# script called GameController.

    GameController script

    GameController script

  2. Add code to spawn waves of fruit.
  3. using System.Collections;
    using System.Collections.Generic;
    using UnityEngine;
    
    public class GameController : MonoBehaviour {
    
        public float leftWidth;
        public float rightWidth;
        public float height;
        public float awayDepth;
    
        //The fruit prefabs we'll drop- set in the Unity editor
        public GameObject[] fruits;
        
        //Score/Timer
        private int score;
        private float timer; 
    
        //List of spawned fruits in play
        List<GameObject> spawnedFruits;
    
        // Use this for initialization
        void Start () {
            //Set score and timer to start values
            score = 0;
            timer = 120f;
    
            spawnedFruits = new List<GameObject>();
    
            //Values chosen to spawn near view frame. Feel free to play around with them.
            var cam = Camera.main;
            leftWidth = cam.transform.position.x - 1;
            rightWidth = cam.transform.position.x + 1;
            height = cam.transform.position.y + 1.5f;
            awayDepth = cam.transform.position.z + 2;
    
            //Start spawning fruits
            StartCoroutine(SpawnWaves());
        }
    
        //Spawn fruits
        IEnumerator SpawnWaves()
        {
            var cam = Camera.main;
            while (true)
            {
                //Spawn the fruit within the player's view
                Vector3 spawnPosition = new Vector3(Random.Range(leftWidth, rightWidth), height, Random.Range(awayDepth, awayDepth + 2));
    
                //Instatiate random fruit
                spawnedFruits.Add(Instantiate(fruits[Random.Range(0, 6)], spawnPosition, Random.rotation));
    
                //Wait before spawning another
                yield return new WaitForSeconds(3f);
            }
        }
    }
    
  4. Create a new empty game object named GameController and give it the GameController tag.

    GameController Object

    GameController Object

  5. Drag the GameController script onto the GameController Object. Set the values as per the picture below.

    Set values in GameController Script

    Set values in GameController Script

  6. If you play your game on the emulator or device, fruit will spawn and bounce off the floor. But that fruit is falling really fast! Let’s change the gravity so the fruit falls slower. In Menu:Edit->Project Settings->Physics, change Gravity Y from -9.81 to -.81.
    Defying Gravity

    Defying Gravity


  7. **Awesome! But now fruit will fall forever if they don’t bounce off the floor (accidents happen). Let’s put in a “deadzone” to delete fruit that’s falling forever. **

  8. Create a new 3D Plane named DeadZone, and give it the “Finish” tag. Change its Position to X: 0 Y: -10 Z: 0 and scale to X: 30 Y: 1 Z: 30 to make it 30×30 meters and 10 meters below your head. Then uncheck the Mesh Renderer to make it invisible.
    Create 3D Plane with "Finish" tag

    Create 3D Plane with “Finish” tag


    Change the transform values and turn off mesh renderer

    Change the transform values and turn off mesh renderer

Game Logic

We have most of the pieces, now it’s time to put them together!

  1. In our GameController script, we’re going to add a couple of new things. We’re going to add some logic to have a timer count down until the end of the level, a function that will add to our score when we get a fruit, and a function that will clear the field if we tap a bomb.
  2.     //Adds Score when tapping fruit
        public void AddScore()
        {
            score += 100;
        }
    
        //This function is called every fixed framerate frame
        void FixedUpdate()
        {
            //Update Timer and Screen Text
            timer -= Time.deltaTime;
        }
    
        //Called by bomb to clear field 
        public void BombClear()
        {
            while (spawnedFruits.Count > 0)
            {
                Destroy(spawnedFruits[0]);
                spawnedFruits.RemoveAt(0);
            }
    
            //Take away time and score
            timer -= 30;
            score -= 500;
        }
    

    You might have noticed if you click on any of the fruit/bomb prefabs, there’s a “missing script” in the Inspector. We’re going to fix that now!

  3. In the OurAssets -> Scripts folder, create a new C# script named FallingObjects. (Or drag and drop the FruitFall script from the finished Github repo. We’ll be attaching the script to the bomb, which is not a fruit- hence the name change.) The script below will destroy the gameobject it’s attached to (the fruit/bomb) if it collides with the deadZone and will react if it’s tapped, either triggering the BombClear() method or adding to our score. We’re using the IInputClickHandler interface provided to us by the HoloToolkit to recognize the click/tap gesture. You can read more about that here.
  4. using HoloToolkit.Unity.InputModule;
    using System.Collections;
    using System.Collections.Generic;
    using UnityEngine;
    
    public class FallingObjects : MonoBehaviour, IInputClickHandler{
        
        //Variables
        private GameController gameController;
    
        // Use this for initialization
        void Start () {
    
            //Get GameController
            GameObject gamecontroller = GameObject.FindGameObjectWithTag("GameController");
            gameController = gamecontroller.GetComponent<GameController>();
        }
    
        /*Implements IInputClickHandler, which handles click events. Will add to score and set off bomb.*/
        public void OnInputClicked(InputEventData eventData)
        {
            //Check to see if bomb- deletes all fruits
            if (gameObject.tag.Equals("Bomb"))
            {
                //Play bomb sound
                gameController.BombClear();
            }
            else
            {
                //Add to Score
                gameController.AddScore();
    
                Destroy(gameObject);
            }
    
        }
    
        void OnCollisionEnter(Collision collision)
        {
            //Destroy before they fall forever
            if (collision.gameObject.tag == "Finish")
            {
                Destroy(gameObject);
            }
        }
    }
    
  5. Drag the FallingObjects script onto each prefab where it says “Missing Script”.

Spatial Sound

Now that our fruit is falling, we have no idea where they are! Sound will help us pinpoint where around us the fruit is falling.

  1. Make a new folder in “OurAssets named “Sounds”. Drag the files from the “Sounds” folder in the finished Github repo, into our newly created Sounds folder. You can also use your own “bomb explosion” sound and “falling” sound.
  2. In Unity select from the top menu Edit > Project Settings > Audio. In the Inspector Panel on the right side, find the Spatializer Plugin setting and select MS HRTF Spatializer.
    Spatializer plugin

    Spatializer plugin

  3. On each fruit/bomb prefab, there should be an Audio Source. Change these things for each one (or change for one and copy/paste):
    • Drag fallingsound1.wav from the “Sounds” folder onto AudioClip.
    • Check the Spatialize property.
    • Check the Play On Awake.
    • Change Spatial Blend to 3D by dragging the slider all the way to the right. The value should change from 0 to 1 when you move the slider.
    • Expand 3D Sound Settings, and enter 0.1 for Doppler Level.
    • Set Volume Rolloff to Logarithmic Rolloff.
    • Set Max Distance to 20.
  4. Click on the GameController prefab in the Hierarchy, and add an Audio Source component with the same values as the prefab, but change the AudioClip set to “bombsound.wav” and uncheck Play On Awake.
  5. Now we’re going to add some logic to our scripts to determine when we should or shouldn’t play the sounds. In the FallingObject script, add an AudioSource objectSound variable, get the sound in Start(), and pause it if we hit the ground.
    //... 
    
        //Variables
        private GameController gameController;
        private AudioSource objectSound;
    
     void Start () {
    
            //Get GameController
            GameObject gamecontroller = GameObject.FindGameObjectWithTag("GameController");
            gameController = gamecontroller.GetComponent<GameController>();
    
            //Get falling sound
            objectSound = gameObject.GetComponent<AudioSource>();
        }
    
    //... 
    
     void OnCollisionEnter(Collision collision)
        {
            //Hit the ground
            objectSound.Pause();
    
    //... et all
    

    In the GameController script, we’ll play the bomb sound when the function BombClear() is called.

    //...Variables
    
        //bomb sound
        private AudioSource bombSound;
    
    //...
    
    void Start () {
    
            //Get bomb sound
            bombSound = gameObject.GetComponent<AudioSource>();
    
    //...
    
    //Called by bomb to clear field 
        public void BombClear()
        {
            bombSound.Play();
    
    //... et all
    
    

“Scoreboard”

Let’s display our score and time left to the player! Working with UI text in Unity with the HoloLens is a little different than normal. Luckily, the HoloToolkit has some nifty prefabs we can use! Read more on HoloToolkit UI here!

  1. Drag the 3DTextPrefab in Assets:HoloToolkit->UI->Prefabs into your Hierarchy and rename it ScoreBoardText. Click on ScoreBoardText and click the drop down arrow on the Shader at the bottom of the inspector. Import the font you want to use (I already imported Segoe UI in Holograms->Materials) and drag the font texture into the shader. See the picture for more detail. Also make sure to drag the font (in our case Segoe UI) under “Font” in the Text Mesh component.
    Font Shader

    Font Shader

  2. Set the text in Text Mesh of ScoreBoardText to “Score: 0”. Make sure the Z position in the transform is set to 2 and the scale values for X, Y, and Z are all 0.009.
  3. Duplicate (ctrl+D) ScoreBoardText and rename it Timer. Change the text in Text Mesh to “Timer: 120” and set its Y Position to 0.05.
  4. Now we edit our scripts to update the text game objects to reflect our time and score. Our main addition to GameController is adding and calling UpdateTextDisplay(), which updates the text game object with our values.
  5. using System.Collections;
    using System.Collections.Generic;
    using UnityEngine;
    
    public class GameController : MonoBehaviour {
    
        public float leftWidth;
        public float rightWidth;
        public float height;
        public float awayDepth;
    
        //The fruit prefabs we'll drop- set in the Unity editor
        public GameObject[] fruits;
    
        //Update ScoreBoard/timer 
        public GameObject ScoreBoardText;
        public GameObject TimerText;
    
        //Score/Timer
        private int score;
        private float timer;
        //Reference to Text UI control where the score should be displayed
        private TextMesh scoreMesh;
        private TextMesh timerMesh;
    
        //bomb sound
        private AudioSource bombSound;
        
        //List of spawned fruits in play
        List<GameObject> spawnedFruits;
    
        // Use this for initialization
        void Start()
        {
            //Start score at 0 and display
            score = 0;
            scoreMesh = ScoreBoardText.GetComponent<TextMesh>();
            UpdateTextDisplay();
    
            timer = 120f;
            timerMesh = TimerText.GetComponent<TextMesh>();
    
            spawnedFruits = new List<GameObject>();
    
            //Get bomb sound
            bombSound = gameObject.GetComponent<AudioSource>();
    
            //Values chosen to spawn near view frame. Feel free to play around with them.
            var cam = Camera.main;
            leftWidth = cam.transform.position.x - 1;
            rightWidth = cam.transform.position.x + 1;
            height = cam.transform.position.y + 1.5f;
            awayDepth = cam.transform.position.z + 2;
            StartCoroutine(SpawnWaves());
    
        }
    
        //Adds Score when tapping fruit
        public void AddScore()
        {
            score += 100;
            UpdateTextDisplay();
        }
    
        void FixedUpdate()
        {
            //Update Timer and Screen Text
            timer -= Time.deltaTime;
            timerMesh.text = "Time: " + Mathf.Round(timer);
    
        }
    
        //Puts text on screen
        private void UpdateTextDisplay()
        {
            string displayString = "Score: " + score.ToString();
    
            if (scoreMesh != null)
            {
                scoreMesh.text = displayString;
            }
        }
    
        //Called by bomb to clear field 
        public void BombClear()
        {
            bombSound.Play();
    
            while (spawnedFruits.Count > 0)
            {
                Destroy(spawnedFruits[0]);
                spawnedFruits.RemoveAt(0);
            }
    
            //Take away time and score
            timer -= 30;
            score -= 500;
            UpdateTextDisplay();
        }
    
        //Spawn fruits
        IEnumerator SpawnWaves()
        {
            var cam = Camera.main;
            while (true)
            {
                //Spawn the fruit within the player's view
                Vector3 spawnPosition = new Vector3(Random.Range(leftWidth, rightWidth), height, Random.Range(awayDepth, awayDepth + 2));
    
                //Instatiate random fruit
                spawnedFruits.Add(Instantiate(fruits[Random.Range(0, 6)], spawnPosition, Random.rotation));
    
                //Wait before spawning another
                yield return new WaitForSeconds(3f);
            }
        }
    }
    
  6. Save your code and switch back to the Unity editor. When you click on the GameController object, you’ll see in the Inspector that two new fields popped up in the GameController script. Drag the ScoreBoardText and Timer game objects from the Hierarchy into those fields. They’ll now update!

Voice Commands

Now for some extra fun. We’re going to add in a couple of voice commands to enrich our experience. If you want to learn more about this topic, check out the Voice module on Holographic Academy.

  1. Open up the project settings where we enabled Spatial Perception. Enable the Microphone capability so we can hear voice commands.
  2. Make a new Scene named Menu and save it in your Scenes folder. Open the Menu scene if you haven’t yet.
  3. Bring up the Build Settings under Menu: File -> Build Settings. Make sure your Menu scene is added to the Scenes In Build (click Add Open Scenes to add it) and make sure Menu is before Main so it loads first.
    Build Settings

    Build Settings

  4. Once again, delete the main camera and add the HoloLensCamera prefab, the InputManager prefab, and the CursorWithFeedback prefab.
  5. We’re going to make a “button” that we can tap to start the game, but also responds to a voice command to start as well!
    Make a 3D cube named Button and set its Transform Position to X: 0 Y: 0 Z: 2 and its Scale to X: 0.5 Y: 0.2 Z: 0.01.
  6. Drag the 3DTextPrefab into the Hierarchy once more, and rename the game object to StartText. Make sure to change the font settings to Segoe UI or the font of your choice.
    Change the text mesh to say:
    Look Around The Room!
    Then Tap To Start
    (Or say “Start Game”)
  7. Create a new C# script in your scripts folder called MenuButton. Drag the MenuButton script onto the Button (cube) game object. The script is as follows.
    using HoloToolkit.Unity.InputModule;
    using System.Collections;
    using System.Collections.Generic;
    using UnityEngine;
    using UnityEngine.SceneManagement;
    
    public class MenuButton : MonoBehaviour, IInputClickHandler{
    
        public void OnInputClicked(InputEventData eventData)
        {
            SceneManager.LoadScene("Main");
        }
    
        void OnStart()
        {
            SceneManager.LoadScene("Main");
        }
    
    }
    
  8. Now our button will start the game if tapped! But we also want it to respond to voice. Create another C# script named SpeechManager and also drag it onto the button game object. When we say “Start Game”, it will broadcast calling the “OnStart” function, which we have in the MenuButton script. We also have some other keywords that’ll be used during the game.
    using System.Collections.Generic;
    using System.Linq;
    using UnityEngine;
    using UnityEngine.Windows.Speech;
    
    public class SpeechManager : MonoBehaviour
    {
        KeywordRecognizer keywordRecognizer = null;
        Dictionary<string, System.Action> keywords = new Dictionary<string, System.Action>();
    
        // Use this for initialization
        void Start()
        {
            keywords.Add("Start Game", () =>
            {
                // Call the OnStart method on every descendant object.
                this.BroadcastMessage("OnStart");
            });
            keywords.Add("Restart Game", () =>
            {
                // Call the OnReset method on every descendant object.
                this.BroadcastMessage("OnRestart");
            });
            keywords.Add("Main Menu", () =>
            {
                // Call the OnMenu method on every descendant object.
                this.BroadcastMessage("OnMenu");
            });
    
            keywords.Add("Show Grid", () =>
            {
                // Call the ShowGrid method on every descendant object.
                this.BroadcastMessage("ShowGrid");
            });
            keywords.Add("Hide Grid", () =>
            {
                // Call the HideGrid method on every descendant object.
                this.BroadcastMessage("HideGrid");
            });
    
            // Tell the KeywordRecognizer about our keywords.
            keywordRecognizer = new KeywordRecognizer(keywords.Keys.ToArray());
    
            // Register a callback for the KeywordRecognizer and start recognizing!
            keywordRecognizer.OnPhraseRecognized += KeywordRecognizer_OnPhraseRecognized;
            keywordRecognizer.Start();
        }
    
        private void KeywordRecognizer_OnPhraseRecognized(PhraseRecognizedEventArgs args)
        {
            System.Action keywordAction;
            if (keywords.TryGetValue(args.text, out keywordAction))
            {
                keywordAction.Invoke();
            }
        }
    }
    
  9. Switch back to the “Main” scene (your main gameplay level). Drag the Speech Manager script onto your GameController object.
  10. As you might have seen in the Speech Manager script above, there are a couple of voice commands that we haven’t yet coded in. We’ll be adding two functions to the GameController script, OnRestart() and OnMenu(), so that “Restart Game” will restart the game and “Main Menu” will take us back to the main menu. We’ll also update FixedUpdate() so that we go back to the main menu when the game ends.
    using UnityEngine.SceneManagement;
    
    //...
    
     void FixedUpdate()
        {
            //Update Timer and Screen Text
            timer -= Time.deltaTime;
            timerMesh.text = "Time: " + Mathf.Round(timer);
    
            if (timer <= 0)
            {
                OnMenu();
            }
        }
    
    //...
    
     //Voice command for reseting level
        void OnRestart()
        {
            SceneManager.LoadScene("Main");
        }
        
        //Voice command for loading menu
        void OnMenu()
        {
            SceneManager.LoadScene("Menu");
        }
    &#91;/csharp&#93;
    
    </li>
    
    <li>Final step! Let's add functionality to be able to view our spatial map in game! In the Hierarchy, drag the SpatialMapping game object over the GameController object until it's highlighted- there should now be a dropdown arrow next to GameController with SpatialMapping under it (see picture below). Doubleclick the Spatial Mapping Manager script on the SpatialMapping game object to open in the editor. Add these two functions after the last function in the script.
    
    [csharp]
       //Called by voice command- Shows visual mesh
            void ShowGrid()
            {
                drawVisualMeshes = true;
            }
    
            //Called by voice command- Hides visual mesh
            void HideGrid()
            {
                drawVisualMeshes = false;
            }
    

    GameController parent of SpatialMapping

    GameController parent of SpatialMapping

Play Your Game!

And that’s it! Play your game!
Your Fruit Warrior MR game should open at the Main Menu, start by tapping or voice command, spawn fruit with spatial sound for 120 seconds, clear the field and dock your time/score when you tap a bomb, show/hide the spatial mapping mesh, and show your score and time!

What’s Next

To practice game design- there’s many ways to to improve the game right here.

  • Add feedback for the user after a fruit is tapped (eg. sound effect, visual effect like poof of smoke, animation, etc).
  • Add power-ups that change gameplay, like spawn fruit faster or slower, add more bombs, increase points.
  • Create an online leaderboard for top scores!
  • Add a soundtrack and more levels!

More HoloLens- there are more tutorials on the Holographic Academy website. After you’ve completed the tutorials, try making your own apps for HoloLens that expand on aspects we touched on, such as more gestures, or Placement or Occlusion in spatial mapping.