Mesh Effects – HoloLens Shader Pack
There are a lot of HoloLens apps on the store, and there are just about as many spatial mesh visual effects.
So this week I wanted to explore some of them.
Basically, the Spatial Mapping Manager is designed to let you replace the Spatial Material (which by default is the white wireframed one).
There is a nice Unity Asset Store package I came accross called the “HoloLens Shader Pack” for $27.50.
Here’s how to use it with the MixedRealityToolkit:
- Crreate a new scene
- Delete the main camera and directional light
- Drag in the HoloLensCamera prefab
- Drag in the InputManager prefab and the Default Cursor prefab (not strictly required).
- Drag in the SpatialMapping prefab
Sanity test build that and make sure you can look around and see the spatial mapping.
COOL!
Now, lets integrate the shaders
- Drag the SpatialMappingCalcNormals onto the SpatialMapping gameobject. (It will call recalulate normals on each surface mesh).
- Now we’re going to simply replace the surface material “Wireframe” with any of these surface mapping materials from the pack:
Whats the best way to look at these? You can build each one for the Hololens, but here’s a quicker way: See it in the Unity editor.
Follow my tutorial for saving a room and then we’ll load that room in the Unity editor.
Add FileSurfaceObserver component to the SpatialMapping game object. Set your room file name, eg: office.room is office:
Now hit PLAY.
Drag on a new material while Playing and press “L” to reload the mesh.
Here’s a walk though video:
Spatial Sound
We’re in a 3d space, so we definately want the audio emitted from Holograms around us to be spatialised audio too.
So how do we do that? Do we just use 3d Audio Source’s on our game objects with standard unity? Will that work with the speakers in the HoloLens?
So, the answer is “not quite”. We need to enable the Microsoft Spatial Sound plugin to correctly spatialise sound.
Enabling Correct Spatial Sound in Unity3d
The Microsoft Spatial Sound plugin is the documented method used in both the Academy 220 Spatial Sound Tutorial and Mixed Reality Toolkit Spatial Sound.
Go to Edit->Project Settings->Audio, then select the Spatializer Plugin “MS HRTF Spatializer”:
Now, your Audio Source components will have a new flag to indicate they will use the plugin.
Here is a before and after difference for the Audio Source component I placed on my jet 3d model:
So, now simply checkmark the Spatialize checkbox and build and try out your sound in the HoloLens.
Here I set a looping jet sound and checkmark the Spatialize option:
Now run it in the Hololens andas I walk around the sound is coming from the exact spot in 3d space. WOW!
And the best part – no code required!
Sound Occlusion
Spatial sound is all well and good, but it opens up a new problem – what if the sound is nearby, but in an ajacent room? There is a wall blocking the sound.
Unity itself doesn’t handle this. But the Mixed Reality Toolkit gives us a way to manage this scenario with Audio Emitters and Audio Occuluders.
Here is a scenario (as demonstrated in the Mixed Reality Toolkit AudioOcculusionTest scene – a wall blocking the users path to an audio source:
In this case, the wall should cause the audio to be less audible.
The way it works, is that you can add an AudioOcculder component to any object with a collider (the wall in this case). Then add an AudioEmitter component to any GameObject with an AudioSource. When audio is playing, the emitter will determine if there are any colliders in the way to the users Audio Listener, and automatically alter the audio source signal.
Shown below the wall has it’s AudioOcculder and the Audio Source has it’s AudioEmitter:
There is a performance impact with an AudioEmitter scanning the scene, so it is not recommended to have too many of these. Better to turn off the audio at a certain distance from the user.
More details here:
https://github.com/Microsoft/MixedRealityToolkit-Unity/tree/master/Assets/HoloToolkit/SpatialSound
Leave a Reply