Done: Night at the museum ☑️

I’ve been AFK for a while (holiday) so I just wanted to post the details of the final version of “night at the museum” I finished a few weeks ago that I delivered as part of my Udacity VR course.

The project went well (in that I passed) but I have to admit it was an exercise in restraint as much as anything. Like a lot of people, I tend to have big plans for everything I make but those plans are not always practical. This project was no exception. While I am ultimately happy with what I created, I do consider it very much a minimum viable project. When it comes to working on assignments like this there are a number of things to consider. First and foremost for me is available time.

I’m totally loving learning VR development, at this point, I spend almost all my free time doing it. But working full time and having a young family is a busy time in life, so if I want to progress projects I have to be efficient with my time or things can stagnate. Also, I currently pay for my studies via a monthly subscription so every extra month a project slips into comes at a significant extra cost.

Just to give you a few examples of compromises made on this project:

  1. Playing content at each of the stations was fairly limited and doesn’t have much finesse. For example, the final build allows the user to play audio from all 5 stations at the same time, this leads to a fairly horrible experience if a user does this
  2. I reused a museum model from a previous project because it was faster than building my own. This means it’s not really an ideal setup. Space feels a little constrained and doesn’t provide much room to move about. Most real museums are quite spacious so it didn’t really fit the aesthetic I was aiming for
  3. I wasn’t that happy with the spatial audio implementation. The environment had a bit too much reverberated for my liking and given more time I would have improved this a bit
  4. There’s no environmental audio, just the content of the stations. I’d preferred to add some atmosphere to the scene just to add the feel of the place

There are loads of other things that could be better about the final deliverable, but ultimately what was made “did the job”. Creating anything is always a series of compromises and I think delivering something you can live with and (resourcing permitting) can be built upon, is more important than getting things perfect. Perfection, after all, is the enemy of the shipped.

If you’re interested in the project’s code, it’s all available on my Github account. Enjoy.

Optimising C# scripts in Unity 🏎

C# logoThe more I create C# scripts in Unity the more I learn about optimising those scripts. With code there are 100s of ways to solve a problem, not all of them efficient, so learning to check your codes performance seems like a must.

Things to look for

So far I’ve come across a few simple things to look out for on the projects I’ve been coding. I’ll post more as I come across them in future.

  • Anything that’s called every frame creates consistent load on the CPU, GPU or both. If what’s called is inefficient or large it will have a negative effect on frame rate. In VR this is particular bad given low frame rates can make your user physically sick. Be sure to look at what’s in the Update and FixedUpdate methods and do your best to optimise the code as much as possible.
  • Working on mobile we will almost always be maxing out the graphics capability of the device. This effects the performance of the Update method. On the other hand FixedUpdated runs in sync with Unity’s physics engine. This will make sure your physics will be the most accurate and consistent. FixedUpdate is called every “physics step”, so is regularly used for adjusting physics (rigid body) objects. Make sure you always put your physics code in FixedUpdate.
  • Avoid using GameObject.Find. It basically requires the system to cycle through all of the GameObjects in the game to find the right one, which obviously is expensive for the CPU. If you know what object we are using ahead of time, define the object at the top of the script and reference it that way. GameObject.Find is useful for testing out ideas, but that’s about it.

Adding spatial audio in VR 🎧

Part of my Night at the Museum project is adding audio descriptions to the objects on display. To do this in Unity I’m making use of the spatial audio capabilities of the Google Cardboard SDK.

As with most things in Unity, there’s a fairly reasoned logic to how this is done. You need three main things:

  1. An audio source. You can think of this as the media player, it is literally the source of the audio within your environment
  2. An audio clip. This is the actual audio file that will be played by your audio source. So think of this as the song that will be played by your media player
  3. An audio listener. This is effectively your virtual ears within the virtual world. Normally this is attached to your main camera.

To give you an example of how I this works in practice. I first attached a GvrAudioListener to my main camera. This will allow my player to hear the sounds produced. Next, add a GvrAudioSource to a GameObject you want to produce a sound. Now load your audio source (your media player) with an audio clip. There are a few different supported formats, any old mp3 will work nicely. Just drag your MP3 into the audio click field of your GvrAudioSource. By default, the audio will play on wake, so just hit play to enter game mode and you should hear your new audio right away.

For more information on the ins and outs of Google Cardboard Audio capabilities, check out the documentation.

Pro Tip:

If you find your audio doesn’t play, try clicking edit -> project settings -> audio and make sure the “Spatializer plugin” is set to Gvr.

More progress 👨‍💻

Managed to get everything to compile to the phone again. Also, I can move about the place now 🙂

The big issue was around the legacy Google Cardboard SDK junking up the project. Completely deleting all SDK related files, old and new, then reinstalling the new was the quickest solution.

Once that was done, recreating key items like the events system and camera using the supplied GVR SDK components made life relatively straightforward.

Big thanks to synthercat for his wise guidance.

Check out the video below.

Clicking and framerate fixed

Getting back on track 🔧

So that thing I broke, it’s getting fixed. Little by little, day by day.

So far I’ve managed to get:

  • VR working properly again (at least I think I have)
  • select objects again
  • the FPS back over 60

Unfortunately I still can’t compile to my phone but hopefully, the weekend sorts that. It’s funny how breaking this thing has been possibly the most educational part of the course so far. I’m not saying I’d make a habit of it, but it was a net positive overall.

Here’s the latest update:

Clicking and framerate fixed

How to make stuff ⚙️

Making stuff is difficult, especially when that stuff is new. This is something I’m all too aware of. I was progressing nicely with my Night at the Museum project when (in my wisdom) I decided to upgrade my production tools mid-project. A word of advice, never do that.

Spending hours fixing a long list of issues is frustrating, but it isn’t a total loss. The key to success when you hit a bump in the road (like sucking at it, hating self etc) is to keep going! It’s not always easy but if you ever want to get good at anything, the key is preserving. Even in the face of self-loathing and a general distrust of your own abilities.

When you’re doing something new, embracing the tough times and roadblocks as an opportunity is key. You’ve got to see it as a way to toughen your resolve. If you can’t turn adversity into a win, then that adversity will win and you’ll stop progressing.

So the next time you find yourself making a virtual museum experience and your:

  • VR cameras stop working
  • you can no longer select items
  • your frame rate drops to a nausea-inducing 27 fps
  • you can no longer compile to a device
  • all of the above is your fault because you made an idiotic decision
  • delays cost you $300 per month in additional course fees (seriously send money now!)

Don’t give up, keep going! Not only that, tell other people about it. Seeing other peoples screw-ups is encouraging to others on the path to something new.

Project: Night at the Museum 🏛

I’m working on a project for my studies with Udacity. The goal is to create a VR experience that demonstrates my research into a VR company. I’ll do a series of posts on this as I progress along the path to completion.

To start here’s my first crack at the initial documentation for the project:

Statement of purpose

The purpose of the "Night at the museum" application is educate the user on the HTC Vive VR solution. The experience is based on an exhibit style scene, which contains 5 "stations", each covering some aspect of the HTC Vive.

Persona

The target audience for this application is anyone interested in VR that would like to learn more about the HTC Vive VR solution. Here’s a basic persona from them:

S03e13_percival_1600x1200Name: Wade Watts
Age: 18
Role: Budding VR enthusiast  
VR Experience: Experience with google cardboard but limited beyond that
Quote: "VR is very exciting, I can't wait to learn more"
About this person: Wade has grown up with computers, game consoles, tablets and smart phones. He's a digital native that loves to escape into his devices. VR is the ultimate expression of that escape.

Concepts

I’ve started by looking at different museum and exhibit styles to select a something that fits the aesthetic I’m looking for. After looking at altogether too many museums (if that’s possible) I eventually settled on classic Greek style building with exhibits setup like sculptures on pedestals. I’m using the following images as inspiration for the design:

I like this style, it reminds me of some of my favourite museum tours, and I’ve also done some work in a previous project that shares a similar style. This should allow me to reuse some of the assets from that project, meaning I can move faster. Always important when you’re pressed for time.


Concept build

Having established a concept for the style of the experience a setup to make a concept build. Using the assets from previous work made putting something workable together relatively easy.

I added a terrain GameObject with a few mountains in the distance to give the scene a sense of depth. I dropped in a “temple” model that is a good fit for a museum and added a few items to flesh out the concept. For the actual stations I acquired an HTC Vive model from sketchfab user Eternal Realm.  I also kept an eye out for the scenes frame rate to make sure I wasn’t introducing anything at this early stage that had a big impact on performance. I still have a few more models to acquire but I’m happy with the direction the concept is heading in.

Screen Shot 2017-08-16 at 9.06.16 PM
A view from a potential starting position for the user.

Screen Shot 2017-08-16 at 9.06.29 PM
The users view approaching the entrance of the museum itself.

Screen Shot 2017-08-16 at 9.06.37 PM
A simple example of what a “station” may look like within the museum. A floating Vive headset over a stone platform.


Next steps

Now that I’ve got the basics in place the next thing on the agenda is the first round of user testing. I shouldn’t have to much trouble acquiring a suitable test subject to get things moving. I will be interesting to see how the scene changes and progresses over the course of development.

Types of VR 🐴🦄🌈

As discussed earlier, there are loads of terms floating around in the world of VR. But it’s not just things on the spectrum between AR and VR. There are also various types of experiences within just VR.

Rather than try to cover every possible variant here, I’ll talk to the types I’ve experienced personally, and as a result what I think are the important components that make for good VR.

660px-6DOF_en
6 degrees of freedom (6DOF)

I’ve used quite a range now, from 3 degrees of freedom (3DOF) Google Cardboard, right up to a high immersion 6 degrees of freedom (6DOF) HTC Vive setups. Here’s an overview of what I’ve used so far:

Cardboard (3DOF)

I’ve got what can only be described as a growing collection of Google Cardboard headsets. The more you get involved/exposed to the VR community and events it seems sort of inevitable that you’ll pick up a few along the way. This is part of what makes Cardboard so great. It’s highly accessible. If you’ve got a modern iOS or Android-based phone you’ve got 99% of what you need to get going. Cardboard headsets range in sophistication and price, from $15 bits of literal cardboard, to “fancier” setups that strap to your head with headphones for $50+.

IMG_4518
Emma sporting one of my “fancy” Cardboard headsets

Accessibility aside, the sorts of experiences you can do in a Google Cardboard really are a perfect entry into VR. It’s a fantastic way to take users to inaccessible locations and show them things. Examples include estate agents showing an out of town client through a property, or placing a marine biology student at the bottom of the sea. The biggest and most obvious shortcoming is the low levels of both emersion and interactivity available to carboard users. Effectively you can only interact via the users’ gaze or a single button interface on the headset itself. This sort of thing is fine for a certain segment of VR but obviously, you’re not exactly going to be entering the Matrix anytime soon.

One of the things that can’t go unsaid about Cardboard’s accessibility is how it lowers the barrier to entry to VR development. As I said in my hardware for VR post, you can get into VR on a very low budget with cardboard as a target device, and that’s a big deal.

If you’re interested in giving Cardboard a go, take a look at Google’s Cardboard site.

Gear VR (3DOF + 3DOF controller)

The Gear VR is a fascinating step up from Cardboard. In many ways, it’s a very sensible direction to take VR. At around $100 USD, plus the cost of a compatible Samsung phone, the cost is more accessible than a full emersion setup. Obviously, a phone strapped to your head is only going to take you so far visually. You’re also only going to get a 3DOF (that’s pitch, roll, and yaw tracking of the users head) which limits the feeling of presence. However, the addition of a controller, even a simple 3DOF one, means far greater levels of interactivity. To my mind, having any control scheme that simulates a users hand greatly increases the value of the setup.

Having said all that, the Gear VR isn’t something I’d recommend unless you’re already a Samsung phone user. Its benefits are not worth the $1000 NZD investment over a cardboard setup. It’s also worth mentioning this segment of the marketing is increasingly competitive. Google have Daydream in this space. I haven’t used it personally but conceptually it looks quite similar. If you think a setup like this is something you’d like to explore you should consider both before making your decision. You should also be aware that Daydream looks to be releasing stand-alone headsets later this year that may be the most interesting take on this segment yet. Unless you’re in a mad rush for a VR setup I’d consider waiting until later in the year before buying.

HTC_Solo_Controller
A promotional sketch of the HTC Daydream standalone headset

HTC Vive (6DOF + 6DOF hand controllers)

I like to think of the HTC Vive as a window to the future rather than the revolution that some tout it to be. The experience itself is unmistakably amazing. The level of immersion is so high that I stopped thinking about controlling a computer simulation, and started just naturally interacting with it as if it was a real environment. It’s actually quite a strange thing to describe, but having used this setup dozens of times I’m convinced this as far more to do with your hands than it does your head.

The second you enter a VR environment, and you have some control of it via controls that closely approximates your hands, you stop thinking about the tools and just start being. Before you know it you are truly in VR. Based on the impact of hand controllers I’m a bit surprised more isn’t being done with cardboard headset combine with hand controllers, but then again there are likely significant technical challenges I’m not aware of.

HTC Vive headset in action
The HTC Vive in action

Conclusion

Unsurprisingly the Vive is by far my favourite VR experience. However, the reason I say it’s a window to the future is the ticket price. These things cost a fortune. A Vive with hand controllers cost $799 USD. Which on its own is a big investment for most. Not inaccessible, but significant. The real kicker here is you need a high-end Mac or PC to connect the headset too. Taking that into account you can be looking at a $3000+ NZD machine to get started. Not exactly mainstream pricing.

It’s just a matter of time before these things transform into a more mainstream format. It’s quite likely the coming stand-alone headsets are the beginning of that. How much they cost and how good they are is yet to be seen, but I for one am excited at the possibilities.

The sooner more people have immersive VR experiences and reasonable consumer prices the faster this industry prices. With gateway drugs like Cardboard giving people their first taste for a low price, it’s sure to an explosive industry once the hardware catches up.

Process: Puzzler 👨‍💻

Over the past few months I’ve been studying with Udacity to learn VR software development. The course has been great so far, so I thought I’d share a little of what I’ve been up to with a game called “Puzzler”.

About Puzzler

Puzzler is a simple VR experience for Google cardboard. Basically anyone with a phone and a $15 cardboard headset can give it a try. The game has a simple UI where the user is thrust into a dungeon, where 5 magic orbs present a puzzle. Successfully playing a single round of “Simon” gets the player out of the dungeon to victory.

Here’s a brief video of the “final” version of the game:

The process

Of course, I didn’t just wake up one morning and build this thing. There is quite a process to go through to create a good VR experience. As I mentioned on this blog before, a good process can make all the difference when you’re building for someone that isn’t you. Which, let’s face it, is almost always.

The approach taken for this work was no exception so let’s break it down a bit.

Statement of purpose

I started the process by creating a simple statement of purpose for the game:

Puzzler is a mobile VR app which gives new VR users a quick taste of VR via their existing smart phone. The entire experience should take no more than a few minutes and be accessible to most anyone that's physically able.

With this in mind, I selected the nearest available human to be a test subject and personify my ideal target audience. My 6-year-old daughter Emma was the winner on the day. When starting out the build, first I documented a little about her, so I had a clearly defined outline to work too. Here’s the persona for Emma:

Persona

Image:
Emma
Name: Emma
Age: 6
Role: Child
VR Experience: Has played a few simple VR games, but not many
Quote (that sums up their attitude): “Can I play with your phone thingy, it’s cool.”
About this person: Emma is an enthusiastic VR user, she loves to explore and have adventures. She’s young and enjoys content that's exciting and interesting, but not too scary or intimidating. She will ask lots of questions and enjoys the discovery. I think it's fair to say she has moxie.

With a clear purpose and audience/persona defined it was time to get going with an initial design and an “alpha” build of Puzzler.

Concept sketches

I started by creating a bunch of really nasty looking sketches for what puzzler might look like. And when I mean nasty I mean nasty.

Here are a few concepts I had for the game initially:

My thinking with the above designs was my audience is young. Given that, it was better to use a large simple UI that could be easily understood and a very simple scene design that wasn’t overly difficult to understand. Once I sketched out something I was happy with I then took the designs and built something in Unity.

User testing

With the first cut done it was time to get going with user testing. It’s so important to get an early version in front of your audience to test assumptions. My early tests were really all about determining the basics, like the scale of the scene in relation to the subject.

Basically, from there on it was a process of iterating on the project. A mix of making assumptions, asking questions, and testing it all as often as was practical. To give you some idea of the things I asked, here’s a few of the Q&A sessions I had with Emma:

User test 1

Me: How big do you think you are in this scene?
Emma: I feel a bit smaller, normally I think I’d be taller than that barrel. I feel little.
Me: What’s the mood or atmosphere of the environment?
Emma: A bit spooky, but I like the purple balls. Not too scary, there are no witches, I don’t like witches. It’s a bit bright for a dungeon though.
Me: Is there anything you find difficult to see, or anything visual you think could be improved?
Emma: No I can see everything, but I can’t hear anything should I be able to hear things?
Conclusion: Emma picked up on the spooky dungeon, she was feeling too small so I adjusted the player's height slightly. I moved the orbs to be closer to the player since Emma liked them so much. I also added some sound to the environment.

User test 2

Me: How big did you think you are in this scene?
Emma: I think it's right. I'm the right height.
Me: What’s the mood or atmosphere of the environment?
Emma: Spooky, the sounds are spooky, it sounds like night time. It’s a bit dark.
Me: Is there anything you find difficult to see, or anything visual you think could be improved?
Emma: The balls are too close to me. I feel like I’m going to bang into the balls.
Conclusion: Emma feels the right size now, but I think the scene is a bit too spooky for her now and the orbs are making for feeling a bit close. I’ll move the orbs to a different location and increase the lighting a bit so it's not so scary.

IMG_4518
Emma hard at work user testing

User test 3

Me: How do you feel about the scene
Emma: It looks cool, I like the lights, it feels dark outside and spooky inside, but warmer.
Me: Do you understand how to start the game
Emma: Yes I click the big “Start”
Me: Do you know how to play the game?
Emma: Yes it's like “Simon says”. I do what the puzzle does.
Me: Is there anything else?
Emma: The balls are in the way of the door. I have to crash into them when I win. The game is too hard for me.
Conclusion - The game has matured to the point Emma is happy with it and can play it well, she still isn’t happy with the placement of the orbs so I’ve moved them back and did a quick retest, she’s now happy. I also reduced the complexity of the game to 5 steps.

Breakdown of final deliverable

So, in the end, we’ve got a dungeon puzzler that isn’t too scary for a 6-year-old to play. Emma likes the game and can successfully play it.

The basic break down of the “final product” is:

  • The user is presented with a simple UI screen to start the game.

Screen Shot 2017-07-30 at 3.06.40 PM
The start screen as see from the Unity editor

  • On clicking start the player is moved into the dungeon where they are presented with 5 magic orbs. Emma’s feedback had a significant influence on orb placement and the feel of the dungeon.
  • The orbs chime and blink in a random sequence which the player must complete to “escape”.

Screen Shot 2017-07-30 at 3.05.49 PM
The magic orbs as seen from the Unity editor

  • If the player fails to repeat the sequence a “fail sound” plays, and the puzzle is repeated to give the player another chance. If they get it correct they’re moved out of the dungeon and are presented with congratulations and the ability to restart.

Conclusion

Over all, it was a fun educational exercise doing this project. Plenty of things to learn from a process perspective but also in Unity 3D development. The work has given me a few ideas for other projects to experiment with, and further solidified my view of user testing and rapid iterations in the development of a product.

Next steps

Next I think I’ll move on to a new project, but stay focused on something for that target audience. I think it would be of value to include a few more testers, including Emma’s brother and friends.

If you’re interested in playing around with the above project I’ve shared the source code via bitbucket. The other projects I’ve done via the Udacity course are also available on the Geekpulp bitbucket page.

Hardware for VR development 🖥

Starting out in VR development it’s easy to think you’ll spend the earth on special hardware to get going. The reality is that’s just not true. The below 360 images (which I took with the google street view app on iOS) is fairly rough as 360 images go. The room was a mess as I was in the middle of a fairly ugly hardware transition. The stitching isn’t very good, so there are lots of bung parts to it, but you get the idea of the space and gear I work with.

Until recently I was using a late 2009 Mac Mini and was managing just fine on my Udacity VR developer course. Well, I was managing, not sure about the fine part. So I upgraded to a late 2012 mac mini, still 5-year-old hardware, and I’m going gangbusters doing google cardboard development with it.

Here are a few more details on the hardware setup I’m running in the 360 image above:

Hardware

Displays – Two second-hand 1080p displays I purchased on Trademe for $300 total. The one on the left is in portrait orientation for my code editor. The one on the right in landscape for the Unity editor. Both are on this monitor stand I purchased from PB tech for around $100.

Input devices – There’s a bunch of keyboards in the shot but the one I’m actually using now is a Logitech K380. I managed to get a refurbished one on 1-day for $40 shipped. Honestly, it’s such a good keyboard I’d be happy to pay retail for it. Logitech claims 2 AAA batteries will last 2 years. Far better than the 2 months I was getting out of my Apple keyboard and a fraction of the price. For a pointing device, I’m using a stock standard magic mouse. I’d prefer a Logitech MX Master 2S, but it’s a nice to have rather than a must.

Computer – I have two in play now. My development machine is a late 2012 Mac Mini I purchased on Trademe for $500. Even though it’s from late 2012 it’s actually the fastest model of Mac Mini ever made, the quad-core i7. I’ve put 16GB of ram in it and replace the spinning disk with an SSD. I’m really happy with its performance. It does everything I need for Unity development, especially given the contrast of my other machine the late 2009 Mac Mini. That machine is now running as the server for the house. I’ve modified it a bit adding 4.5TB of storage and 8GB of ram. It can no longer run the latest version of MacOS but it’s doing a great job as a cache and Time Machine server.

Future plans

Audio hardware – Something I’m really aware of with VR is the impact of audio. Obviously, visuals are important in VR, but good quality audio can also have a huge impact on immersion. It’s also very useful in directing the users’ attention. I’ve got a few bits of hardware on order to support creating more of my own audio for various projects. The main bit of kit is a USB audio interface, or more specifically a Focusrite Scarlett Solo (2nd Gen). I got mine via Amazon as it actually worked out a bit cheaper than buying local, even with the shipping costs via youshop.

I’m trying to play a longer game with this purchase. It’s likely better than I need at this stage, but it will last me 15 years or more.  With the budget in mind, for the rest of my audio gear, I opted for a cheap XLR mic, arm, and pop filter from Aliexpress. I’ll upgrade those later when I have more funds available and other parts of my setup mature.

360 camera – At the moment I’m just using my iPhone 6s and various 360 apps. Over time I’m expanding my capabilities as I need/can afford them. I’m in the market for a 360 camera, likely something like the Ricoh Theta S or a 2017 Gear VR camera. I’m hoping to purchase one of these in the next 3-4 months.

Mac Pro and a 6DOF VR setup – Ultimately I want to replace my Mac Mini with a far more powerful setup so I can expand into more immersive VR development. I’m aiming for a new Mac Pro when they become available later next year. I’m also delaying the purchase of something like a Vive or an Oculus as long as I possibly can. It’s such early days in VR hardware and I’d prefer to wait to buy when there’s gen 2 or even 3 out. I suspect 2018 is going to be an expensive year.

The thing to take away here is you don’t need to spend the earth to get started with VR. Some second-hand hardware and a drive to learn will take you a long way before you need to invest more.