Making musical toys at Unity Hackweek

Initial Idea & Inspiration

At Unity Hackweek 2018, a group of us worked on a music maker, which was eventually named Super Sound Synthesizer 3000. The aim was to create a music sequencer that lives in the Unity Editor as an editor window.

The idea for this came from mentoring at Girl Game Maker Day. We had provided code and art for the girls to make their game, but music was an afterthought (as it too often is in game jam environments, unfortunately). Even asking the girls to download some free music from the internet was a challenge, as most free music sites require registering, and many of them were too young to have an email address of their own. We needed something simpler to create music without learning a new tool.

After this initial idea, a lot of the early inspiration came from Doodle Studio 95. It’s fun, it’s playful, and it’s simple to create sprites and animations that can go directly into your Unity project. One of the main things we wanted to achieve with our music maker was that it should be easy for a beginner to create something, but someone more skilled can also create something great if they want to; Doodle Studio does this so well (as shown by the trailer for it!)

The team

I started rounding up the team a couple of months before Hackweek even started. First, I approached Andy Selby, since I knew he enjoys making digital music, and he had made music-related Unity projects in the past. He said “oh, yeah alright then”. One down.

Next I asked Siobhan Gibson and Hope Rudd. I can't remember which order I asked them in, but they both said something along the lines of “oh, that sounds fun”. Ideal.

Siobhan is really passionate about fun and simple game-making tools, so I hoped she’d be keen. Hope is new to programming and I knew this would be a great chance for them to deep dive into a project and work collaboratively with a team, so I’m glad they agreed! Hope is also in a band, so extra points for them.

Other people joined our Slack channel to ask questions, but it ended up just being the 4 of us in the team. I didn’t realise until later on how beneficial this would be. We all knew each other’s strengths and weaknesses, so we knew how to work together (even though none of us had actually worked in a group together before).

Further research

On the first day of Hackweek, we planned to only research, experiment and prototype. This gave us a chance to get used to the atmosphere of the hacking hall, and all decide how the project should look and feel.

Siobhan spent the day looking at other playful music tools (because us saying ‘Doodle Studio but music’ is not a sufficient design spec).

One of the stand-out tools that Siobhan found was Song Maker. We all fell in love with it pretty much immediately.

Toc and roll is also great. We especially loved the way you could use your voice and a sample, and that is has effects pedals! Siobhan was able to quickly made a lovely song (that actually got stuck in my head) and had a fun time showing us how to use it, which the the exact sort of thing we wanted from our music maker.

Siobhan used these tools as inspiration and created a mock-up in excel for how ours could look.

Andy has been making digital music for years, so he was able to provide great insights from the more professional angle, too. On the more professional side, we looked at tools like FLStudio and Reaper.

Microphone Recorder

As a sort of coding warm-up, Hope and I pair-programmed a very mini prototype of one of the desired features. We created an editor window where you enter how long you’d like to record for in seconds, then press record to start a microphone recording. This recording is saved as a .wav file inside the Unity project. We did this so we could get an idea how the workflow could work, and how wav files are created and saved. It also gave Hope a chance to make Editor Windows, and learn about the process that goes into creating tools. (Side note: Hope is a total joy to work with. They made notes about everything we went over, and asked good questions as we went. Not only did they learn a lot (hopefully!) but it made me slow down what I was doing and think about it.)

We used this code to save the AudioClip as a .wav file, and we had a lot from the Unity Manual to learn about how Microphones work with Unity.

UI elements

Super Sound Synthesizer 3000 was made using Unity’s UI Elements feature. We chose this instead of the existing Editor GUI so that we’d have more flexibility and a chance to play with a new area of Unity. UI Elements is much closer to web development than EditorGUI. This intimidated me, but Andy was great at learning the information and explaining it back to me. We had to do some odd tricks to make things work (for example, we had to manually flip through frames of a sprite animation because there is no support for gifs out of the box).

Buttons using GUILayout are simple to implements, as shown here: GUILayout.Button. A button in UI Elements requires a few more lines of code, as shown in their example project: UIElementsExamples.

Wav saving fail

The biggest hurdle was trying to get the audio created with our synthesizer to save as a .wav file. I investigated a few different methods:

• Joining audioclips
• This gets confusing when more than one clip plays at a time
• An asset package: Audio Clip Combiner
• Recording from “in-game” audio
• We couldn’t figure out how to access it without using something like Soundflower

We ended up using a hacky version of the microphone recorder that Hope and I had worked on earlier in the week. It waits until the audio loop is back at the beginning, and it records for the length of time of one loop. It was terrible and included all the background noise from the room, but creating our own way to record in-editor audio somehow could have been another Hackweek project! Definitely not something to do on a Thursday evening.

Last few hours

Earlier in the week, I needed a sound to test the synthesizer. Luckily, I could just easily record a small clip using the Microphone Recorder window. Naturally, I meowed. I did not think about how this would be played whenever someone opened the window. It sort of became a meme within the team. We HAD to include this in the final product, so in the last few hours, Siobhan created a cat icon, and I added The Cat Button to our window.

We also wanted to add tempo buttons, as we pretty much already had the code to do it. To make it beginner-friendly, we opted for a tortoise, a heartbeat, and a hare to illustrate slow, medium, fast. It worked really well and looked cute!

The Final Product

Watch the video here

The Future

We all had loads of fun working on this, and we'd like to keep it up! One of the things Hope started working on in the week was a way to add effects to the audio (like Reverb, for example) but we didn't have time to get this added. Hopefully, we can make some cool guitar-pedal-esque UI and get this hooked up :)
As UI Elements will eventually be usable for in-game UI, we'd like to make a small in-browser demo for this toy, too.