Making musical toys at Unity Hackweek

Initial Idea & Inspiration

At Unity Hackweek 2018, a group of us worked on a music maker, which was eventually named Super Sound Synthesizer 3000. The aim was to create a music sequencer that lives in the Unity Editor as an editor window.

The idea for this came from mentoring at Girl Game Maker Day. We had provided code and art for the girls to make their game, but music was an afterthought (as it too often is in game jam environments, unfortunately). Even asking the girls to download some free music from the internet was a challenge, as most free music sites require registering, and many of them were too young to have an email address of their own. We needed something simpler to create music without learning a new tool.

After this initial idea, a lot of the early inspiration came from Doodle Studio 95. It’s fun, it’s playful, and it’s simple to create sprites and animations that can go directly into your Unity project. One of the main things we wanted to achieve with our music maker was that it should be easy for a beginner to create something, but someone more skilled can also create something great if they want to; Doodle Studio does this so well (as shown by the trailer for it!)

The team

I started rounding up the team a couple of months before Hackweek even started. First, I approached Andy Selby, since I knew he enjoys making digital music, and he had made music-related Unity projects in the past. He said “oh, yeah alright then”. One down.

Next I asked Siobhan Gibson and Hope Rudd. I can't remember which order I asked them in, but they both said something along the lines of “oh, that sounds fun”. Ideal.

Siobhan is really passionate about fun and simple game-making tools, so I hoped she’d be keen. Hope is new to programming and I knew this would be a great chance for them to deep dive into a project and work collaboratively with a team, so I’m glad they agreed! Hope is also in a band, so extra points for them.

Other people joined our Slack channel to ask questions, but it ended up just being the 4 of us in the team. I didn’t realise until later on how beneficial this would be. We all knew each other’s strengths and weaknesses, so we knew how to work together (even though none of us had actually worked in a group together before).

Further research

On the first day of Hackweek, we planned to only research, experiment and prototype. This gave us a chance to get used to the atmosphere of the hacking hall, and all decide how the project should look and feel.

Siobhan spent the day looking at other playful music tools (because us saying ‘Doodle Studio but music’ is not a sufficient design spec).

One of the stand-out tools that Siobhan found was Song Maker. We all fell in love with it pretty much immediately.

Toc and roll is also great. We especially loved the way you could use your voice and a sample, and that is has effects pedals! Siobhan was able to quickly made a lovely song (that actually got stuck in my head) and had a fun time showing us how to use it, which the the exact sort of thing we wanted from our music maker.

Siobhan used these tools as inspiration and created a mock-up in excel for how ours could look.

Andy has been making digital music for years, so he was able to provide great insights from the more professional angle, too. On the more professional side, we looked at tools like FLStudio and Reaper.

Microphone Recorder

As a sort of coding warm-up, Hope and I pair-programmed a very mini prototype of one of the desired features. We created an editor window where you enter how long you’d like to record for in seconds, then press record to start a microphone recording. This recording is saved as a .wav file inside the Unity project. We did this so we could get an idea how the workflow could work, and how wav files are created and saved. It also gave Hope a chance to make Editor Windows, and learn about the process that goes into creating tools. (Side note: Hope is a total joy to work with. They made notes about everything we went over, and asked good questions as we went. Not only did they learn a lot (hopefully!) but it made me slow down what I was doing and think about it.)

We used this code to save the AudioClip as a .wav file, and we had a lot from the Unity Manual to learn about how Microphones work with Unity.

UI elements

Super Sound Synthesizer 3000 was made using Unity’s UI Elements feature. We chose this instead of the existing Editor GUI so that we’d have more flexibility and a chance to play with a new area of Unity. UI Elements is much closer to web development than EditorGUI. This intimidated me, but Andy was great at learning the information and explaining it back to me. We had to do some odd tricks to make things work (for example, we had to manually flip through frames of a sprite animation because there is no support for gifs out of the box).

Buttons using GUILayout are simple to implements, as shown here: GUILayout.Button. A button in UI Elements requires a few more lines of code, as shown in their example project: UIElementsExamples.

Wav saving fail

The biggest hurdle was trying to get the audio created with our synthesizer to save as a .wav file. I investigated a few different methods:

• Joining audioclips
• This gets confusing when more than one clip plays at a time
• An asset package: Audio Clip Combiner
• Recording from “in-game” audio
• We couldn’t figure out how to access it without using something like Soundflower

We ended up using a hacky version of the microphone recorder that Hope and I had worked on earlier in the week. It waits until the audio loop is back at the beginning, and it records for the length of time of one loop. It was terrible and included all the background noise from the room, but creating our own way to record in-editor audio somehow could have been another Hackweek project! Definitely not something to do on a Thursday evening.

Last few hours

Earlier in the week, I needed a sound to test the synthesizer. Luckily, I could just easily record a small clip using the Microphone Recorder window. Naturally, I meowed. I did not think about how this would be played whenever someone opened the window. It sort of became a meme within the team. We HAD to include this in the final product, so in the last few hours, Siobhan created a cat icon, and I added The Cat Button to our window.

We also wanted to add tempo buttons, as we pretty much already had the code to do it. To make it beginner-friendly, we opted for a tortoise, a heartbeat, and a hare to illustrate slow, medium, fast. It worked really well and looked cute!

The Final Product

Watch the video here

The Future

We all had loads of fun working on this, and we'd like to keep it up! One of the things Hope started working on in the week was a way to add effects to the audio (like Reverb, for example) but we didn't have time to get this added. Hopefully, we can make some cool guitar-pedal-esque UI and get this hooked up :)
As UI Elements will eventually be usable for in-game UI, we'd like to make a small in-browser demo for this toy, too.

Using Linq to count how many times an object occurs in a list

You have a list full of objects. Some of the objects are duplicates, and you want know know how many of each distinct item you have in the list.

I searched around for some answers, and I did find some StackOverflow answers that almost got me to the result, but it wasn’t quite right.

In my example, the list I want to sort is a list of custom type Run ie List<Run>(). In my code, Run is a struct that looks like:

public struct Run
{
     public string RunID;
     public string Config;
}

The StackOverflow responses were mostly long and hard-to-read Linq statements. I like Linq a lot, but it’s not always easy on the eyes, especially once you start adding extensions.

Instead, I came up with the following:

void ProcessList (List<Run> runConfigs)
{
     var grouping = runConfigs.GroupBy(o => o.Config);
     foreach (var configGroup in grouping)
     {
          string configName = configGroup.Key.ToString();   
          string count = configGroup.Count<Run>().ToString();
     }
}

Doing the above code allows me to use configName and count however I please (which in my case was outputting them to a csv).

Here’s the same example but for a List<string>():

var grouping = listOfStrings.GroupBy(x => x);
foreach (var stringGroup in grouping)
{
    string name = stringGroup.Key.ToString();
    string count = stringGroup.Count().ToString();
}

If you step through this code with the debugger, it will show that 'name' is one of the distinct values in the list, and 'count' is how many times that object appears in the list. These values can now be used however you like.

What is dynamic batching, and why is it off by default?

While reading this tutorial, I came to a section where it said something like “as you can see, dynamic batching has stopped working”. I did not see that. In the image in the tutorial “Saved by batching” on the stats panel had dropped to 0, but my stats panel still showed a number in the 1000s.

When I checked the player settings, I noticed dynamic batching was turned off. After a bit of investigation, I learned that dynamic batching is now off by default for 3d projects as of Unity 2018.1; but why is this useful?

A mesh is the part of a 3D object that describes its shape. When certain criteria are met, Unity is able to combine meshes from different GameObjects into one big mesh before they are drawn. This is called dynamic batching. The "dynamic" part means it happens while the game is running.

Sometimes (but not always!), it takes less time to draw one big mesh than it would to draw lots of smaller meshes. Dynamic batching occurs on the CPU, and drawing meshes occurs on the GPU. You can think of dynamic batching as spending CPU time to save GPU time. This means that it becomes more likely that the game becomes CPU-bound (meaning that the frame rate is lower than you want because the CPU is taking the most time), but you should ideally want as much as possible to happen on the GPU; that’s what it’s there for.

If your game is already CPU-bound, then dynamic batching will not help. But if your game is GPU-bound (meaning that the frame rate is lower than you want because the GPU is taking the most time), then it may help.

(FYI: The GPU equivalent of dynamic batching is GPU instancing.)

For these reasons, dynamic batching is desirable for old mobile devices that don’t have a GPU (or have a bad GPU), but it is not efficient for games intended for more modern devices.

Here is a helpful article that goes into a bit more detail: Why Are My Batches (Draw Calls) So High?
Also: https://docs.unity3d.com/Manual/DrawCallBatching.html

Accessing Shader Graph properties via code

You can create and name properties in shader graph using the Blackboard

These names really just serve as labels, so if you want to access these properties via code (to use material.SetColor, for example) then you need to peek at the shader code.

Right-click the master node and select 'Copy shader'

Paste that into text editor of choice and look at the Properties code block at the top 

My property was named _ColorTest and the actual name of it is Color_A2.. etc

Now I can do

material.SetColor("Color_A208B217", Color.magenta);

 

Coroutines

I'm working through a tutorial on catlikecoding.com and wanted to take a moment to make some notes about coroutines.

I have used coroutines in almost every game I've made, but not stopped to think about how they really work.


Source for the below text: http://catlikecoding.com/unity/tutorials/constructing-a-fractal/

My own notes and additions are in bold/blue

What does yield do?

The yield statement is used by iterators to make life easy for them (this part could potentially be explained better. Iteration is the act of repeating a process. Usually, when people talk about working iteratively, they are talking about working, making a small change or fix, then continuing to work and repeating that process, as opposed to doing it all in one go and fixing everything at the end. In this case, I think the sentence is talking about iterative functions and loops.).

To make enumeration possible, you'd need to keep track of your progress. This involves some boilerplate code (boilerplate code refers to code that has to be included in lots of different sections without being changed eg declarations or the basic empty template in HTML) that is essentially always the same. What you'd really want is to just write something like return firstItem; return secondItem; until you are done. The yield statement allows you to do exactly that.

So whenever you're using yield, an enumerator object is created behind the scenes to take care of the tedious bits.

Alternate definition: 'The yield statement is a special kind of return, that ensures that the function will continue from the line after the yield statement next time it is called.'

How do coroutines work?

When you're creating a coroutine in Unity, what you're really doing is creating an iterator. When you pass it to the StartCoroutine method, it will get stored and gets asked for its next item every frame, until it is finished.

The yield statements produce (return) the items (actions?). The statements in between – the stuff that you want to happen – are side-effects of the iterator doing its job.

You can yield special things like WaitForSeconds to have more control over when your own code continues, but the overall approach is simply that of an iterator.

Alternate definition: 'A coroutine is like a function that has the ability to pause execution and return control to Unity but then to continue where it left off on the following frame.' (source)

Post frequency increasing

If you notice that I'm posting more regular blogs, it's because I want to also use this blog as a sort of personal notebook where I store things that I find useful, but other people might also like (inspired my friend Ming Wai's blog)

I will still post bigger blogs every now and then, but I also want to write about smaller things to remind myself about them or understand them better.

Shaders aren't as scary as I thought: My first shader with Unity Shader Graph

Disclaimer: I have worked at Unity since September 2016, but this is not a marketing blog or anything like that. This is my personal blog with my genuine experiences and reactions. Also, my boyfriend works on the Shader Graph team, but he wasn’t actually even in the country when I was making these.

***THIS IS NOT A TUTORIAL***

If you’ve ever played any of my games, you will know that I am definitely not an artist. Art is fun and I appreciate it, but I’m not good at it.

For those new here; hello, I’m Sophia. I’ve been working as a programmer for a few years (in various roles/companies). I studied Physics at university, and I’m pretty good at maths, so I have been asked to help when my friends have had issues writing their shaders before. Small stuff like figuring out relationships between variables etc.

While I’ve always wanted to make shaders, I never really knew where to start. It felt very counterintuitive to write code relating to visuals but with no visual feedback.

During Scenario Test Week at work, my team has been working on a golf game. I was working on pickups/modifiers, and I wanted some cool effects to happen when certain modifiers turn on. My initial thought was ‘I need fire’. Since 2018.1 is the release that includes Shader Graph, it felt like the time had come for me to finally make my first shader.

I was going in pretty much completely blind. I just started adding nodes.

I’m not sure why I thought the Hue node would be relevant at all, but it turned out pretty fun. It's not included in the gif, but the Time node is definitely involved here.

I came back to this “Disco Fire” later. I didn’t want to get distracted before I even started.

Time and noise were definitely relevant, but I just didn’t really know how. Most of these connections are definitely irrelevant, but I wanted to show what I actually made.

So this sort of resembles fire, but it’s definitely not great.

At this point, I went back to the Disco Fire, because it was way more fun. This ended up being used on the golf ball when the speed boost is applied, so I was pretty happy with that one and moved on.

I don’t know why I didn’t include all the nodes for this one :( sorry. It's meant to be lava-ish.

I guess with this one I was going for like a… moss… thing? Idk. shaders, man. The checkerboard node was because I wanted to input more than one colour, and the Voronoi noise was because I wanted it to be sort of blobby (like lava).

This one is definitely my favourite, and I actually started to kind of understand what I was doing. As you can see, there are a lot fewer pointless noodles, and the final look is a lot closer to the lava effect I was trying to achieve. I ended up using this one for the slow ball pickup because it made me think of stone.

Here’s each of them before I added shadows. I was backporting these shaders to a Unity project that wasn’t using the Lightweight Render Pipeline, so I had to make a few changes to the shader file (I don’t want to go into this in this non-tutorial, but happy to make a separate post showing what I did, if anyone wants.)

Aaaand here's my lil test scene showing the fast ball in action (with shadows!)

So that was my experience with using Shader Graph to make my first ever shader! Hopefully this will encourage more people to try using it because it's really not as scary as it looks.