AudoVox
How can I make a player feel like a producer? I started off a few years ago making music and producing. I became fascinated with learning how to compose, make patterns, mix and more. I loved music with a passion and wanted to extend that into my thesis project without the user having to know music theory, since I actually do not understand music theory myself. I saw making music as more of a game than a theory. After some prototyping of actual games, I decided that it wouldn’t be a game at all. I believe that's also why this “experience” I have created is important.
Video:
Goals:
The goal of this project was to make it like a museum experience. Since this was less of a game itself, I wanted users to be able to play at all experience levels of games and music. Something kids can play and mess around with since it was that easy to do. With no goal in this game and scenes that can be switched, the harder part was making this a game that anyone can walk up to and understand with little instruction. If I had to narrow my audience down, It would be for non musicians, people who don’t understand music theory to be able to understand music a little bit more. With everything on the right key, with no goal of even rhythm, any age of any level can play this game.
Research:
My research explores how modular synthesizers and dynamic soundscapes encourage creative expression. I desire to develop innovative tools for changing the way people interact with sounds. Sounds that are given to users and sounds that users can create themselves, possibly with their own voices. My overall question ranges from the accessibility of modular synth system designs to the integration of audio-visual feedback and then gamification of sound creation. This research investigates ways in which physical and digital tools can make the manipulation of sound intuitive and captivating for a wide range of users.
Precedents:
I had this idea at first of having the game as a phone and tv game like Jackbox. I liked the multiplayer aspect of jackbox and how it lets the users interact to create the game. Some of my precedents at first were Incredibox, Singing Monsters, and Tenori-On. Something I began to notice when I was looking at my precedents was that all these “games” did not require the users to have any music talent, theory, or even rhythm. I knew from the beginning that I did not want a rhythm game due to the fact that I do not enjoy rhythm games. Sometimes they are also very difficult for nongamers or non musicians to do and I wanted the opposite of that. These were some of the ideas I had for my game at first however my precedents did not change over time working on my thesis. I just decided to add more to the list. Later I added Mu Cartographer, which was recommended to me by a professor that really shifted my perspective on how I wanted to shape the visual look of my game.
Iteration and Development:
My first prototype ironically was more like my final than my second prototype. I was just learning some of the Unity audio code like changing volume and using the audio mixing board and being able to change the mix of a sound dynamically which is code that I end up using later in my finalized version. Users can turn on and off loops that include a synth, drums, and 808s. While also being able to change the distortion and reverb of all three individually. Allowing the play testers to be able to experience what these effects actually do by isolating the sounds. Essentially teaching players what these words, “distortion” and “reverb” mean.
However in my second prototype I decided to change up and make a 2D pixel platformer that relied on sound and design. After making a very quick prototype for movement and the goal of obtaining keys. I presented my idea to the class and another professor for my midterm. However I was told that making a 2D platformer was only limiting my idea of using sound and design. That's when I discovered Mu Cartographer as a precedent. I then saw a different goal in mind.
Visualizing the Goal:
During my break I had some ideas of where I wanted to take my game since it was a little bit different I had to look at my options. Do I want a direction with my game or not? Do I want players to get to an end goal? Did I want players to make a song? Or do I just need them to feel like a producer and have them experience music production changing and altering sound and visuals? After heavy thinking I thought it would be different to have the players walk through some different scenes and there will be some game playing but not really with an intended purpose. Many other games will have an end goal, however I wanted to stand out with something different than usual. Something unusual.
My goal for the game at the moment was to create two scenes, one with a synth generator and the other is a version of break out, however every brick plays a note that is in scale, by using the mixers the scale changes and so does the reverb. Along with the color of the background and size of the particle.
Musical Breakout Scene:
I worked hard on getting the sound in the game to work properly. I first built the game in 2D. Getting the particles to work and the sound. I figured it would be a bit easier to work in 2D, then move to 3D just so I could have a game ready for play test Which is what happened. I ended up having some technical difficulties right before the play-test. It was no problem though, I just showed the 2D version.
Eventually I was able to get the 3D version up and running to playtest at an NYU playtest. I had small problems with coroutines and some of the materials color changing too. From some of the feedback I was able to add multi balls. After further playtesting , I wanted the knobs to not just change the visual and the sound, but the game itself, So just players had more of an incentive to turn the knobs on the side while playing. I wanted to change the size of the ball with a slider however I didn’t believe that was in the best interest of the game since then players could just make the ball as big as possible and play the game way easier. Then I had an idea of changing the speed of the ball as well so that players couldn’t just make the ball big the whole time. I showed this to play testers and it seemed to be a good idea.
Synth Creation Scene:
The synth creation scene was a quick scene I created using the Unity line renderer to change the shape of the line while listening to the sound of the synth you’re creating. The Synth was just different mathematical formulas. What was interesting though is how I got these formulas to blend together.
Firework Scene:
I wanted to work on one more scene before my Spring Break started. This scene was a firework scene that I had some ideas for. However I just wanted to first start with building the fireworks. After some time I wasn’t really pleased with the firework particles I was creating myself. So I ended up finding an asset on unity that uses URP(Universal Render Pipeline). I imported the asset and worked with it to change the rate of each five particles to go along with bands of frequency.
The harder part was figuring out the code to apply the audio.GetSpectrumData() into multiple audio sources, which I used a foreach loop to do so. Lastly I linked the sliders to the eight different audio loops I created myself in FL Studios, so now players can move up and down sliders to change the rate of fireworks and volume.
While this worked great, I wanted the knobs to do something too. I thought about mixing the audio using reverb or a high pass filter. But then I realized that audio sources have their own pan for each audio. So I decided to add a pan to each knob so you can pan each eight sounds with its corresponding knob. Lastly I needed something to change the visuals according to the average pan of each knob. I found the Post Processing global volume, lens distortion would be perfect. Over Spring break I worked on the last two scenes for my thesis. I wanted two new innovative ways to look at music and sound in games.
Roots Scene:
The first one was based on plant roots. I used a line renderer again to create a moving sin function constantly creating new points and getting rid of points out of the camera's view giving the perception of an infinite loop. Then I created new lines around the existing main line moving. Then destroying these smaller lines after sometime too so that the scene can stay on forever without CPU problems. I decided to go with the same route as the firework scene with each slider corresponding with different audio source loops. However the knobs are going to correspond with mixing the loops, using the audio mixer in Unity, each knob does different things like high pass, low pass, pitch, chorus, reverb and more.
This visual is supposed to be looking at the timbre of music, “the unique sound quality or ‘tone color’ of a musical note, sound, or tone, distinguishing different instruments or voices, even when playing the same note.” So the roots change color based on the tone of the music, and the size changes on the volume. Along with the small roots changing too. Overall the whole formation of the line renderer changes upon the music that's being played. The struggles I had with this scene was figuring out how to listen to the audio after it had been mixed from the audio mixer. Once I figured that out, it was smooth sailing.
Tenori-On Scene:
For the last scene I really wanted to make a scene like the mini game Tenori-On. I created a grid using spheres in an 8 by 8 format. The used notes in the same key on each sphere row and made a for loop that repeated a BPM. This was actually very simple, however the harder part was my drum mode. I wanted to create a separate grid that you could switch to that played drum sounds so you can layer the note and the drum sounds on top of each other. This was very hard to do though and so as an easier way out. I decided to have it on the same grid with users being able to switch between modes with a button on the mixing board. Overall this took me a couple days to get this to work but I finished it right in time for Spring break to be over.
Reiterating the Synth Creation Scene:
I then had to present my game, or what I had so far of it, for midterms presentations. When I showed my game I got feedback to change the synth creation scene as it was probably the weakest scene I had. Overall I agreed and also thought the constant scene sound was kind of annoying to listen to.
Over the next couple of weeks I changed the sound by adding two more synths that you can layer on top of the main scene, and kept the knobs changing the synth’s shape. However I added a drum section on the last four sliders that allows users to not only add drums individually like hi-hats, snares, and kicks, but also add the rate of them. Exp. If the slider maxed out the snare plays every quarter note, if the slider is 2/3s mark the snare plays every eighth , if the slider is at the ⅓ mark the snare plays every 16th note. Sounds complicated. Now next was to change the visual. I found this unity asset called “Limitless Glitch Effects” by Limitless Unity Development. I loved the creative ways you could add glitch effects so I learned how to change the effects in scripts and then changed this visual of old retro TVs falling down on the screen.
Showcase (Thesis Show)
My show went way better than anticipated. I am very thankful for this experience and for the process and people who came through to play.