Sound Moves

3D-Audio-Driven Interactions for Audio Games

Sound Moves: 3D audio driven interactions for mobile games

Sound Moves is an investigation into 3D sound-driven audio gaming, supported by the Royal National Institute of Blind People (RNIB). Sound is traditionally a secondary element in interaction design, used mainly for feedback, notifications, progress tracking, and to support visuals in creating mood, tone, and immersion. As modern interfaces become more diverse and context-sensitive, sound is bound to gain new relevance.

Exploration

How does sound shape the way we move? Series of games and workshops using sound a driver of interaction

Final Concept: Interaction with 3D Sound Objects in Audio Mobile Games

In gaming, sound being the primary driver of interaction makes sense. It encourages us to pay attention to the subtleties of a medium ideally suited for storytelling, immersion, and emotional engagement. It also serves the 100 million visually impaired gamers in the UK alone, who are rarely fully included in the design of games.

Interaction with 3D audio is underexplored in current mobile audio games, so I designed a new mode and interaction types, allowing the player to interact with sound objects in an anchored 3D sound space. The concept was notably tested by Ben Breen, Lead Game Accessibility Consultant at the Royal National Institute of Blind People (RNIB).

The Tech

The Interaction Toolkit

Five Interactions

1. Inventory Checking
   Players can access virtual objects by moving their phone around their belt, as if selecting from a holster. Each retrieved object is represented by its characteristic sound.
2. Weapon Use & Switching
   Players can activate and use weapons with immersive, spatialised auditory feedback. A quick double-tap gesture switches between weapons.
3. Audio Progress Bar
   To indicate progress during ads or dialogue, the bass component of the audio is spatialised from left (just starting) to right (nearly finished), giving players an intuitive sense of time remaining.
4. Grabbing a Sound
   Players can “grab” a sound by pointing their phone toward it and long-pressing the screen. Once grabbed, the sound follows their arm movements.
5. Throwing a Sound
   After grabbing a sound, players can release it by making a throwing motion — thrusting their arm in the desired direction to project the sound away.

Demo: 2 Audio Games

With the most successful interactions of the toolkit, I designed two games: GrowlAttack, a classic zombie shooter with 4 different levels, and Shush, an RPG where the player needs to find their ‘magazines’ in their sleeping baby sibling’s room without waking it up. The player needs to interact with the ‘good’ and ‘bad’ fairies of slumber, who are trying to sabotage the player’s mission.

The audio in the recordings will not be spatialised properly for you, because it is recorded from the pov of the player. Note that they never look at a screen and orient themselves using sound only!

Exhibitions: Sound Moves - Arcade Style

Sound Moves was exhibited as an arcade-style game. The demo showcased 3D sound source localisation through five hip-level buttons arranged in a circle around the player (one at the front center, two at the front sides, and two at the back sides), which players pressed in response to the directions of a whimsical ‘grandpa fairy’ character guiding the experience.