Communicating with sound
Sound and music are great for communicating strong feelings and create reactions in digital experiences. Use it to form a setting, to redirect focus or to conjure up a certain emotional state. If you change the sound, or music, you can completely transform the context and the effect on the audience.
Sound and space
Sound is very much connected to the space – it travels through and interacts with the surroundings. The reflections of the room make a big difference to how we hear it. It is important to remember this when simulating sound. Sound is an essential, useful and powerful tool to program the listener towards one or another state of being. Music and sound have the ability to take the listener from one place to another.
- Use sound and music to enhance digital experiences.
- A person can guess the size of a space based on the sound they hear.
Interpreting sound
Did you know that certain sounds create a similar response in pretty much all humans? Whereas other sounds are connected to culture and affects how we interpret it. On an individual level, the state that a person is in will affect what they hear. Art is created when the individual creates it in their own brain – their own reality.
Stereo
The human ears are used to stereo sound – as we have two ears. Having two sound sources, channels, equals stereo.
It is enough for you to have two speakers to imitate the movements from left to right for the ears within our field of view. This is important to understand because humans are used to pointing their nose to the direction of where the “important” sound is coming from.
Working with sound/music professionals
As you know, communication is really important. When planning and working on your idea, you need to know which experts you should talk to and how to communicate with them. This is vital for a successful project. You must understand each other’s vocabulary. If something is unclear to you, ask! It is better to ask too many questions than not asking at all.
Immersive sound
Binaural audio
What is binaural? The literal meaning of the word is to have two ears. Binaural hearing allows humans to determine the direction and origin of a sound. Binaural audio, two channels, is a way of recording sound that reproduces the sound we would hear in the real world. A binaural microphone, basically a dummy head that imitates the human head, has silicone ears that imitates the consistency of the human ear. The ears have microphone capsules inside. They imitate how sound is perceived or get into our brain. And with that it is possible to play it back. If we play the sound back through headphones that are near our ears, it is enough just to have stereo – two channels, left and right, to try to recreate the sonic reality that is around us. Playing simple stereo will not simulate left, right and distance – back and forth.
How we hear sound
In real life if a sound comes from the left side, it will reach our right ear a little after the left ear, as that ear is further away. The sound goes directly into the ear that is aimed at the sound source, the other ear doesn’t get that direct sound and the sound is affected by reflections in the space. All these kinds of things direct sound and reverberations, reflections, the sound of the room or space we are in helps us locate the sound and help us simulate it.
The shape of our head and ears
Another thing that affects how we hear is the shape of our head and the form of our ears. Each person is built slightly differently. Binaural algorithms, decoders or approaches that are used in sound software are dependent on a kind of standard model of the head. That means that binaural might work better for some people than others. Some people might more precisely hear that something is moving around their head.
- Two channels, close representation of natural spatial listening.
- Best experienced through headphones.
Ambisonic sound
In immersive sound there is something called ambisonics – it is like a complete virtual 3D sound space, sound synthesize. You are trying to synthesize the sound field that is an everyday thing we have around our head all the time. Higher order ambisonics is what is used today. It has a more precise resolution.
Sound field
A sound field is a concept that is quite simple. The idea of a sound field is that we always have a sphere of sounds around our head. A sphere of sound means that the sounds can be up, down, left, right – the full 360. Consider all the sounds that our ears receive in a day. The sound hits you from all directions. Some sound sources are far away, some are close, some of them are loud but far away, others quiet and close.
Ambisonics tries to simulate that up, down, backward, forward, side to side in relation to the listener that is in the middle. It is a surround sound that is fully spherical. Ambisonics recreates/synthesize something that is real. Much like music synthesizers which are sound generators. In this case we are trying to synthesize space. Or convert it, with the help of technology.
Ambisonics in VR
Ambisonics was adopted by video applications and virtual reality. It makes sense. If you have a full 360 picture, you want 360 sounds. To get really good sound in a VR experience, the user needs to use headphones or be in a place where there are multiple speakers set up for an ambisonic sound – directional or spotlight speakers. Some headphones have head trackers that allow for greater immersion.
Recording ambisonic sound
Ambisonics is a method for recording, mixing and playing back three-dimensional 360-degree audio. To simulate, record, convert with microphones in the soundscape with different sound sources, position of sound in space. Surround or sound field microphones have capsules pointing in different directions, they are arranged in a way that they record everything that is around us. Then by combining the signals from the capsules they make this virtual ideal space as if that capsule would be in one point of space.
- Ambisonic 360 sound is like real life sound.
- Ambisonics, 4+ channels, can be decoded for speakers or for use in headphones.
Degrees of freedom
The degrees of freedom for the sound scape is really important when conceptualizing artistic works. How will your audience take part of your piece – are they sitting in a fixed chair, pointing their head in one direction? Will they be able to move their head and spin around in the chair. Or are they free to move around in the space. If they are, how can and will they do that.
Plan your sound
Don’t leave the sound until the end. Plan it early in the process because it has a huge effect on the audience. You should make a conscious choice whether your creation should be a soundscape of 3 degrees of freedom (DoF) or 6 DoF.
Creating digital artistic work
There are three aspects to creating a digital experience:
- the artistic idea.
- the techniques; artistic techniques implemented with/within. instrumental/technological possibilities and limitations.
- the instruments; tools and technology.
You have an artistic idea, you have the instruments (tools and technology), and then you have the techniques that connect it all. To create impactful digital creations, you need to develop strong artistic ideas. Find the relevant and effective tools, available to you, to implement your artistic idea. The best ideas usually come from something that you are really interested in and energized by.
Use sound to enhance your digital experience
Reflect on what type of sound scape you want for your piece in the early stages of your creative process. Then start experimenting and testing different ideas.
Consider if your sound should move very slow, very fast or maybe something in between? Do you want it to move gradually or should it jump? The more it jumps, the faster it moves and the audience’s attention becomes scattered rather than focused. Play around with distance; is it close, far, localized or is it impossible to tell where the sound is coming from. Or should you use an ambient sound that is everywhere.
Sound can tell a viewer that something is far off in the distance, or that it is very close. Plan your sound to re-direct focus or make a user react in a certain way. It will allow you to create feelings of happiness, fear and wonder. It can also make a viewer react physically. Bird noises closing in fast can make a viewer duck down.
Sound priority
Certain sounds take priority over other sounds. If the everyday sounds in a neighborhood are interrupted by a blaring car horn, sudden screams or a siren (loudness, warning signals) our ears will immediately prioritize that sound.
So, play around with the sound and utilize it to make the audience get immersed in the experience. Remember, total silence can also be a way of working with sound. No sound can be either very peaceful or disconcerting.
Recording realistic soundscapes
In some cases, if you want to create realistic soundscapes – streets, jungle or ocean somewhere in the background, you can take mono or stereo and position them in space and create a soundscape on your own. You can use the same principle to create musical compositions where the sound is not just in the front but everywhere. It makes for quite a realistic setting. And it can create a sense of distance, type of room or space. The speakers are programmed into the software so you control what sound and loudness that comes out of them.
Good sound for the viewer
To get really good sound for a VR experience the viewer needs to use headphones or be in a place where there are multiple speakers set up for ambisonic sound. Directional och spotlight speakers.
Download the Checklist for sound (pdf)
Download the Terminology sheet for sound (pdf)
Lecturer
Mantautas Krukauskas is a composer and sound artist, Associated Professor at the Department of Composition of Lithuanian Academy of Music and Theater in Vilnius, where he is also a co-founder and Head (since 2016) of Music Innovation Studies Centre, academic lab for studies, art and research, with a focus on music technology, innovation in music and music education, interactive arts, immersive media, and interdisciplinarity. His compositions, including chamber music, sound art and other works, music for theatre and dance productions have been performed in Lithuania, Austria, Germany, France, Canada, USA, and other countries. Professional profile also includes electronic music performance and work within the creative industries sphere with music production and arrangement.
linkedin.com/in/mantautas-krukauskas
