On a crisp afternoon late last year, I made my way to Manhattan’s Upper East Side to meet Edgar Choueiri, a professor of mechanical and aerospace engineering at Princeton University. Choueiri also heads the school’s 3D Audio and Applied Acoustics lab, and over the last decade, he has dedicated his time to the development, application, and refinement of binaural recording systems — a century-old method of audio recording that captures lifelike 3D audio in picture-perfect fidelity.
In his warmly lit apartment, a pair of speakers loomed over me as I sank into a leather couch in the living room. Sitting in an adjacent chair, Choueiri swiped through his iPad to set up a binaural audio demonstration. Moments later, Amber Rubarth’s folksy voice filled the room as she sang an acoustic rendition of Louis Armstrong’s “A Kiss to Build a Dream On.” The room transformed into an intimate live music venue and the soundscape was vivid: Rubarth was positioned right in front of me, with her band members on either side. A cellist on the right plucked on the strings of the instrument. A couple of soulful verses later, a violinist on the left grabbed my attention. I instinctively turned to look in that direction. If only for a few split seconds, my brain was tricked into believing the performance was actually live, and not recorded.
“You can hear a bird flying over your head. You’ll hear a whisper in one ear.”
“It puts you in the exact sound field as originally intended,” says Choueiri of binaural recordings. “You can hear a bird flying over your head. You’ll hear a whisper in one ear. And if you record a band, you’ll hear it exactly as the band was positioned when playing.”
For decades, binaural recording was a novelty, and overlooked for less technically demanding methods. But with the rise of virtual reality hardware like the Oculus Rift, Sony’s Morpheus, and Samsung’s Gear — systems dependent on realistic 3D audio to fully immerse their users — binaural audio is on the cusp of a renaissance.
Binaural recording systems are unique because they emulate the workings of the human head. The architecture of our anatomy dictates how we understand the sounds we hear: with an ear on either side of a thick skull and spongy brain, we hear sounds enter our left and right ears at different times. If a dog barks by our left ear, it takes a few extra microseconds for the bark to reach the right ear; the sound will also be louder in one ear than the other. In addition, sound waves interact with the physical constitution of the listener — the pinna (or outer ear), the head, and the torso — and the surrounding space, creating listener-specific variations otherwise known as head-related transfer function. The brain scrutinizes these miniscule interaural differences of time and strength in order to localize sound with immaculate precision.
Experience binaural audio yourself
Traditionally, recordings have been made using two methods: mono and stereo. Mono uses a single microphone to pick up sound, while stereo uses two, spaced apart from each other. Binaural recording takes the stereo method one step further by placing two microphones in ear-like cavities on either side of a stand or dummy head. Because the dummy head recreates the density and shape of a human head, these microphones capture and process sound exactly as it would be heard by human ears, preserving interaural cues. The effect is best experienced over headphones, with a clear distinction between left and right perspectives. It’s a simple concept, but when done with high-quality microphones and played over equally high-quality headphones, the effect is eerie: it fools the brain into believing it’s hearing the sounds firsthand.
The illusion creates three-dimensional audio, which is not to be confused with surround sound. Interchanging those terms is the quickest way to amuse — and sometimes annoy — an acoustician. A surround sound system employs multiple speakers to create a 360-degree field around the listener. Say you’re watching a film presented in surround sound: you’ll see an explosion on one side of the screen, and a speaker on the corresponding side of the theater will emit a barrage of sounds. A theater equipped with Dolby Atmos, the company’s most recent iteration of the technology, effectively envelops the audience. But by recreating exactly what your ear hears in any given scenario, binaural audio accomplishes a far more natural 3D sound without a costly armada of speakers.
The technique at the heart of binaural audio can be traced back to Clement Ader, a 19th-century French engineer. In 1881, Ader devised the Theatrophone, a telephonic system of transmission to broadcast a Paris Opera show. Pairs of microphones were systematically spaced in front of the stage, covering the breadth from left to right. Signals from the show were then transmitted via telephone receivers to listeners on the other end. With a pair of receivers, one mounted on each ear, listeners could hear the show from their designated suites at the gallery of Palais de l’Industrie.
In 1933, AT&T Bell Laboratories brought binaural audio to the Chicago World’s Fair. The acoustics research department of the company created a mechanical dummy, which it named Oscar, with microphones placed on its cheeks in front its ears. Oscar sat in a glass room capturing sounds while visitors gathered outside used headphones to hear exactly what the dummy heard. The technique revised the experience introduced by Ader, but both inventions offered poor sound quality.
Through World War II and the decades that followed, progress in binaural faced significant obstacles: primitive techniques failed to achieve accurate, high-fidelity recordings. But in 1973, Neumann, a renowned German microphone company, introduced the breakthrough KU-80, a prototype binaural recording device. Neumann’s iteration consisted of a detached dummy head with microphones placed in the eardrums – the position captured cues with more precision than any of its predecessors. Three generations of dummy heads later, the KU-100, introduced in 1992, featured omnidirectional microphones, expertly preserving the spatial cues and the overall quality of sound. It continues to be the go-to dummy head for binaural recordings.
Binaural audio found a home in select audio recording communities: the technology worked well for radio plays and experimental sound art. Meanwhile, musicians like Lou Reed, Thom Yorke, and Imogen Heap have all experimented with binaural audio productions. Most recently, the ASMR (Autonomous Sensory Meridian Response) community has been employing binaural recordings to trigger physical responses that they believe can be soothing and calming.
One of thousands of binaural ASMR YouTube videos promising healing and tension-releasing properties.
Now, almost a century after the demise of the Theatrophones, investors are starting to revisit 3D audio technology: the prototype of Sony’s VR headset Project Morpheus includes a custom 3D audio binaural solution in its development kit. “3D audio adds to the feeling of presence that we strive so hard to achieve with the visuals in VR,” says Richard Marks, senior director of research and development at Sony Computer Entertainment America. “When sound is perceived to come from the same direction as a visual stimulus, the credibility of the virtual experience is greatly increased. While purely visual VR experiences can be made, adding 3D audio greatly magnifies the impact and depth of a VR experience.”
3D audio offers a more expansive experience than its visual counterpart. “Unlike with the visuals, 3D audio is not limited to the field of view of the display and can be rendered to give a ‘complete 360-degree’ experience,” says Marks. “One of the biggest challenges for VR design is that the user can look in any direction, and may not even be looking when something momentous occurs. But using a 3D audio cue, it is possible to steer the user’s attention to look in the direction of the sound, similar to techniques that are used in live theater.”
Video director Chris Milk — who’s previously done work for Kanye West, U2, and Arcade Fire — has harnessed the immersive capabilities of binaural recording to bring concerts to the virtual arena. Last year, he produced “Hello, Again,” a revolutionary VR film of a Beck concert. To match the 360-degree visual experience, Milk developed a binaural system to produce 360-directional sound. “If we were going to let you look in every direction, a one-directional sound source would not work,” he says. “You would look behind but you would still hear as if you were looking in front of you. So we needed something that would dynamically change based on where you were looking in the visual.”
He ingeniously devised a system that used an omni-binaural head with eight ears, with four pairs of 3Dio’s Free Space microphones implanted at every 90 degrees. He followed it up with a software that would process the sound from those ears to provide the effect necessary. Viewed with a VR headset, the binaural sound on the headphones matched the visual illusion of the Beck video every step of the way.
Jean-Pascal Beaudoin, the director of Apollo Studios, stresses the importance of 3D audio for VR experiences. “If a sound is clearly coming from your left, and you turn your head 90 degrees to look in that direction, you now expect that sound to be located right in front of you,” says Beaudoin. “If it isn’t, and it’s still at your left, chances are you’ll be pulled out of the experience before you even realize it.”
Beaudoin provides 3D sound for VR experiences produced by Felix & Paul Studios, the VR production company commissioned by Samsung Gear to produce a series of 360-degree live action videos. “I am most interested in the creative possibilities that 3D audio opens up in terms of storytelling, emotions, and interactivity,” says Beaudoin. “We are barely scratching the surface of the merging VR medium, but with each project, we make progress in establishing a new grammar.”
Despite its potential, binaural audio is not widespread within the VR content world yet. “Audio sometimes takes a back seat in content production,” says Adam Somers, lead audio engineer at Jaunt, a Palo Alto-based developer and one-stop shop for capturing and creating cinematic virtual reality. “So much work goes into visual elements, whether it’s a game or film, that sometimes audio is an afterthought. But talking about VR as a medium for content, it’s really important that the medium has the capability of binaural audio built in from the beginning to make it easier for content creators to utilize binaural audio.” Jaunt ensures the presence of binaural audio in every piece of content created through their platform.
Back in Manhattan, Choueiri is considering another problem: since the inception of the technology, binaural audio has been reserved for headphone listening. But Choueiri wants to make the technology accessible over external speaker systems for a wider audience. The challenge is that with speakers, a right ear not only hears its respective cues, but also picks up information meant for the left ear. “It messes up the cues, so instead of hearing 3D sound, the brain just locates the speakers,” Choueiri said. “It’s like watching 3D movies without the glasses on.”
Slowly but surely, binaural is becoming a linchpin in virtual reality development
For decades, this confusing crosstalk between speakers has perplexed the audio community. But Choueiri’s BACCH SP, a filter that enables a pair of speakers to retain the aural cues, creates the illusion of 3D audio for the listeners. Jawbone has employed Princeton University’s algorithm over the last two years to create the LiveAudio filter for its wireless bluetooth speaker, Jambox. Loading the mini-speaker with the digital filter optimizes audio to create a three-dimensional experience. While effective, the experience is limited to a sweet spot — the device needs to be centered in relation to the listener. The illusion instantly collapses when the listener moves from the spot. Choueiri says a version of that software, the BACCH-dSP app, coupled with a head-tracking feature, can sustain the illusion irrespective of the listener’s head movements. That app is scheduled to show up in the store for Mac OS soon, bringing 3D audio experiences to laptops.
Slowly but surely, binaural is becoming a linchpin in virtual reality development. Oculus’ most recent prototype, Crescent Bay, unveiled at CES last month, integrates binaural technology with Rift’s head tracking for complete audio-visual immersion. And while Sony’s Project Morpheus hasn’t announced final specifications of the product yet, their emphasis on 3D audio is evident. As Adam Somers of Jaunt put it, “Binaural audio is critical to an immersive experience within the context of VR. We consider audio to be 50 percent of the immersive experience.”
