Apple Patent reveals their Work on a Spatial – 3D Audio Engine that will take VR Gaming to the Next Level

Today the US Patent & Trademark Office published yet another very detailed patent application relating to Spatial Audio. During June’s WWDC-20 event we learned that Apple will be bringing “Spatial Audio” to Apple’s AirPods Pro.

In today’s patent application from Apple we learn about spatial audio from a different perspective. Today’s Spatial Audio or 3D Audio patent is geared towards developers using Apple’s new “Audio Engine” designed to deliver excitingly realistic audio in future games, especially VR games where gamers are immersed in simulated environments where rushing-water or waterfalls are able to pop out in a whole new audio dimension. 

3D Audio Rendering

Patently Apple posted several patent reports on the topic of “Spatial Audio” over the last few months as presented below:  

01: Apple files new ‘Spatial Audio’ patent for its future HMD and is likely to apply to AirPods Pro and Next-Gen Apple TV

02: Apple Takes Binaural Audio to the next-level by adding Head-Tracking for Next-Gen AirPods Pro and Future HMD

03:  Apple’s Spatial Audio File Format Revealed in new Patent Filing prior to Spatial Audio debuting on AirPods Pro

Today’s spatial audio patent is a little different from most on this topic, as it appears to be aimed mainly at tools for developers. More specifically, an “Audio Engine” for gaming developers who are designing Simulated Reality games that could be AR/MR/VR based.

Further, Apple notes in patent point #75: “In one or more embodiments, a graphic rendering module is a software program (application) that allows a developer of higher layer applications (e.g., games) to define a spatial representation of the objects, called out in the higher layer application, in a graphical scene (and is responsible for rendering or drawing the visual aspects of 3D or 2D graphic objects in the virtual environment that is being displayed (projected.)

In one or more embodiments, such a framework may include geometry objects that represent a piece of geometry in the scene, camera objects that represent points of view, and light objects that represent light sources. The graphic rendering module may include a rendering API like Direct3D, OpenGL or others that have a software abstraction layer for the GPU.”

Apple further notes that “an audio engine” is provided for acoustically rendering a three-dimensional virtual environment. The audio engine uses geometric volumes to represent sound sources and any sound occluders. A volumetric response is generated based on sound projected from a volumetric sound source to a listener, taking into consideration any volumetric occluders in-between. The audio engine also provides for modification of a level of detail of sound over time based on distance between a listener and a sound source.

Apple’s patent FIG. 2 below illustrates a representational view for explaining an example 3D virtual audio environment; FIG. 3A illustrates representational views for explaining example objects having geometric volumes; FIG. 3B illustrates a representational view for explaining an example object having simplified bounding volumes.

Apple’s patent FIG. 7 above illustrates an example system for performing 3D audio rendering.

In the example of Apple’s patent FIG. 2 above, the virtual scene in a game includes two listeners (e.g., listener #212A, listener #212B) and two objects that may be displayed, namely a river (#210) and a house (#215).

The River is an object that is a sound source, which is represented volumetrically in the 3D virtual environment. Consequently, instead of being configured to produce sound from one or more discrete points that constitute the object (as in FIG. 1), the river is said to produce sound from the surface of the entire volume of the shape that represents it. The house is a sound occluding object that occludes sounds from the river to the listener (#212A). The house is also represented volumetrically in the 3D virtual environment.

However, a sound producing object such as a river may also have a more complicated shape. FIG. 3B illustrates an example of a river have a winding path. In one aspect, several simplified bounding volumes (e.g., 36, 37, 38, 39) are determined that as a whole or together represent the geometric volume of a river. Although the volumes 36-39 are not shown as overlapping in FIG. 2, these volumes may in some instances overlap.

With AirPods Pro building in Spatial Audio, future gaming will sound more realistic than ever imagined and only get better when Apple delivers their Video Headset in the not-too-distant future. 

This is a very detailed patent that developer-types could be interested in. For more details, review Apple’s patent application 20200296533.  Considering that this is a patent application, the timing of such a developer tool to market is unknown at this time.  With that said, it’s very possible that Apple may have revealed details of their new “3D audio engine” to developers at this year’s WWDC-20 considering that they announced Spatial Audio for AirPods Pro. 

Article Source: https://www.patentlyapple.com/patently-apple/2020/09/apple-patent-reveals-their-work-on-a-spatial-3d-audio-engine-that-will-take-vr-gaming-to-the-next-level.html