Gameplay in HTML5: Audio

Homework #7 review

Instructor's solution

http://EpsilonDelta.us/UW_GameplayHTML5/Homework/Solutions/7

Audio in HTML5

Overview

With HTML5, apps can play and control sounds. Games can play sound effects, background music, and more. Support still varies between browsers, but the important functionality is now ready for use.

Formats

Overview

  • Digital audio represents the waveform of sound as numbers representing the variation in amplitude of the waves as sampled many times per second.
  • Audio file formats are distinguished by
    • Sample settings
    • Compression
    • Legal encumbrances

Sample settings

  • Digital audio is characterized by
    • Sampling rate: the number of samples per second.
    • Sample size: The number of bits per sample.
    • Number of channels
  • Sample rate generally ranges from about 8 kHz (8,000 samples per second) for telephone quality to 44.1 kHz for CD audio to 48 kHz or more for high fidelity recording.
    Human hearing ranges from 20 Hz to 20 kHz. The Nyquist-Shannon sampling theorem states that the sampling rate should be twice the highest frequency being sampled.
  • Generally either 8 or 16 bits per sample are used. Larger samples allow for more dynamic range.
  • One channel is monaural sound, and generally best for sound effects.
    Two channels is stereo, typical for music.
    More channels can be used for 5.1 surround sound, etc.
Most audio file formats are compatible with a range of sample rates and multiple channels.

Compression

Sound files are also characterized by the compression used.
  • No compression. Also called PCM (pulse-code modulation). Easiest to manipulate.
  • Lossless compression. When decompressed all of the original (PCM) data is unaltered.
  • ADPCM (adaptive pulse-code modulation). About 3.5:1 compression with some loss. Fast compression and decompression.
  • Lossy compression. Sophisticated algorithms can reduce data size significantly with minimal reduction in audio quality.

Legal encumbrances

Certain file formats are open-source and free for anyone to use. Others are encumbered by software patents and sometimes uncertainty regarding ownership of rights, generally regarding compression algorithms.

Some important audio file formats

Format Compression Legal Browsers Notes
WAV Uncompressed (PCM) Open Chr, FF, Saf, Op, iOS Easy to process
FLAC Lossless Open Good for archiving
MP3 Lossy Several patents until 2015+ Chr, IE, Saf, iOS, And Supported on many devices
AAC (M4A) Lossy Patents on codec Chr, IE, Saf, iOS Successor to MP3. YouTube, iTunes, etc.
Ogg Vorbis Lossy Open Chr, FF, Op Ogg is container format. Vorbis is audio codec.
WebM Vorbis Lossy Open Chr, FF, Op Audio-video container
Ogg Opus Lossy Open FF Low latency
Chr=Chrome, FF=FireFox, Op=Opera, IE=Internet Explorer, Saf=Safari, And=Android
No single format is supported by all current browsers, but all of the modern ones support either Ogg Vorbis or MP3.

Codec comparision

The IETF working group developing the Opus codec has published this comparison based on listening tests:

Detecting format support

We can ask whether a particular format is supported:
var audio = new Audio(),
    mp3Support = audio.canPlayType( 'audio/mpeg' );
            
The argument is the MIME type of the format.
For containers like Ogg we should also specify the codec(s):
var oggVorbisSupport = audio.canPlayType( 'audio/ogg; codecs="vorbis"' );
            
Modernizr simplifies these tests for ogg, mp3, m4a, and wav. E.g.:
var oggVorbisSupport = Modernizr.audio.ogg;
            
The value of the variable is always one of these, in decreasing order of likelihood of support:
  • "probably"
  • "maybe"
  • "" (empty string)

Example

This will work if our browser "probably" supports Ogg Vorbis or MP3:
var formatExt;
var exts = [ "ogg", "mp3" ],
    i;
for ( i = 0; i < 2; ++i )
    if ( Modernizr.audio[ exts[i] ] === "probably" )
    {
        formatExt = exts[i];
        console.log( formatExt + ' is supported, "probably".' );
        break;
    }
            

If that fails, we might check for (and supply) other formats and/or check for "maybe" support.

Creating audio elements

Directly in HTML

You can declare an audio element directly in your HTML like this:
<audio>
  <source src="audio/MySound.ogg"  type='audio/ogg; codecs="vorbis"'>
  <source src="audio/MySound.mp3"  type='audio/mpeg'>
</audio>
            
This allows the browser to choose the file it supports. You can also add a controls attribute to the audio element which will place visible playback controls on the Web page.

In JavaScript

For games we usually create audio element objects in JavaScript:
var myaudio = new Audio( "audio/MySound." + formatExt );              
            
Unless we want to use the browser controls, we usually do not need to append this to any DOM element.

Controlling audio

Play

The play() method starts playing the sound referenced by the audio object.
var snd = sounds[ "computer" ];
snd.play( );
            

Pause

The pause() method stops playback, which may be resumed with play().
var snd = sounds[ "asaying" ];
snd.play( );
setTimeout( function() { snd.pause( ); }, 4*1000 );
setTimeout( function() { snd.play( ); }, 10*1000 );
            

(This code is just to exercise the methods. The timing approach is not the best, as it depends on the the initial play() call working quickly.)

Looping

Setting the loop property causes a sound to be repeated (endlessly).
var snd = sounds[ "hatecomp" ];
snd.loop = true;
snd.play( );
setTimeout( function() { snd.loop = false; }, 7*1000 );
            

Current time

The currentTime property is the current playback location, measured in seconds. It can be set to skip to ahead (or back) to any point in the sound stream.
Here we play the final measures of Diane Nalini's stellar Kiss Me Like That.
var snd = sounds[ "Diane_Nalini_Kiss_me_like_that" ];
snd.currentTime = 208.4;
snd.play( );
            

Volume

The volume property ranges from 0.0 to 1.0. (No, you can't make it 1.1!)
var snd = sounds[ "maketenlouder" ];
snd.play( );
setTimeout(
    function()
    {
        snd.volume = 0.25;
        snd.play( );
    },
    10*1000 );
            

Muting

The muted property is a boolean which, when true, silences the sound without pausing playback and without affecting the volume property.
var snd = sounds[ "Ella_Fitzgerald_Let's_Do_It_(Let's_Fall_in_Love)" ];
snd.currentTime = 12.0;
snd.play( );
setTimeout(
    function()
    {
        snd.muted = true;
    },
    6.0*1000 );
setTimeout(
    function()
    {
        snd.muted = false;
    },
    12.1*1000 );
setTimeout(
    function()
    {
        snd.pause( );
    },
    21.5*1000 );
            

Monitoring audio

Overview

Audio elements have many properties which can be read to determine the state of loading and playback. They also fire many events which you can listen for.
We have already discussed the loop, currentTime, volume, and muted properties, which we generally set more than we get.

ended

It is usually necessary to keep track of which sounds are playing. We generally know when a sound has started playing, because our program calls play(), but we need to know which sounds have finished playing. HTML5 audio offers both an ended property and an ended method.
var snd = sounds[ "fw11" ];
snd.play( );
for ( var i = 0; i < 7; ++i )
{
    setTimeout( function()
                {
                    console.log( "Ended: " + snd.ended );
                },
                i*1000 );
}
$(snd).on( "ended",
        function( event )
        {
            console.log( "The sound (" + snd.src + ") just ended" );
            $(snd).off( "ended" );
        } );
            

More

Other useful properties include
  • duration (seconds). (May not be known before play begins.)
  • readyState: a number (0-4) indicating how much of the required data the browser has received
  • seekable: a TimeRanges object
Other useful events include
  • canplaythrough: when the browser estimates it can play to the end without running out of data.
  • timeupdate: fired during playback, roughly every 0.25 second.

Limitations

  • This is a new API and not fully specified or implemented yet.
  • As noted, no standard, universally supported format.
  • Seeking (with currentTime can be unreliable.
  • Latency, especially across the Internet, can be significant.
  • Mobile browsers have serious limitations with audio, such as:
    • iOS cannot play multiple sounds simultaneously!
    • Neither Android nor iOS offer volume control.
    • Android 2.x and iOS (?) do not always fire the ended event, so you may need to use the pause event and check the currentTime.
  • Advanced audio API's, to allow synchronization of media elements, access to sample data, filtering and effects, etc., are still being developed. (The W3C Working Draft references two proposals, one from Google and another from Mozilla.)

Homework #8

http://EpsilonDelta.us/UW_GameplayHTML5/Homework/Homework_8.html