Also, how great is this: While trying to make this big change to our sound system, audio has broken entirely in Chrome 46 on my laptop 😞 . Bug filed with the Chrome team.

Update on our deprecated check for whether audio is unlocked: I wrote a little diagnostic and tried running it on several platforms to see if I could reveal which approach would work best. To review:

• Approach 1: source.playbackState === source.PLAYING_STATE || source.playbackState === source.FINISHED_STATE used by howler.js and our current implementation.
• Approach 2: s.context.state == "running" used by SoundJS
• Approach 3: context.currentTime > 0 as suggested by the page reporting that playbackState is deprecated.

Here's what I learned:

Browser	onload	ontouch	Notes
iOS9 Safari			All three approaches should work.
iOS8.3 Safari			Approach 2 will not work.
iOS7 Safari			Approach 2 will not work.
iOS6 Safari			Approach 2 will not work.
Android Chrome			Music does not autoplay but audio seems to unlock fine if either button is pressed. Confusing, because approach 1 is giving a false positive, Approach 2 is inconsistent, sometimes gives 'running' and other times gives 'suspended', and Approach 3 appears to work immediately on load.
Android Firefox			Like Android Chrome the music does not autoplay but works fine after touch (maybe through no merit of our own). Approach 1 gives false positive, Approach 2 seems broken, Approach 3 sometimes works (may need a timeout even longer than 30ms before checking).
Desktop Chrome			Audio is unlocked immediately. Approach 1 is giving a lucky true positive, Approaches 2 and 3 seem valid.
Desktop Firefox			Audio is unlocked immediately. Approach 1 is giving a lucky true positive, Approach 2 is inconsistent (sometimes 'running' other times 'suspended'), Approach 3 seems to work if the timeout before checking is increased from 0ms to 30ms.

I was unable to get working Windows phone setup to test with.

On the whole, Approach 1 seems to be valid only for iOS (and who knows for how long, since it's deprecated) while Approach 3 seems the most generally correct it's also a bit finicky and requires the 30ms+ delay to be consistent. It also seems like we might not need any unlocking logic on other platforms - although Android has a similar behavior preventing autoplay on load, it seems to do magic unlocking by itself.

I suggest we first feature-check Approach 1 by seeing if the enums exist and use that if we can (after a 0ms setTimeout). If Approach 1 is not available, we use a 50ms timeout and check using Approach 3.

Here's the implementation of the hybrid unlock check (also just pushed, but written here sans diagnostic steps for clarity):

/**
 * Performs an asynchronous check for whether the given source and context
 * actually played audio.  When finished, calls provided callback passing
 * success/failure as a boolean argument.
 * @param {!AudioBufferSourceNode} source
 * @param {!AudioContext} context
 * @param {!function(boolean)} onComplete
 * @private
 */
Sounds.prototype.checkDidSourcePlay_ = function (source, context, onComplete) {
  // Approach 1: Although AudioBufferSourceNode.playbackState is supposedly
  //             deprecated, it's still the most reliable way to check whether
  //             playback occurred on iOS devices through iOS9, and requires
  //             only a 0ms timeout to work.
  //             We feature-check this approach by seeing if the related enums
  //             exist first.
  if (source.PLAYING_STATE !== undefined && source.FINISHED_STATE !== undefined) {
    setTimeout(function () {
      onComplete(source.playbackState === source.PLAYING_STATE ||
          source.playbackState === source.FINISHED_STATE);
    }.bind(this), 0);
    return;
  }

  // Approach 2: Platforms that have removed playbackState can be checked most
  //             reliably with a longer delay and a check against the
  //             AudioContext.currentTime, which should be greater than the
  //             time passed to source.start() (in this case, zero).
  setTimeout(function () {
    onComplete('number' === typeof context.currentTime && context.currentTime > 0);
  }.bind(this), 50);
};

(Just to clarify, approach #3 in earlier comments is approach #2 in code snippet comment)

LGTM! One small nit Q

Windows Phone (at least the one I tested, a Nokia Lumina 920 Windows Phone 8.1 IEMobile 11.0) is broken in some interesting ways, and not limited to audio.

• My diagnostic box on the audio_test page doesn't show up at all. I'm pretty sure this is because it's using HTML5 <audio> instead of Web Audio and we have not implemented unlock logic for the fallback player.
• On all pages, sound effects play as soon as they are loaded, even if we haven't tried to play them yet. This is not an issue in the desktop versions of IE, so it may be a bug in IEMobile. I'll do some research.
• Audio, and music in particular, took a very long time to load (several seconds on Wi-Fi to my local server over ngrok). It's possible that using HTML5 audio we load the entire file before we begin music playback (whereas Web Audio tells us when enough of the file has been downloaded to expect gapless playback, and we start it right then). When moving quick, this sometimes made it seem like music failed to play when in fact it was still loading.
• Music tracks often stop before they have played all the way through.
• Non-audio issues: We aren't blocking navigation gestures or touch-scrolling like we do on iOS and Android. This makes blockly difficult to use (and droplet frankly impossible) as half the time you will get a page scroll or back/forward navigation instead of or in addition to dragging a block.

Overall I'd describe the experience as "painful, but usable." I don't think it should be in scope for this change to fix up the Windows Phone experience, but it might be worth putting some time into at least for the Minecraft tutorial. I'll do some testing against production to make sure the audio issues above are not a regression, and if not I'd like to merge this initial set of mobile audio improvements.

Testing the phone against production, I can't get any audio to play at all - so even with all its issues, this change might be an improvement to Windows Phone audio.

Investigating the autoplay issue I ran across these two terrifying statements. The first is Microsoft admitting that autoplay is weird on Windows Phone:

"To improve interoperability with other popular browsers, the autoplay property has no effect in Internet Explorer for Windows Phone 8.1 Update."
autoplay attribute - MSDN

The second is Mozilla admitting that autoplay completely ignores its value:

"autoplay
A Boolean attribute; if specified (even if the value is "false"!), the audio will automatically begin playback as soon as it can do so, without waiting for the entire audio file to finish downloading."
<audio> - HTML - MDN

Wat.

Wonder if/how howler/soundjs handle this in their audio tag fallback?

It's possible our approach is very clunky. Here's what we do (abridged):

    var audioElement = new window.Audio(file);
    if (!audioElement || !audioElement.play) {
      return;
    }

    if (!isIE9()) { // Pre-cache audio
      audioElement.play();
      audioElement.pause();
    }

    var eventListener = function () {
      this.onSoundLoaded();
      audioElement.removeEventListener('canplaythrough', eventListener);
    }.bind(this);
    audioElement.addEventListener('canplaythrough', eventListener);

howler.js

howler.js seems to be more meticulous with its HTML5 audio. During setup, they call new Audio() but don't assign it to anything just to test whether audio needs to be disabled entirely. Then, for each sound, in Howl.prototype.load they also call new Audio() (without any arguments) and save it. They also do a lot of manual setup:

        var newNode = new Audio();

        // listen for errors with HTML5 audio (http://dev.w3.org/html5/spec-author-view/spec.html#mediaerror)
        newNode.addEventListener('error', function () {
          if (newNode.error && newNode.error.code === 4) {
            HowlerGlobal.noAudio = true;
          }

          self.on('loaderror', {type: newNode.error ? newNode.error.code : 0});
        }, false);

        // setup the new audio node
        newNode.src = url;
        newNode._pos = 0;
        newNode.preload = 'auto';
        newNode.volume = (Howler._muted) ? 0 : self._volume * Howler.volume();

        var listener = function() {
          // round up the duration when using HTML5 Audio to account for the lower precision
          self._duration = Math.ceil(newNode.duration * 10) / 10;

          // setup a sprite if none is defined
          if (Object.getOwnPropertyNames(self._sprite).length === 0) {
            self._sprite = {_default: [0, self._duration * 1000]};
          }

          if (!self._loaded) {
            self._loaded = true;
            self.on('load');
          }

          if (self._autoplay) {
            self.play();
          }

          // clear the event listener
          newNode.removeEventListener('canplaythrough', listener, false);
        };
        newNode.addEventListener('canplaythrough', listener, false);
        newNode.load();

Note that the explicitly set the src, position, volume, and preload attribute after construction, and also that they call HTMLMediaElement.prototype.load() - a step that we omit.

HTMLMediaElement.load()
Resets the media element and restarts the media resource. Any pending events are discarded. How much media data is fetched is still affected by the preload attribute. This method can be useful for releasing resources after any src attribute and source element descendants have been removed. Otherwise, it is usually unnecessary to use this method, unless required to rescan source element children after dynamic changes.
HTMLMediaElement - MDN

Maybe this stabilizes their implementation?

SoundJS

SoundJS is by far the most elaborate. It reuses a pool of audio tags. New audio tags are created like this:

var tag = document.createElement("audio");
tag.autoplay = false;
tag.preload = "none";
return tag;

Then I think a lot of magic happens elsewhere in the library, but this section looks especially relevant - a kind of "on play, load if not loaded, otherwise play": (note, the _playbackResource and tag here are both the audio tag)

        if (this._playbackResource.readyState !== 4) {
            var tag = this._playbackResource;
            tag.addEventListener(createjs.HTMLAudioPlugin._AUDIO_READY, this._readyHandler, false);
            tag.addEventListener(createjs.HTMLAudioPlugin._AUDIO_STALLED, this._stalledHandler, false);
            tag.preload = "auto"; // This is necessary for Firefox, as it won't ever "load" until this is set.
            tag.load();
            return;
        }

        this._updateVolume();
        this._playbackResource.currentTime = (this._startTime + this._position) * 0.001;
        if (this._audioSpriteStopTime) {
            this._playbackResource.addEventListener(createjs.HTMLAudioPlugin._TIME_UPDATE, this._audioSpriteEndHandler, false);
        } else {
            this._playbackResource.addEventListener(createjs.HTMLAudioPlugin._AUDIO_ENDED, this._endedHandler, false);
            if(this._loop != 0) {
                this._playbackResource.addEventListener(createjs.HTMLAudioPlugin._AUDIO_SEEKED, this._loopHandler, false);
                this._playbackResource.loop = true;
            }
        }

        this._playbackResource.play();

I'm not sure any of this motivates me to update our HTML5 audio approach though 😦 .

LGTM after keypress / keydown issue gets addressed