Greetings, Professor Falken: Web Speech APIs Part I

Utterance Events

Because everything is async, if you wanted to do something like sync up some UI with spoken text, you'll need to know when things happen by watching for events on the utterance. Utterances have several events: The ones you probably most care about are:

• onstart "Fired when this utterance has begun to be spoken"
• onend "Fired when this utterance has completed being spoken. If this event fires, the error event must not be fired for this utterance."
• onerror "Fired if there was an error that prevented successful speaking of this utterance. If this event fires, the end event must not be fired for this utterance" We'll come back to this one...

Let's try it out...

let utteranceOne = new SpeechSynthesisUtterance(
      `We come from the land of the ice and snow`
  ),
  utteranceTwo = new SpeechSynthesisUtterance(
      `From the midnight sun where the hot springs flow`
  ),
  syncUIHandler = (event) => {
    // There is an event.utterance in chrome,
    // but that seems to be non-standard,
    // you want event.target which is the
    // utterance object (though it should be read-only)
    document.querySelector('#zepplin-out').innerText = event.target.text
  }

  utteranceOne.onstart = syncUIHandler
  utteranceTwo.onstart = syncUIHandler

  utteranceTwo.onend = () => {
    console.log('done')
  }

speechSynthesis.speak(utteranceOne)
speechSynthesis.speak(utteranceTwo)
        

Assuming that that worked for you: Yay! Couldn't be simpler, right? Got your head well around that? Good... Ok, now let me explain why you probably don't.

I think that most people would associate the words "when this utterance has begun to be spoken" with "some kind of sound has begun to flow out of the speakers" but this isn't the case. If we added a speechSynthesis.pause() to the end of that code sample, the first utterance's .onstart would be called, and the words would appear on the screen, but no sound would play. This is universal in all the browsers. Luckily, there ares are also .onpause and .onresume events (but recall, if pause didn't work for you above, this one won't either). Let's see...

let utteranceOne = new SpeechSynthesisUtterance(
      `Maybe it's not too late`
  ),
  utteranceTwo = new SpeechSynthesisUtterance(
      `To learn how to love, and forget how to hate`
  ),
  syncUIHandler = (event) => {
    let el = document.querySelector('#ozzy-out')

    if (event.type == 'pause') {
      el.innerText = 'Paused'
    } else {
      // Chrome has an event.utterance object,
      // but other browsers don't
      el.innerText = event.target.text
    }
  }

  utteranceOne.onpause = syncUIHandler
  utteranceOne.onstart = syncUIHandler
  utteranceOne.onresume = syncUIHandler
  utteranceTwo.onstart = syncUIHandler
  utteranceTwo.onpause = syncUIHandler
  utteranceTwo.onresume = syncUIHandler

speechSynthesis.speak(utteranceOne)
speechSynthesis.speak(utteranceTwo)
speechSynthesis.pause()

setTimeout(() => {
    speechSynthesis.resume()
}, 2000)
        

Did it work? Here's some bad news... there are a lot of notes here:

If you're on Android and nothing is happening, you might have missed the note about .pause not working right there. and you should click this button to clear the queue and restore normalcy (don't you wish that button worked on the whole world).

At least in Chrome (that's just the one I use most) it appears to be very easy to create situations where callbacks are not called, despite it being blatantly obvious that they happened. There have been a couple of bugs opened about this. In my experience, this seems to be closely related to either the 'shared queue' note above or garbage collection causing utterance objects to be cleaned up before they are really fired because of longer text, so you've got to deal with all that yourself.

One more thing... If .onError is called, the event it raises will have an .error attribute (or, it's supposed to) which should be an error code indicating why. The draft lists 11 'kinds' of problems and they are both informatice to the kinds of things you should be thinking about, but some of these don't seem to happen IRL and leave undefined spec behavior gaps.

Expressiveness

Speech is complicated if you want it to seem natural. In my History of Speech I link to a recording of Bell Lab's Voder in the 1940s demonstrating the many ways one might intone the phrase "she saw me". Similarly, since we're just using strings of text, we're theoretically subject to all of the sorts of potential problems about ambigious strings. For example given the string "12/4 = 3" we'd probably like that to be pronounced "twelve divided by four equals three" whereas we might like "12/12/2012" to be pronounced like "December twelfth, twenty twelve".

The first bit of good news is that all of the speech synthesis implementations I've tried actually do a pretty good job using punctuation to inflect and pause and frequently can deal with a lot of those problems automatically... For example, all of the below pronouce sensibly for me without further effort:

speechSynthesis.speak(
  new SpeechSynthesisUtterance(`
    1. Pi is about 3.14
    2. We loaded the 4x4
    3. Please meet me at 3.14pm EST
    4. My birthday is 2/17/1974
  `)
)
            

But it's not bullet proof and it's automatic, not consistent. To some, this is unacceptable. The first example in this document purposely misspelled the name of Professor Falken from War Games because the best guess sounds like an expletive. To wit, in my history article I also point to several "official" standards that were developed in the W3C to deal with just these sorts of problems. I'm mentioning this here because the Speech API says that the text provided to an utterance can be one of these: A well-formed SSML (Speech Synthesis Markup Language) document. SSML lets you express all kinds of stuff. This would allow you to create a single utterance which varied all sorts of things throughout and let you express how you wanted each bit to be said. The spec gives implementations a way out saying that if the implementation doesn't support SSML, they have to just take the text content. That's a pretty elegant, progressive compromise I suppose. Well done standards!

It also completely doesn't work that way. All the implementations I've looked at will actually try to pronounce all the markup. That's bad. I suggest you just avoid that.

The really good news though is that you can actually achieve a lot of the same amount of control by doing your own processing/creation of utterances.

Language

Utterances also have a .lang property which you can set. It has to be a string representing a BCP 47 language tag (like en-US). If you don't set it, it will by default be the language of the document. This can give hints to pronounciation. Let's try an example from The Godfather:

let apollonia = new SpeechSynthesisUtterance(
   `io so l'inglese:
      Monday Tuesday Thursday Wednesday
      Friday Sunday Saturday
    `
  )
apollonia.pitch = 1.1
apollonia.lang = 'it-US'
speechSynthesis.speak(apollonia)

If you set the .lang to an invalid language, say, for example "Hobbit" or "Vulcan" it will simply use the default. I couldn't find that in the spec somewhere, but that seems to be the behavior in Chrome.

A million voices cried out

Let's talk about the most painful part of this: voices. You might have noticed that the example above from The Godfather sounded quite a lot different - it was using a different SpeechSynthesisVoice, automatically.

From here out, I'm going to refer to SpeechSynthesisVoice as simply "voice" because, well, that's a mouthful.

You can also set the voice yourself, but there are a number of gotchas here...

(function () {
    speechSynthesis.onvoiceschanged = function () {
       let el = document.querySelector('#voices-changed-1')
       // Hey, at least it's a real Array! (despite the spec)
       speechSynthesis.getVoices().forEach((voice) => {
          el.innerHTML = voice.name
       })
    }
}())
                

let voice = speechSynthesis.getVoices().find((voice) => {
    return voice.name == 'Bubbles'
  }),
  lyrics = new SpeechSynthesisUtterance(
      `It's as simple as something that nobody knows
       That her eyes are as big as her bubbly toes`
  )
  lyrics.voice = voice
  speechSynthesis.speak(lyrics)