Listen up: Web Speech APIs Part III

Events

In order to actually do anything with it, we're going to want to tell this recognizer what we actually want from it via attaching some listeners to it. There are a lot of events that we can plug into, as you will see below but in practice I haven't personally required many of them.

• audiostart "Fired when the user agent has started to capture audio."
• soundstart "Fired when some sound, possibly speech, has been detected."
• speechstart "Fired when the speech that will be used for speech recognition has started."
• speechend "Fired when the speech that will be used for speech recognition has ended."
• soundend "Fired when some sound is no longer detected."
• audioend "Fired when the user agent has finished capturing audio."
• result "Fired when the speech recognizer returns a result."
• nomatch Don't worry about this one for now, we'll come back to it.
• error "Fired when a speech recognition error occurs."
• start "Fired when the recognition service has begun to listen to the audio with the intention of recognizing"
• end "Fired when the service has disconnected"

A few of these are self explanatory in terms of their lifecycle: Starts have to occur before their corresponding ends. However, one could be forgiven for wondering, for example, "which happens first according to these descriptions? audio start or sound start?" To answer your question, let me just quote the spec: Unless specified below, the ordering of the different events is undefined. For example, some implementations may fire audioend before speechstart or speechend if the audio detector is client-side and the speech detector is server-side.

If you're thinking to yourself: "Wow, that seems potentially really confusing and hard to work with," same.

Methods

The recognition object also has 3 methods in charge of whether it is actively listening and feeding things to the above events:

• start
• stop
• abort

If you're wondering what the difference between stop and abort is, it's actually pretty simple.: Stop means stop listening, but not stop processing and feeding any pending events to its listeners. Abort is a hard disconnect of all the things.

Starting up

Before we set up an example, it's worth explaining that this feature is "different". Playing some media is something that has been a part of the Web for a really long time, it's not something that requires a special permission. But listening to your microphone is different, as it should be. That's something that the user has to explicitly give permission for and be made aware when it's happening. As such, when you call .start() Chrome will ask the user if it's ok for this website to use the mic. Keep in mind, they could well say "no" so you'll want to prepare for that eventuality - it will call the error handler with the event.error property of not-allowed.

Even this, however, is incomplete. "Ok to access the mic" is a pretty 'simple' answer, but it's far reaching. If I want to be able to click a field and speak into it (like Google search on Android), does that mean I am also giving permission to listen to the conversations in the room at all other times as long as there's a tab open? That sounds terrible in most use cases, but there are use cases that are kind of almost that. This is tricky to manage, as you can imagine, so the draft and implementation say that the UA (browser) has to indicate to the user when it is and isn't actually listening and it has to stop when the tab is no longer active. This can take a few different forms, and we'll come back to that - but generally speaking, Chrome on the desktop will (currently) give a user two indicators that distinguish the difference between these two things.

Once the user has given permission, in the right of the address bar, it will say it is accessing the mic. That doesn't mean actively listening for speech.

Once we actively start listening, in the tab itself, it will show a red "recording" dot. This dot will disappear when we stop.

I personally have trouble believing that the average user will really understand the difference without some training which could happen the first time the mic permission is granted or something, but I've never seen. I'm also not entirely sure how this translates for someone who can't actually see the screen, for example. This is interesting, because that is actually the default assumption on mobile so, on Android it will play audible listening/done listening tones. I'll come back to why this too is currently imperfect, but let's move one.

The most basic example

Alright, so now you know what to expect, as a user and we know basically what the API looks like, let's talk about code.

const recognition = new webkitSpeechRecognition(),
      basicExampleOut = document.querySelector('#basicExampleOut')

recognition.onerror = (evt) => {
    if (evt.error == 'not-allowed') {
        basicExampleOut.innerText =
            `I can't listen if you don't grant me permission :(`
    } else {
        basicExampleOut.innerText =
            `Whoops I got an ${evt.error} error`
    }
}

recognition.onresult = (evt) => {
    // What is this crazy thing?! Don't worry, we'll get to it.
    let whatIHeard = evt.results[0][0].transcript
    basicExampleOut.innerText = whatIHeard
}

recognition.start()
                

Note that once you got a result, it stopped listening for more results automatically. By default, this is how it works. Now let's talk about that crazy evt.results[0][0].transcript thing.

recognition.onresult = (evt) => {
        // evt.results is an array-like collection of
        // SpeechRecognitionResult objects
    let speecRecognitionResults = evt.results,

        // it's a collection of exactly one by default (see below),
        // but that is also array-like
        speechReognitionAlternatives = speechRecognitionResults[0]

        // but that also contains by default exactly one
        alternative = speechRecognitionAlternatives[0]

        // and _that_ thing has all the properties
        console.log(alternative.transcript)

}
                            

var recog = new webkitSpeechRecognition()
recog.maxAlternatives = 10
recog.onerror = (evt) => {
   debugger
}
recog.onresult = (evt) => {
   let alternatives = Array.from(evt.results[0])
   alternatives.forEach((result) => {
      console.log(
        `
        confidence: ${result.confidence}
        transcript: "${result.transcript}"
        `)
   })
}
recog.start()
            

confidence: 0.7706348896026611
transcript: "tutu"

confidence: 0.9056409001350403
transcript: "two two"

confidence: 0.9139654040336609
transcript: "to to"

confidence: 0.9311671257019043
transcript: "2 2"

confidence: 0.933784008026123
transcript: "- 2"

confidence: 0.9352703094482422
transcript: "too too"

confidence: 0.9483599662780762
transcript: "22"

confidence: 0.9183915853500366
transcript: "TuTiTu"

confidence: 0.9120156168937683
transcript: "to 2"

confidence: 0.9352703094482422
transcript: "tu tu"

confidence: 0.9614496827125549
        transcript: "Desmond Tutu"

Wrapping up

So, it's interesting. It works experimentally, if you can use only the parts that work. It isn't nearly as far along 'in practice' as the text-to-speech APIs and even the limited implementations we have have some wonky bits... In my next post, I'll talk about how I am dealing with the wonky bits of this, and putting together some interesting things you can do with it.