Thoughts on Voice Recognition: Web Speech APIs Part IV

What's wrong with the current APIs

These are just my personal opinions after using it a lot for the past couple of months. Sometimes eating the dogfood gives you new perspectives and nothing in this should be taken as a criticism of the brave souls that worked hard to bring us what we have. It's easy to forget that we're standing on their shoulders. Hats off to them.

Let's assume that the API worked consistently, as advertised by the existing draft, across browsers and on all devices. The trouble with is that I see is mostly just that it doesn't "fit" in the Web platform today. It's not really a high level API, nor is it quite the low level we probably need.

// let's save bikeshedding a name for another day...
let transcriber = new FictionalTranscriber()

transcriber.transcribe(someAudioStream).then((response) => {
  // we'll talk more about this
})

let service = new FictionalTranscriber()

transcriber.transcribe(someAudioStream).then((response) => {
  console.log(`
    I am ${response.confidence}% confident
    that I heard "${response.transcript}".
  `)
})

let transcriber = new FictionalTranscriber()

transcriber.transcribe(someAudioStream).then((response) => {
  // if we didn't match anything sufficiently
  if (!response.transcript) {
     console.log(`Sorry, I didn't catch that`)
  } else {
    console.log(`
      I am ${response.confidence}% confident
      that I heard "${response.transcript}".
    `)
  }

})

let transcriber = new FictionalTranscriber({maxAlternatives: 10})

transcriber.transcribe(someAudioStream).then((response) => {
  console.log(`
    I am ${response.confidence}% confident
    that I heard "${response.transcript}".
  `)

  if (response.alternatives) {
    console.log(`But here are ${evt.alternatives} alternatives... `)
    response.alternatives.forEach((alternative) => {
      console.log(`
        I am ${alternative.confidence}% sure it could
        have been "${alternative.transcript}"
      `)
    })
  }

})

export default class FictionalGoogleCloudTranscriber
  extends FictionalTranscriber {
    // override
    request: (audio) => {
      // return some promise that meets the expected API
      return fetch(....).then(...)
    }
}

let transcriber = new FictionalTranscriber({
  serviceURI: 'blah',
  grammars: whateverThisIs
})

transcriber.transcribe(someAudioStream).then((response) => {
  console.log(`
    I am ${response.confidence}% confident
    that I heard "${response.transcript}".
  `)
})

Low Level First, Not Exclusively

The idea isn't ever to stop at the low level APIs. It's not even that we absolutely positively have to have the low level APIs entirely in place. As I explain in What Would Bruce Lee Do? The Tao of the Extensible Web, in some cases we can come back and expose details of a carefully designed black box after the fact. At a minimum, describing the API in common platform terms and the thought exercise of what a suitable low level API would look like allows us to design the boxes and layers carefully.

For example, given such an API as described above, it would make sense to add a higher level API which made use of it and exposed it... A pattern which explained the common lifecycle of grabbing a device, getting permission, listening to a channel, chunking that input, sending it to the transcriber and ultimately pumping out transcripts. Perhaps something like..

// Here we create an object not entirely unlike what
// the API already had - it defaults to being initialized
// with the audio context/primary mic, sends a chunk
// of sound to our transcriber and relays the promise
// results or errors to events
let speechRecognizer = new FictionalSpeechRecognizer()


// event based methods relay promise results
speechRecognizer.onresult = (evt) => { /* evt.response.transcript */ }
speechRecognizer.onerror = (evt) => { /* evt.error */ }

// the audioContext is exposed: You can control it
speechRecognizer.audioContext.start()
speechRecognizer.audioContext.stop()

The general idea being that we can answer the questions "what's under there?" by saying "it has an audio context and a transcriber" and we can expose those for extensibility... For example, we could allow you to swap out the transcriber implementation without forcing you to reimagine any other part in the system..

let speechRecognizer = new FictionalSpeechRecognizer({
  transcriber: new FictionalGoogleCloudTranscriber()
})

Or if you can get an audio context another way (say, one of N line inputs, or from a file), you can swap that out easily without forcing you to reimagine anything else either..

let speechRecognizer = new FictionalSpeechRecognizer({
  audioContext: someAudioContext
})

Given such an explanation, we could easily experiment with further layers, floating ideas as custom elements. For example: An input decorator that could progressively enhance an textual form controls to allow you to populate them with speech. All you'd need to say is 'they have one of those speech recognizer things and provide a toggle that pipes voice to it and the output back into the field'.. Something like this..

<x-voice-listener>
  <input name="input">
</x-voice-listener>
        

That's kind of interesting because there were specs trying to do something like that back to the early 2000's and it seems to be where Google restarted the converstaion in 2012 as well. We could figure that out, and do it with real world experimentation and testing! As you can see from this super rough demo, there's a lot to think about.. Should there be an attribute to autostart? Should it automatically stop? Should other attributes be present? paused, for example? I don't know, but we could try a lot of things and see what works.

So now what?

Since there's something implemented on Chrome and Android, you have a lot of 0's in the number of machines that actually support the existing API. As with a lot of 'new' features on the Web, you'll probably want think about how you can use progressive enhancement for a lot of use cases... In some cases, the API is good enough that you can paper over its kinks and that might be 'good enough'.

Unfortunately, the low-level API sketched out in this document isn't possible to achieve using the existing Speech APIs because they don't provide adequate explanations or hooks. Fortunately, however, it is entirely plausbile to make this API real today using a remote service that conforms to a design like this. Such services that transcribe sounds of speech to text are plentiful - Google's Cloud Platform and Amazon both provide APIs capable of achieving this for limited purposes would be free enough to experiment with and prove out, and fairly low cost beyond that.

I have a kind of rough POC that I'll be building out, for purposes of demonstrating ideas to limited groups, but wide publicly sharable demonstration is potentially cost prohibitive (unless some vendor wants to donate free time). However, if you have a business use for this, perhaps it's worthwhile. If you're interested in helping out with that, or hearing more, let me know.