Speech recognition: how it works and why phone calls put it to the test

A speech recognition system receives an audio signal, extracts acoustic features from it, then looks for the most likely sequence of words matching that signal. Recent systems rely on neural networks trained on very large volumes of transcribed audio. The Whisper model, described by its authors in a paper published in December 2022, was trained on 680,000 hours of multilingual audio.

Recognition does not stop at transcription. In a voice agent, the resulting text is then analysed to work out what the caller wants: to book an appointment, check opening times, report a problem. An error at the first stage carries over to the next: if “Tuesday” is heard as “Wednesday”, the rest of the conversation starts on the wrong footing.

Traditional telephony deliberately transmits limited audio. Recommendation G.711 of the International Telecommunication Union (ITU), used in telephone networks, sets a nominal sampling rate of 8,000 samples per second, which mechanically restricts the transmitted frequencies to below 4 kHz. Anything above that limit is lost, which impoverishes the signal that recognition has to work with.

So-called wideband codecs, such as ITU Recommendation G.722, which encodes audio up to 7 kHz, improve matters, but a call may cross several networks and the weakest link determines the final quality. On top of this come packet losses on mobile or internet connections, which make fragments of syllables disappear.

Callers are not phoning from a studio. They may be on a building site, in a car, in a busy street or in a kitchen with the extractor fan running. Background noise masks part of the voice and can be mistaken for words. Regional or foreign accents, fast speech, rare proper names and numbers rattled off on the spot are other sources of error.

The conversation itself poses difficulties. People hesitate, correct themselves, talk at the same time as the other person or start answering before the question is finished. A system has to know when the caller has finished speaking, without cutting them off too early or leaving them waiting in an awkward silence.

No provider can guarantee perfect recognition in all conditions. A well-designed voice answering service starts from that fact: it repeats back important information, such as a date, a phone number or an address, and asks for confirmation before recording it. It spells out names, or asks callers to spell them, when necessary, and offers to repeat rather than guess.

On the business side, a written transcript of each call lets you read back what was understood and spot recurring errors: a product name that is misrecognised, a question phrased in an unexpected way. This feedback is used to adjust the information given to the agent and to specify the cases where it should transfer the call instead.

In its white paper on voice assistants, the CNIL, France’s data protection authority, points out that the voice is a biometric characteristic that can make it possible to identify a person, and stresses the importance of transparency and security in these systems. For phone answering, this means informing callers about the recording at the start of the conversation, limiting how long data is kept and choosing hosting whose location you know.

Also avoid having callers dictate sensitive information that is not needed to handle their request. The less the system collects, the lower the risk in the event of a transcription error or a security incident.

Alyas answers the phone 24 hours a day based on the information you provide, announces at the start of the call that it is an AI and that the call is being recorded, and places a transcript and a summary of each conversation in your customer area. Its data is hosted in the European Union. The ten-day free trial lets you check, with transcripts to back it up, how well it understands your real callers.