The Music of Speech - Detecting emotion and attitude using AI

Prosody, the music of speech, conveys crucial communicative information: attitude (assertive, reassuring), emotion (relieved, distressed), emphasis (saliency), conversation action (request, command), etc. It is the least conscious, most instinctive activity of speech, and instructs the listener on how the words are to be interpreted, the way in which they are to be understood. Compare, for example, a sarcastic “Really…” to a surprised “Really?!” And now imagine that the hearing impaired or for people with autism could use a device which would distinguish between the two. And imagine that SIRI could also respond accordingly…

Indeed, and remarkably, this aspect of communication remains largely unexploited in speech technologies. This is due to several difficulties, mainly the fact that non-verbal messages in speech occur simultaneously: one could ask a question, be surprised, and be assertive all at the same time. It is a substantial challenge and gap that our lab is working on. In the fireside chat I will share the knowledge, insights and successes of our research in this domain.

[Alt text: Zero Project Conference 2025 banner. A green and white design featuring a digital globe with network lines on the left and the Zero Project logo with green leaves on the right]

For livestream: https://youtube.com/live/pZHcYWlhnog