OpenAI has launched a public beta of the Realtime API, an API that permits paid builders to construct low-latency, multi-modal experiences together with textual content and speech in apps.

Launched October 1, the Realtime API, just like the OpenAI ChatGPT Superior Voice Mode, helps pure speech-to-speech conversations utilizing preset voices that the API already helps. OpenAI is also introducing audio enter and output within the Chat Completions API to assist use circumstances that don’t want the low-latency advantages of the Realtime API. Builders can move textual content or audio inputs into GPT-4o and have the mannequin reply with textual content, audio, or each.

With the Realtime API and the audio assist within the Chat Completions API, builders wouldn’t have to hyperlink collectively a number of fashions to energy voice experiences. They’ll construct pure conversational experiences with only one API name, OpenAI mentioned. Beforehand, creating an analogous voice expertise had builders transcribing an automated speech recognition mannequin similar to Whisper, passing textual content to a textual content mannequin for inference or reasoning, and taking part in the mannequin’s output utilizing a text-to-speech mannequin. This strategy usually resulted in lack of emotion, emphasis, and accents, plus latency.