A new AI system can create natural-sounding speech and music after being prompted with a few seconds of audio. Audio, developed by Google researchers, generates audio that fits the style of the prompt, including complex sounds like piano music, or people speaking, in a way that is almost indistinguishable from the original recording. The technique shows promise for speeding up the process of training AI to generate audio, and it could eventually be used to auto-generate music to accompany videos.

AI-generated audio is commonplace: voices on home assistants like Alexa use natural language processing. AI music systems like OpenAI’s Jukebox have already generated impressive results, but most existing techniques require people to prepare transcriptions and label text-based training data, which takes a lot of time and human labor. Jukebox, for example, uses text-based data to generate song lyrics.

Audio, described in a non-peer-reviewed paper last month, is different: it doesn’t require transcription or labelling. Instead, sound databases are fed into the program, and machine learning is used to compress the audio files into sound snippets, called “tokens,” without losing too much information. To generate the audio, a few seconds of sound are fed into audio, which then predicts what comes next.

Roger Tannenberg, who researches computer-generated music at Carnegie Mellon University, says audio already has much better sound quality than previous music generation programs. In particular, he says, audio is surprisingly good at re-creating some of the repeating patterns inherent in human-made music. To generate realistic piano music, audio has to capture a lot of the subtle vibrations contained in each note when piano keys are struck. The music also has to sustain its rhythms and harmonies over a period of time.