Idle Amps

Finite State Ballads

Story songs and the films made for them. The music, the voices, the pictures and the motion are all generated.

Open playlist →

On YouTube

Finite State Ballads is a playlist of songs that tell a story, each with a film built to it. There is noir, metal, folk, a twelve-minute rock opera and two sermons. The playlist's own description is "AI all the way down," and that is accurate: none of the people are real and nothing was filmed.

A few of them:

  • The Ledger at Room 4B. A rock opera in five movements. A hatchet man finds a machine in the server room that is running the firm.
  • Whale Fight! A love triangle under the pack ice, drawn in crayon. A blue whale mistakes a submarine's sonar for a love song.
  • An AI Sermon | Falling Sideways. An old preacher's dream, told as a sermon. The camera never stops turning.
  • Die Jagd. An evening run that turns into a chase through the woods, with German death metal in her earbuds.
  • What Falls. One tree's year, from the maple letting go to a young leaf turning gold.

What made them

Everything runs on my own PC through ComfyUI. Claude writes the ComfyUI graphs, runs them, and looks at the frames that come back.

  • Music and vocals: ACE-Step 1.5.
  • Stills and keyframes: Z-Image Turbo.
  • Keeping a character the same from shot to shot: Qwen-Image-Edit, given one to three reference pictures.
  • Motion: Wan 2.1 image-to-video (14B) with the lightx2v 4-step LoRA. Uni3C steers the camera. InfiniteTalk does lip-sync when someone sings on screen.
  • Finishing: RIFE for in-between frames, SeedVR2 or 4x-UltraSharp for upscaling.
  • Checking: faster-whisper listens to each take, to tell me whether it sang the words and where each line falls.

Almost all of it runs at home. Now and then a long render queue goes to rented GPUs at Modal: What Falls rendered its 54 clips there for about $3.30.

How a song becomes a film

Each one starts as words: a lyric of mine, a treatment, or a story worked out with Claude. The words go to ACE-Step, and then the hunt starts, because most takes don't sing the lyric properly. The Butchers Lament took about 60 takes. The key turned out to matter. In E minor the model sang the last line on four seeds out of five. In A minor it never did.

Once a take is chosen, Whisper marks where every line falls, the film is cut to those marks, and the pictures are made to fit.

I built four different small programs to do that, each to try a different idea:

  • script_film. A script lists every shot, line and sound. The film is rendered, then measured against the script. This produced the silent-film videos (The Survey, Two Eighteen, The Retainer): the singer is off screen, a recurring cast acts the song out, and any line the take didn't sing becomes a title card.
  • treatment_film. Paste in a treatment and get a film back, with three stops for me to approve: the track, the look, and the keyframes. It finishes in 4K. The Switch was made this way.
  • maze_film. The film is played like a game. Extend one clip at a time, score it, throw away the ones where the model broke (melting walls, impossible physics), back up and go on from the last good frame. Falling Sideways took 138 attempts to get the 84 clips in it.
  • pulse_film. The picture plays the music's rhythm. The beats are measured and each voice in the music drives its own part of the picture.

The glitches are sometimes left in on purpose. As the Whale Fight! description says, the machine shows through, and the song's surreal anyway.

Also here

Open playlist →

More from Idle Amps