Audio alignment kit (forced alignment)

audio-align is a Gipity kit that gives any app forced alignment: hand it an audio URL plus the lyrics or transcript, get back precise word-level timings (start_ms, end_ms, confidence). Built for karaoke captioning but useful for subtitling, language learning, dubbing alignment, and lyric videos.

Install

gipity add audio-align

The installer drops the kit into src/packages/audio-align/, wires the import map (import audioAlign from '@gipity/audio-align'), and registers the audio-align GPU job in your gipity.yaml. Run gipity deploy dev to ship it.

What it does

Pipeline (Python, runs on a Gipity Jobs L4 GPU):

1. Download the audio from audio_url
2. Demucs vocal isolation (4-stem model, takes the vocals stem) - improves alignment accuracy vs. raw mixed audio
3. torchaudio.pipelines.MMS_FA forced alignment of vocals against the lyric tokens
4. Librosa onset refinement - snaps each word's start to the nearest audio onset within ±50ms
5. Returns JSON with words, phrases (one per lyric line), and metadata

Cold start ~10s, ~30-60s wall time for a 3-min song. ~$0.01 in credits per song (L4 at $0.80/hr × 100% margin). Model weights cache in /cache/ after first run.

Pattern: wrapper function + browser helper

Browser code can't directly submit jobs (no app-token job endpoint yet), so the app dev writes two thin wrapper functions that submit/check status on the user's behalf; the browser calls them via @gipity/audio-align. Copy the wrappers from src/packages/audio-align/examples/wrapper-function.js into your project's functions/ directory.

# gipity.yaml additions
- name: audio-align-submit
  handler: functions/audio-align-submit.js
  auth: user
- name: audio-align-status
  handler: functions/audio-align-status.js
  auth: user

// browser
import audioAlign from '@gipity/audio-align';

const result = await audioAlign.align({
  appGuid: 'p_yourapp01',
  userToken,
  audioUrl: 'https://media.gipity.ai/...',
  lyrics:   'first line
second line
',
  onProgress: ({ pct, message }) => {
    progressBar.value = pct;
    statusEl.textContent = message;
  },
});

console.log(result.words);

Output shape

{
  "words":   [{ "word": "hello", "start_ms": 1234, "end_ms": 1456, "confidence": 0.95 }, ...],
  "phrases": [{ "text": "line one", "start_ms": 1234, "end_ms": 2100,
                "word_idx_start": 0, "word_idx_end": 2 }, ...],
  "metadata": {
    "duration_ms":    180000,
    "sample_rate":    16000,
    "used_demucs":    true,
    "refined_onsets": true
  }
}

confidence is the per-word MMS_FA score. Useful for flagging low-confidence words in an editor UI (typical threshold: < 0.5 = needs manual review).

Common gotchas

• Lyrics must match what's sung. If the singer ad-libs words not in the lyrics, alignment drifts after that point. Confidence drops on those words; surface them in your UI.
• Already-isolated input (a cappella, dry vocal stem) - pass skip_demucs: true to skip the demucs step (saves ~20 sec of GPU work).
• Lyric phrasing: one phrase per line. The kit segments output by line breaks for the phrases array.
• on_complete chaining is supported - declare on_complete: <function-name> on the job and the platform fires that function with { run_guid, status, output, error, duration_ms, job_name } when alignment terminates. Use for chaining align -> render without browser polling.

See also

• jobs.md - underlying job tier (GPU compute classes, billing, fat image contents)
• app-files.md - upload audio to get a URL for audio_url
• app-llm.md - for a custom display-map step (punctuation/quotes normalization)