# Audio alignment kit

`audio-align` is a Gipity kit that gives any app forced alignment: hand it an audio URL plus the lyrics or transcript, get back precise word-level timings (`start_ms`, `end_ms`, `confidence`). Built for karaoke captioning but useful for subtitling, language learning, dubbing alignment, and lyric videos.

## Install

```bash
gipity add audio-align
```

The installer drops the kit into `src/packages/audio-align/`, wires the import map (`import audioAlign from '@gipity/audio-align'`), and registers the `audio-align` GPU job in your `gipity.yaml`. Run `gipity deploy dev` to ship it.

## What it does

Pipeline (Python, runs on a Gipity Jobs L4 GPU):

1. Download the audio from `audio_url`
2. Demucs vocal isolation (4-stem model, takes the vocals stem) - improves alignment accuracy vs. raw mixed audio
3. `torchaudio.pipelines.MMS_FA` forced alignment of vocals against the lyric tokens
4. Librosa onset refinement - snaps each word's start to the nearest audio onset within ±50ms
5. Returns JSON with words, phrases (one per lyric line), and metadata

Cold start ~10s, ~30-60s wall time for a 3-min song. ~$0.01 in credits per song (L4 at $0.80/hr × 100% margin). Model weights cache in `/cache/` after first run.

## Pattern: wrapper function + browser helper

Browser code can't directly submit jobs (no app-token job endpoint yet), so the app dev writes two thin wrapper functions that submit/check status on the user's behalf; the browser calls them via `@gipity/audio-align`. Copy the wrappers from `src/packages/audio-align/examples/wrapper-function.js` into your project's `functions/` directory.

```yaml
# gipity.yaml additions
- name: audio-align-submit
  handler: functions/audio-align-submit.js
  auth: user
- name: audio-align-status
  handler: functions/audio-align-status.js
  auth: user
```

```js
// browser
import audioAlign from '@gipity/audio-align';

const result = await audioAlign.align({
  appGuid: 'p_yourapp01',
  userToken,
  audioUrl: 'https://media.gipity.ai/...',
  lyrics:   'first line
second line
',
  onProgress: ({ pct, message }) => {
    progressBar.value = pct;
    statusEl.textContent = message;
  },
});

console.log(result.words);
```

## Output shape

```json
{
  "words":   [{ "word": "hello", "start_ms": 1234, "end_ms": 1456, "confidence": 0.95 }, ...],
  "phrases": [{ "text": "line one", "start_ms": 1234, "end_ms": 2100,
                "word_idx_start": 0, "word_idx_end": 2 }, ...],
  "metadata": {
    "duration_ms":    180000,
    "sample_rate":    16000,
    "used_demucs":    true,
    "refined_onsets": true
  }
}
```

`confidence` is the per-word MMS_FA score. Useful for flagging low-confidence words in an editor UI (typical threshold: < 0.5 = needs manual review).

## Common gotchas

- **Lyrics must match what's sung.** If the singer ad-libs words not in the lyrics, alignment drifts after that point. Confidence drops on those words; surface them in your UI.
- **Already-isolated input** (a cappella, dry vocal stem) - pass `skip_demucs: true` to skip the demucs step (saves ~20 sec of GPU work).
- **Lyric phrasing**: one phrase per line. The kit segments output by line breaks for the `phrases` array.
- **on_complete chaining** is supported - declare `on_complete: <function-name>` on the job and the platform fires that function with `{ run_guid, status, output, error, duration_ms, job_name }` when alignment terminates. Use for chaining align -> render without browser polling.

## See also

- `jobs.md` - underlying job tier (GPU compute classes, billing, fat image contents)
- `app-files.md` - upload audio to get a URL for `audio_url`
- `app-llm.md` - for a custom display-map step (punctuation/quotes normalization)