{
  "name": "audio-align",
  "title": "Audio alignment kit (forced alignment)",
  "description": "Forced alignment kit: audio + lyrics -> word-level timing JSON. Demucs vocal isolation + MMS_FA, runs as a Gipity Jobs GPU task.",
  "guid": "sk_kit_align",
  "category": "Kits",
  "requiredTools": [
    "job_submit",
    "job_status"
  ],
  "content": "# Audio alignment kit\n\n`audio-align` is a Gipity kit that gives any app forced alignment: hand it an audio URL plus the lyrics or transcript, get back precise word-level timings (`start_ms`, `end_ms`, `confidence`). Built for karaoke captioning but useful for subtitling, language learning, dubbing alignment, and lyric videos.\n\n## Install\n\n```bash\ngipity add audio-align\n```\n\nThe installer drops the kit into `src/packages/audio-align/`, wires the import map (`import audioAlign from '@gipity/audio-align'`), and registers the `audio-align` GPU job in your `gipity.yaml`. Run `gipity deploy dev` to ship it.\n\n## What it does\n\nPipeline (Python, runs on a Gipity Jobs L4 GPU):\n\n1. Download the audio from `audio_url`\n2. Demucs vocal isolation (4-stem model, takes the vocals stem) - improves alignment accuracy vs. raw mixed audio\n3. `torchaudio.pipelines.MMS_FA` forced alignment of vocals against the lyric tokens\n4. Librosa onset refinement - snaps each word's start to the nearest audio onset within ±50ms\n5. Returns JSON with words, phrases (one per lyric line), and metadata\n\nCold start ~10s, ~30-60s wall time for a 3-min song. ~$0.01 in credits per song (L4 at $0.80/hr × 100% margin). Model weights cache in `/cache/` after first run.\n\n## Pattern: wrapper function + browser helper\n\nBrowser code can't directly submit jobs (no app-token job endpoint yet), so the app dev writes two thin wrapper functions that submit/check status on the user's behalf; the browser calls them via `@gipity/audio-align`. Copy the wrappers from `src/packages/audio-align/examples/wrapper-function.js` into your project's `functions/` directory.\n\n```yaml\n# gipity.yaml additions\n- name: audio-align-submit\n  handler: functions/audio-align-submit.js\n  auth: user\n- name: audio-align-status\n  handler: functions/audio-align-status.js\n  auth: user\n```\n\n```js\n// browser\nimport audioAlign from '@gipity/audio-align';\n\nconst result = await audioAlign.align({\n  appGuid: 'p_yourapp01',\n  userToken,\n  audioUrl: 'https://media.gipity.ai/...',\n  lyrics:   'first line\nsecond line\n',\n  onProgress: ({ pct, message }) => {\n    progressBar.value = pct;\n    statusEl.textContent = message;\n  },\n});\n\nconsole.log(result.words);\n```\n\n## Output shape\n\n```json\n{\n  \"words\":   [{ \"word\": \"hello\", \"start_ms\": 1234, \"end_ms\": 1456, \"confidence\": 0.95 }, ...],\n  \"phrases\": [{ \"text\": \"line one\", \"start_ms\": 1234, \"end_ms\": 2100,\n                \"word_idx_start\": 0, \"word_idx_end\": 2 }, ...],\n  \"metadata\": {\n    \"duration_ms\":    180000,\n    \"sample_rate\":    16000,\n    \"used_demucs\":    true,\n    \"refined_onsets\": true\n  }\n}\n```\n\n`confidence` is the per-word MMS_FA score. Useful for flagging low-confidence words in an editor UI (typical threshold: < 0.5 = needs manual review).\n\n## Common gotchas\n\n- **Lyrics must match what's sung.** If the singer ad-libs words not in the lyrics, alignment drifts after that point. Confidence drops on those words; surface them in your UI.\n- **Already-isolated input** (a cappella, dry vocal stem) - pass `skip_demucs: true` to skip the demucs step (saves ~20 sec of GPU work).\n- **Lyric phrasing**: one phrase per line. The kit segments output by line breaks for the `phrases` array.\n- **on_complete chaining** is supported - declare `on_complete: <function-name>` on the job and the platform fires that function with `{ run_guid, status, output, error, duration_ms, job_name }` when alignment terminates. Use for chaining align -> render without browser polling.\n\n## See also\n\n- `jobs.md` - underlying job tier (GPU compute classes, billing, fat image contents)\n- `app-files.md` - upload audio to get a URL for `audio_url`\n- `app-llm.md` - for a custom display-map step (punctuation/quotes normalization)\n"
}