{
  "name": "jobs-reference",
  "title": "Jobs Reference - Testing Pipelines & Common Gotchas",
  "description": "Jobs reference: testing a job pipeline (suppressed submits in tests, driving on_complete directly, the one-time real GPU smoke) and the common gotchas (Demucs recipe, deps conflicts, bash set -e, the output channel).",
  "guid": "sk_plat_jref",
  "category": "App development",
  "requiredTools": [
    "job_list",
    "job_submit",
    "job_status",
    "job_runs",
    "job_logs",
    "job_cancel"
  ],
  "content": "# Jobs Reference - testing & gotchas\n\nThe core flow lives in the [jobs](jobs.md) skill. This is the long tail: how to test a job pipeline without driving a real run, and the gotchas worth knowing before your first GPU job.\n\n## Testing a job pipeline\n\nA job pipeline is two functions plus a handler: a **trigger function** that calls `jobs.submit`, the **handler** that does the work, and an **`on_complete` function** that writes the result to your DB. Test the two functions and the handler **separately** - don't try to drive a real job run from your test suite.\n\n### The wiring is just two function calls - no real job needed\n\nIn a test run `jobs.submit()` is **suppressed**: it enqueues nothing and returns `{ run_guid, status: 'suppressed' }` with a **fresh, unique `run_guid` per call** (a real `jr_…` guid, exactly like a live submit), so the handler never runs and `on_complete` never fires. That's fine - **an `on_complete` function is an ordinary deployed function**, and the platform fires it with a plain body: `{ run_guid, job_name, status, output, error, duration_ms }`. So test the whole loop by calling both functions directly with synthetic data - no real job, no real audio file, and **no helper to \"seed a run\"** (there isn't one, and you don't need one):\n\n```js\ntest('split pipeline: submit records a pending row, on_complete finishes it', async (ctx) => {\n    // 1. Trigger fn: assert it returns the run_guid and wrote its pending row.\n    const { run_guid } = await ctx.fn.call('start-split', { song_guid: 'song_x' });\n    assert.ok(run_guid);   // unique per submit — key your pending row on it as in prod\n\n    // 2. Drive on_complete yourself with the body the platform would send.\n    //    (Call it like any function; if you gated it non-public, use ctx.fn.callAs.)\n    await ctx.fn.call('split-complete', {\n        run_guid, status: 'success',\n        output: { stems: { vocals: 'https://x/v.wav', drums: 'https://x/d.wav' } },\n    });\n\n    // 3. Assert the DB row your on_complete fn updated.\n    const row = await ctx.fn.call('split-status', { song_guid: 'song_x' });\n    assert.equal(row.status, 'done');\n});\n```\n\nThis exercises every line of app code in the pipeline. Match your `on_complete` fn on `run_guid` (unique per submit, so it disambiguates rows just like in prod) or on a field in the synthetic `output`, and **don't rewrite production matching logic just to make it testable**: the direct call above already drives it.\n\n### Smoke the real GPU handler once, from the CLI\n\nThe automated suite can't run a real job, so verify the handler itself manually - once - with a reachable input URL. Build a fixture in the sandbox, mint its URL with `gipity file url` (no deploy - the synced project copy is served directly; delete the file to revoke), submit, then wait:\n\n```bash\ngipity sandbox run bash 'ffmpeg -f lavfi -i sine=frequency=440:duration=5 /work/tests/fixtures/test.wav'\ngipity job submit <name> --data \"{\"audio_url\":\"$(gipity file url tests/fixtures/test.wav)\"}\"\ngipity job wait <runGuid> --timeout 300        # blocks until done + prints output; exits 2 (still running) or 1 (failed)\n```\n\n`gipity job wait` (see the CLI section above) is how you await the run: it blocks up to `--timeout` seconds, prints the `output` when the run finishes, and exits **2** if it's still running when the timeout hits - so re-run the same command to keep waiting. Never wrap a `sleep`+`job status` loop yourself, and never block on `job logs --follow` (it holds an SSE stream open until the run ends, so a cold-start GPU job outlasts your shell).\n\n**Expect a several-minute cold start on the FIRST GPU run.** A GPU model job (LTX-Video, Stable Video Diffusion, Demucs, etc.) downloads several GB of weights to `/cache` on its first run - commonly **3-5 minutes** of `running` with little log output. This is normal, not a hang; subsequent runs reuse the cached weights and are fast. Give `job wait` a generous `--timeout` (e.g. 300) for that first run, or just re-run it across turns until it exits 0.\n\n**To check WHY a run is slow (or catch an early crash) without streaming, use `gipity job logs <runGuid> --no-follow`.** It returns instantly with a `log_tail` field: the most recent stdout/stderr the job has printed (an import error, a \"downloading weights\" line). That's the non-blocking peek - use it instead of holding `--follow` open, which a cold-start GPU job outlasts.\n\nThe fixture never ships in the deployed app - `gipity file url` serves the synced project copy, and deleting the file revokes the link. You only do this **once** to confirm the handler works; the wiring test above (which needs no fixture) is what belongs in your permanent suite.\n\n## Common gotchas\n\n- **GPU without Modal config** - submit returns `MODAL_NOT_CONFIGURED` (503). Operator must set `MODAL_APP_URL` + `MODAL_INVOKE_SECRET` on the server.\n- **Demucs: split with the Python API, never the `demucs` CLI.** The model is preloaded, but the CLI's write step (and a bare `torchaudio.save`) routes through `torchcodec`, which is **not** in the image - so a CLI run separates fine and then fails at save. Run the model in-process and write stems with `soundfile` (libsndfile is installed). This is the whole 4-stem recipe:\n  ```python\n  import soundfile as sf, torch\n  from demucs.pretrained import get_model\n  from demucs.apply import apply_model\n  from demucs.audio import AudioFile\n  model = get_model(\"htdemucs\"); model.eval()          # weights cache in /cache/torch\n  wav = AudioFile(\"/work/song.mp3\").read(streams=0, samplerate=model.samplerate, channels=model.audio_channels)\n  ref = wav.mean(0); wav = (wav - ref.mean()) / ref.std()\n  with torch.no_grad():\n      stems = apply_model(model, wav[None], device=\"cuda\")[0]  # [4, ch, samples]\n  stems = stems * ref.std() + ref.mean()\n  for name, stem in zip(model.sources, stems):           # [\"drums\",\"bass\",\"other\",\"vocals\"]\n      sf.write(f\"/work/{name}.wav\", stem.T.cpu().numpy(), model.samplerate)\n  ```\n- **Deps that conflict with the fat image** - e.g. pinning `torch==2.3` when the image ships a different minor. `requirements.txt` wins via pip override, but CUDA/torch mismatches may produce subtle bugs. Prefer no `deps:` if the fat image already has what you need.\n- **Forgetting `set -e` in bash** - without it, only the LAST command's exit code propagates.\n- **`output` is whatever you `set_output`** - for an agent or function calling `job_submit`, the run's `output` (and what an `on_complete` hook receives) is the value the handler passed to `ctx.set_output(...)`. See \"Job output contract\" above.\n"
}