# Jobs Reference - testing & gotchas

The core flow lives in the [jobs](jobs.md) skill. This is the long tail: how to test a job pipeline without driving a real run, and the gotchas worth knowing before your first GPU job.

## Testing a job pipeline

A job pipeline is two functions plus a handler: a **trigger function** that calls `jobs.submit`, the **handler** that does the work, and an **`on_complete` function** that writes the result to your DB. Test the two functions and the handler **separately** - don't try to drive a real job run from your test suite.

### The wiring is just two function calls - no real job needed

In a test run `jobs.submit()` is **suppressed**: it enqueues nothing and returns `{ run_guid, status: 'suppressed' }` with a **fresh, unique `run_guid` per call** (a real `jr_…` guid, exactly like a live submit), so the handler never runs and `on_complete` never fires. That's fine - **an `on_complete` function is an ordinary deployed function**, and the platform fires it with a plain body: `{ run_guid, job_name, status, output, error, duration_ms }`. So test the whole loop by calling both functions directly with synthetic data - no real job, no real audio file, and **no helper to "seed a run"** (there isn't one, and you don't need one):

```js
test('split pipeline: submit records a pending row, on_complete finishes it', async (ctx) => {
    // 1. Trigger fn: assert it returns the run_guid and wrote its pending row.
    const { run_guid } = await ctx.fn.call('start-split', { song_guid: 'song_x' });
    assert.ok(run_guid);   // unique per submit — key your pending row on it as in prod

    // 2. Drive on_complete yourself with the body the platform would send.
    //    (Call it like any function; if you gated it non-public, use ctx.fn.callAs.)
    await ctx.fn.call('split-complete', {
        run_guid, status: 'success',
        output: { stems: { vocals: 'https://x/v.wav', drums: 'https://x/d.wav' } },
    });

    // 3. Assert the DB row your on_complete fn updated.
    const row = await ctx.fn.call('split-status', { song_guid: 'song_x' });
    assert.equal(row.status, 'done');
});
```

This exercises every line of app code in the pipeline. Match your `on_complete` fn on `run_guid` (unique per submit, so it disambiguates rows just like in prod) or on a field in the synthetic `output`, and **don't rewrite production matching logic just to make it testable**: the direct call above already drives it.

### Smoke the real GPU handler once, from the CLI

The automated suite can't run a real job, so verify the handler itself manually - once - with a reachable input URL. Build a fixture in the sandbox, mint its URL with `gipity file url` (no deploy - the synced project copy is served directly; delete the file to revoke), submit, then wait:

```bash
gipity sandbox run bash 'ffmpeg -f lavfi -i sine=frequency=440:duration=5 /work/tests/fixtures/test.wav'
gipity job submit <name> --data "{"audio_url":"$(gipity file url tests/fixtures/test.wav)"}"
gipity job wait <runGuid> --timeout 300        # blocks until done + prints output; exits 2 (still running) or 1 (failed)
```

`gipity job wait` (see the CLI section above) is how you await the run: it blocks up to `--timeout` seconds, prints the `output` when the run finishes, and exits **2** if it's still running when the timeout hits - so re-run the same command to keep waiting. Never wrap a `sleep`+`job status` loop yourself, and never block on `job logs --follow` (it holds an SSE stream open until the run ends, so a cold-start GPU job outlasts your shell).

**Expect a several-minute cold start on the FIRST GPU run.** A GPU model job (LTX-Video, Stable Video Diffusion, Demucs, etc.) downloads several GB of weights to `/cache` on its first run - commonly **3-5 minutes** of `running` with little log output. This is normal, not a hang; subsequent runs reuse the cached weights and are fast. Give `job wait` a generous `--timeout` (e.g. 300) for that first run, or just re-run it across turns until it exits 0.

**To check WHY a run is slow (or catch an early crash) without streaming, use `gipity job logs <runGuid> --no-follow`.** It returns instantly with a `log_tail` field: the most recent stdout/stderr the job has printed (an import error, a "downloading weights" line). That's the non-blocking peek - use it instead of holding `--follow` open, which a cold-start GPU job outlasts.

The fixture never ships in the deployed app - `gipity file url` serves the synced project copy, and deleting the file revokes the link. You only do this **once** to confirm the handler works; the wiring test above (which needs no fixture) is what belongs in your permanent suite.

## Common gotchas

- **GPU without Modal config** - submit returns `MODAL_NOT_CONFIGURED` (503). Operator must set `MODAL_APP_URL` + `MODAL_INVOKE_SECRET` on the server.
- **Demucs: split with the Python API, never the `demucs` CLI.** The model is preloaded, but the CLI's write step (and a bare `torchaudio.save`) routes through `torchcodec`, which is **not** in the image - so a CLI run separates fine and then fails at save. Run the model in-process and write stems with `soundfile` (libsndfile is installed). This is the whole 4-stem recipe:
  ```python
  import soundfile as sf, torch
  from demucs.pretrained import get_model
  from demucs.apply import apply_model
  from demucs.audio import AudioFile
  model = get_model("htdemucs"); model.eval()          # weights cache in /cache/torch
  wav = AudioFile("/work/song.mp3").read(streams=0, samplerate=model.samplerate, channels=model.audio_channels)
  ref = wav.mean(0); wav = (wav - ref.mean()) / ref.std()
  with torch.no_grad():
      stems = apply_model(model, wav[None], device="cuda")[0]  # [4, ch, samples]
  stems = stems * ref.std() + ref.mean()
  for name, stem in zip(model.sources, stems):           # ["drums","bass","other","vocals"]
      sf.write(f"/work/{name}.wav", stem.T.cpu().numpy(), model.samplerate)
  ```
- **Deps that conflict with the fat image** - e.g. pinning `torch==2.3` when the image ships a different minor. `requirements.txt` wins via pip override, but CUDA/torch mismatches may produce subtle bugs. Prefer no `deps:` if the fat image already has what you need.
- **Forgetting `set -e` in bash** - without it, only the LAST command's exit code propagates.
- **`output` is whatever you `set_output`** - for an agent or function calling `job_submit`, the run's `output` (and what an `on_complete` hook receives) is the value the handler passed to `ctx.set_output(...)`. See "Job output contract" above.