{
  "name": "web-vision-detect",
  "title": "Browser Object Detection (YOLOX)",
  "description": "High-accuracy on-device object detection for web apps: YOLOX on ONNX Runtime Web, custom-model support - the web-vision-detect kit",
  "guid": "sk_plat_wvdt",
  "category": "Templates",
  "requiredTools": [
    "add",
    "file_write",
    "project_deploy"
  ],
  "content": "# Browser Object Detection (YOLOX)\n\n`web-vision-detect` is a **kit** - a reusable building block added into an existing web app. It runs [YOLOX](https://github.com/Megvii-BaseDetection/YOLOX) (Apache-2.0) on ONNX Runtime Web so an app can read the camera (or a still image) and do **real-time object detection** entirely in the browser - boxes, labels, and confidence for the 80 COCO classes, or for a custom-trained model the user brings.\n\nOn-device: no server, no upload, the camera stream never leaves the device. Inference runs on **WebGPU** where the browser has it, with automatic **WASM (SIMD)** fallback. **Web only** - it needs `getUserMedia` (for camera use), WASM, and a canvas, so it runs only on HTTPS or `localhost`.\n\nThis kit is the high-accuracy sibling of `web-vision-mediapipe`: use that one for gesture and pose, this one when detection is the product - counting, labeling, inventory, custom classes.\n\n## Two ways in\n\n**Start a fresh detection app** - add the `object-spotter` starter, a fullscreen camera app with the kit pre-installed that boxes, labels, and counts objects live, detects picked photos, and switches between three speed/accuracy presets:\n\n```\nadd name=object-spotter title=\"...\"\n```\n\n**Add detection to an existing web app** - install the kit into it:\n\n```\nadd name=web-vision-detect\n```\n\nThis copies the kit to `src/packages/web-vision-detect/` and wires the import map in `src/index.html` (the kit specifier plus `onnxruntime-web`). There is no deploy phase - it is pure client-side, so a plain static app needs nothing else.\n\n## Using the kit\n\nThe whole job is two elements - a `<video>` for the camera and a `<canvas>` overlaying it - plus one call:\n\n```js\nimport { mountDetect } from '@gipity/web-vision-detect';\n\nconst vision = await mountDetect({\n  video:  document.querySelector('video'),\n  canvas: document.querySelector('canvas'),\n  model:  'nano',                             // 'nano' | 'tiny' | 's' | custom spec\n  camera: { facingMode: 'environment' },      // rear camera is the default here\n  onFps:  (fps) => { hud.textContent = `${fps} FPS`; },\n  onResult: ({ detections }) => { /* app logic - shape below */ },\n});\n\nawait vision.switchModel('s');     // trade frame rate for accuracy\nconst r = await vision.detect(img); // one-off detection on an <img> or canvas\nvision.stop();                      // release camera + free model memory\n```\n\nEach detection is `{ label, classId, score, box: { x, y, width, height } }` in source-frame pixels - drawing on a canvas sized to the frame lines up 1:1 (`mountDetect` already draws boxes; `onResult` is for app logic like counting).\n\nDetections arrive sorted by descending `score`, and suppression is **class-aware** - a box only suppresses an overlapping box of the *same* `classId`. So one real-world object can surface as several overlapping detections with different labels: an ambiguous animal yields both a `cat` box and a `dog` box. Don't assume one detection per object. When a class must not fire for a look-alike (`dog` but never `cat`), compare the overlapping boxes' scores and require a margin, rather than acting on the first matching label you find.\n\nFor a custom loop, compose the low-level exports instead: `createDetector`, `startCamera`, `createLoop`, `drawDetections`, plus pure-math `decodeYolox` / `decodeYolo` / `nms`. See `src/packages/web-vision-detect/examples/` and its `README.md`.\n\n## Models\n\n| `model` | Download | COCO mAP | Use when |\n|----------|----------|----------|----------|\n| `nano` (default) | 3.7 MB | 25.8 | Instant start, phones, casual demos |\n| `tiny` | 20 MB | 32.8 | Noticeably better accuracy, still fast |\n| `s` | 36 MB | 40.5 | Accuracy is the point; fine on WebGPU |\n\nPresets are official YOLOX exports hosted on the Gipity CDN, fetched on first use and browser-cached. **Custom models:** pass `model: { url, format, inputSize, labels }` - `format: 'yolox'` for YOLOX exports, `format: 'yolo'` for Ultralytics YOLOv8/v11 `model.export(format='onnx')`. This is the deploy path for \"I trained a detector on Roboflow/Ultralytics and want it in an app\".\n\n## Notes and common mistakes\n\n- **Pick the right vision kit.** Gesture or body pose -> `web-vision-mediapipe`. Detection accuracy, counting, or custom classes -> this kit. The MediaPipe kit's EfficientDet-Lite detector is demo-grade; this kit's `tiny`/`s` presets are meaningfully stronger.\n- **The canvas must overlay the video** at the same on-screen size. The kit sizes the canvas backing store to the camera frame; CSS `object-fit: cover` on *both* keeps the overlay aligned. A front camera reads naturally with `transform: scaleX(-1)` on the video only - pass `mirror` and the kit flips box geometry while captions stay upright.\n- **Camera needs a user gesture and a secure origin.** Call `mountDetect` from a click handler, not on page load, and deploy over HTTPS.\n- **Verify it with a real camera - `--camera` plays a picture as the webcam.** `gipity page eval <url> --camera ./street.jpg --wait 8000 \"<read the counts the app rendered>\"` (also on `page screenshot`/`inspect`) runs the app's actual camera → detection → `onResult` path against that image, so the labels and counts you assert are the app's own. Don't ship debug hooks on `window` to test this - see [app-debugging](app-debugging.md).\n- **Inference is async - never run it per rAF tick yourself.** Use the kit's `createLoop` (or `mountDetect`), which skips camera frames while an inference is in flight.\n- **First use downloads the runtime + model** (~13-26 MB of ONNX Runtime WASM shared across models, plus the model file), then everything is browser-cached. Expect a pause on the very first frame; tell users.\n- **Frame rate varies a lot by backend.** WebGPU runs `nano` at camera speed on most laptops and recent phones; plain WASM is several times slower. `result.backend` / `vision.currentBackend()` says which one loaded - stay on `nano` when it reports `wasm`.\n- **License:** YOLOX and the bundled presets are Apache-2.0, ONNX Runtime is MIT - free for commercial use, no copyleft obligation on the app. Ultralytics YOLO models are **AGPL-3.0** - the kit can load one as a custom model, but never bundle one into an app by default; flag the license to the user instead.\n"
}