Browser Vision (MediaPipe)

web-vision-mediapipe is a kit - a reusable building block added into an existing web app. It wraps Google MediaPipe Tasks so an app can read the camera and run gesture recognition, body pose, or object detection entirely in the browser.

On-device: no server, no upload, the camera stream never leaves the device. MediaPipe's own usage metrics (which its npm build sends to Google) are answered locally by the kit, so nothing else leaves either. Inference is WASM + WebGL-accelerated. Web only - it needs getUserMedia, WASM, and a canvas, so it runs only on HTTPS or localhost.

Two ways in

Start a fresh camera app - add the web-vision-cam app: a fullscreen camera app that already switches between gesture, object, and pose detection with a live FPS readout, kit pre-installed:

gipity add web-vision-cam --title "..."

Add vision to an existing web app - install the kit into it:

gipity add web-vision-mediapipe

This copies the kit to src/packages/web-vision-mediapipe/ and wires the import map in src/index.html (the kit specifier plus @mediapipe/tasks-vision). There is no deploy phase - it is pure client-side, so a plain static app needs nothing else.

Using the kit

The whole job is two elements - a <video> for the camera and a <canvas> overlaying it - plus one call:

import { mountVision } from '@gipity/web-vision-mediapipe';

const vision = await mountVision({
  video:  document.querySelector('video'),
  canvas: document.querySelector('canvas'),
  kind:   'gesture',                          // 'gesture' | 'detect' | 'pose'
  camera: { facingMode: 'user' },             // 'user' (front) | 'environment' (rear)
  onFps:  (fps) => { hud.textContent = `${fps} FPS`; },
  onGesture: (name) => playRound(name),       // gesture task: one event per deliberate throw
  onResult: (result, kind) => { /* raw per-frame result - see shapes below */ },
});

vision.gesture();                  // ...or ASK what the hand is holding right now (or null)
await vision.switchTask('pose');   // swap model, camera keeps running
await vision.flipCamera();         // front <-> rear
vision.stop();                     // release camera + free GPU memory

mountVision runs the camera, the inference loop, and the overlay drawing. Also on the handle: latest() (the newest frame's raw result), resetGesture(), detectFrom(source), setCamera(), hasMultipleCameras(), currentTask(), currentFacingMode(), currentMirror().

For a custom loop, compose the low-level exports instead: createTask, startCamera, createLoop, draw, fitCanvas, clearCanvas. See src/packages/web-vision-mediapipe/examples/ and its README.md.

Reading gestures - push or pull

Don't act on the raw per-frame label. The model re-classifies ~30x a second and a hand mid-throw passes through several labels, so an app built straight on onResult fires on noise. The kit ships a commit gate that settles it, readable two ways:

Push - onGesture(name) fires once, on the frame the pose settles, and not again until the hand changes. Right for "do a thing when the user throws a gesture": a shortcut, a menu pick, a shutter.

Pull - vision.gesture() returns what the hand is holding right now, once it has been held steady for holdMs - or null. Right for anything on the app's own clock; a "3, 2, 1, shoot!" countdown samples it at shoot:

const thrown = vision.gesture();   // 'Closed_Fist' | 'Open_Palm' | 'Victory' | ... | null
vision.resetGesture();             // drop the hold before the next round

Don't cache push events and read them as pull. onGesture deliberately won't re-fire while the hand is unchanged, so a player who throws rock two rounds running would score round two off a stale event. On a clock, pull.

Tune both with gestureHold: { holdMs = 500, minScore = 0.5, hand = 0 }. For a custom loop, GESTURES (the recognised list) and gestureName(result) (dig the label out of one raw result, None and sub-threshold scores included) are exported too.

Tasks and result shapes

kind selects the model. Each onResult / task.detect() value is the native MediaPipe result:

Recognised gestures: Thumb_Up, Thumb_Down, Open_Palm, Closed_Fist, Victory, Pointing_Up, ILoveYou. An unrecognised pose is reported as the literal category None, not an empty list - the gesture API above already turns that (and anything under minScore) into null, so only raw onResult consumers need to handle it. For a game: rock = Closed_Fist, paper = Open_Palm, scissors = Victory.

Notes and common mistakes

• Gesture is the strong task. Object detection uses EfficientDet-Lite - fast but modest accuracy. Good for a demo; do not promise production-grade detection. For high-accuracy detection, counting, or custom classes, use the web-vision-detect kit (YOLOX) instead.
• The canvas must overlay the video at the same on-screen size. The kit sizes the canvas backing store to the camera frame; CSS object-fit: cover on both keeps the overlay aligned. A front camera reads naturally with transform: scaleX(-1) on both.
• Mount on page load, not behind a click. The WASM runtime and the model start downloading the moment the module is imported, so that multi-MB fetch finishes while the user is still looking at the camera permission prompt. Keep an "Enable camera" button as the retry path for a browser that insists on a tap first - it simply rejects, and the gate stays up (this is what the web-vision-cam starter does). Gating the first mount behind a click throws away the overlap and blocks the headless check below. A secure origin is non-negotiable, though: getUserMedia fails on plain HTTP, so deploy over HTTPS.
• Verify it with a real camera - --camera plays a picture as the webcam. The kit publishes its own state for exactly this: <html data-vision="loading|ready|error|stopped"> and the live handle on window.__vision. So a headless check reads the deployed app's real camera → inference → gesture path with no webcam, no click, and no app-specific test hook - in one command:

gipity page eval <url> --camera ./palm.png --wait-for '[data-vision="ready"]' --wait-timeout 25000 \
  "(async()=>{ const v = window.__vision; const end = Date.now() + 3000; while (!v.gesture() && Date.now() < end) await new Promise(r => setTimeout(r, 100)); return { held: v.gesture(), hands: v.latest().gestures.length }; })()"
# -> {"held":"Open_Palm","hands":1}
gipity page screenshot <url> --camera ./palm.png --wait-for '[data-vision="ready"]'   # see the round play out

Wait on [data-vision="ready"] (first frame drawn) rather than guessing a fixed --wait, and give it room on a cold load (--wait-timeout defaults to 5s, max 30s). gesture() needs the pose held for 500ms, so it is still null on the first frame: the loop above polls for it. Read the answer off both fields: hands: 0 means the model found no hand in the picture (the picture is at fault - swap it, don't wait longer or debug the app); hands: 1 with held: null means it sees a hand but no pose it knows. A plain page load with no --camera lands on data-vision="error" - that is the app working, not a bug. Don't add your own debug hooks on window to test this (app-debugging) - window.__vision is the kit's own API surface, already there.

• The fixture picture decides whether verification works - frame it like a webcam sees a person. The hand model looks for a hand on a person at webcam distance. A studio close-up of a hand filling the frame on a plain background reads as zero hands, silently - real photo or generated. What works: a person with their upper body in frame, one hand raised beside the shoulder, palm toward the camera. gipity generate image makes usable fixtures when you ask for exactly that:

gipity generate image "candid smartphone photo of a person standing in a living room raising one open hand next to their shoulder, palm facing the camera, five fingers spread, upper body visible, natural daylight, realistic" -o tmp/palm.png

Swap the pose phrase for the gesture you need: "making a peace sign, palm facing the camera" reads Victory, "giving a thumbs up" reads Thumb_Up. Closed_Fist is the exception: generated fists usually come back as a hand with no gesture (held: null, hands: 1). For a fist, use a real photo (ask the user for one), or verify the app's camera → gesture → UI path with the poses that do read and cover the fist branch with a unit test of your own mapping. Vet several candidates in one run before relying on them - --fixture hosts each file and detectFrom runs the model on it with no app wiring in the way:

gipity page eval <url> --camera tmp/palm.png --fixture tmp/palm.png --fixture tmp/peace.png --fixture tmp/thumb.png --wait-for '[data-vision="ready"]' --wait-timeout 25000 --timeout 60000 \
  "(async()=>{ const out = {}; for (const [name, url] of Object.entries(fixtures)) { const r = await window.__vision.detectFrom(url); out[name] = r.gestures.map(g => g[0].categoryName); } return out; })()"
# -> {"palm.png":["Open_Palm"],"peace.png":["Victory"],"thumb.png":["Thumb_Up"]}   ([] = no hand found, ["None"] = hand but no known pose)

• One detect() per frame. Timestamps must strictly increase; mountVision/createLoop already handle this. Do not call task.detect() twice for the same frame.
• First use downloads the model (~3-8 MB) from Google's CDN, then it is browser-cached - expect a short delay on the first frame of each task.
• License: MediaPipe and its default models are Apache-2.0 - free for commercial use, no copyleft obligation on the app.