{
  "name": "app-audio",
  "title": "App API - Audio Services (Sound Effects, Music, Transcription)",
  "description": "Sound effects, music generation, and audio transcription services for deployed apps",
  "guid": "sk_plat_aaud",
  "category": "App services",
  "requiredTools": [
    "project_settings"
  ],
  "content": "# App API - Audio Services (Sound Effects, Music, Transcription)\n\nThree audio services, available on every project with no setup (user_pays by default).\n\n> **Plan availability (Free plan):** **Music** generation is **Pro-only** (blocked on Free). **Sound effects** count toward a shared **3-uses-per-month** audio allowance with text-to-speech (see [app-tts](app-tts.md)). **Transcription** is not gated. When a Free-plan owner hits a block, the endpoint returns **403 FORBIDDEN** with an upgrade message - handle it in the app's UI. All limits lift on Pro.\n\nUse `project_settings` to customize each independently (optional).\n\n> To make a billing-mode choice reproducible (it ships with the app instead of living as out-of-band server state), declare it in a `services` deploy phase in `gipity.yaml` rather than only via `project_settings` - e.g. `{ service: transcribe, billing_mode: owner_pays }` (also `sound` / `music`). See [deploy](deploy.md) and [app-llm](app-llm.md).\n\n> **Billing modes only govern the deployed app's runtime calls.** Direct generation during development - `gipity generate sound|music`, `gipity service call`, or the agent's own generation tools - always bills the caller (you), whatever the service's `billing_mode` says. **Never flip a service to `owner_pays` just to generate assets**: it's unnecessary, and while flipped the live app accepts anonymous generation on your credits.\n\n---\n\n## Sound Effects\n\nGenerate sound effects from text descriptions.\n\n### Endpoint\n`POST /api/<PROJECT_GUID>/services/sound`\n\n### Request\n```json\n{\n  \"text\": \"thunder rumbling in the distance\",\n  \"duration_seconds\": 5,\n  \"prompt_influence\": 0.5\n}\n```\n- `text` (required): Description of the sound, max 1,000 chars\n- `duration_seconds`: 0.5-30 (optional, provider decides if omitted)\n- `prompt_influence`: 0-1, how closely to follow the prompt (optional)\n\n### Response\n```json\n{\n  \"url\": \"https://media.gipity.ai/med_abc12345.mp3\",\n  \"duration_seconds\": 5,\n  \"credits_used\": 2\n}\n```\n\n### CLI\nFor one-off sound effects during development (game SFX, UI sounds, voice-like cries), skip the HTTP call and use `gipity generate sound` - it bills you directly and ignores the service's billing mode. **It writes to `./sound.mp3` by default** - pass `-o <path>` to land the clip in your source tree so it deploys.\n```bash\ngipity generate sound \"cartoon character saying oof\" -o src/assets/sounds/oof.mp3\ngipity generate sound \"thunder rolling in the distance\" --duration 5 -o src/assets/sounds/thunder.mp3\n```\n\n---\n\n## Music Generation\n\nGenerate music from text prompts - instrumental tracks, or full songs singing lyrics you supply.\n\n### Endpoint\n`POST /api/<PROJECT_GUID>/services/music`\n\n### Request\n```json\n{\n  \"prompt\": \"upbeat lo-fi hip hop beat with piano and soft drums\",\n  \"duration_seconds\": 30,\n  \"instrumental\": true\n}\n```\n- `prompt` (required): Music description, max 2,000 chars. With `lyrics`, keep this to style only (genre, mood, instruments, tempo).\n- `duration_seconds`: 3-600 (optional, default ~30s; each model has its own max)\n- `instrumental`: true to force no vocals (optional; cannot be combined with `lyrics`)\n- `lyrics`: exact words for the vocalist to sing (optional, max 5,000 chars). Section tags like `[Verse]` / `[Chorus]` are understood. Only models with `supports_lyrics` accept this - sending it to one that doesn't is a 400 naming the models that do.\n- `model`: which music model to use (optional; omit for the default). Get the list from the models endpoint below - don't hardcode ids.\n\nLyrics example - a real song with your words, not a description of one:\n```json\n{\n  \"prompt\": \"1985 old-school hip hop, TR-808 drums, funk bassline, confident female MC\",\n  \"lyrics\": \"[Verse]\\nThese exact words get sung\\n[Hook]\\nAnd this hook comes back around\",\n  \"duration_seconds\": 120\n}\n```\nSize `duration_seconds` to the lyric (roughly 10s per short section); too-short durations squeeze the delivery.\n\n### Response\n```json\n{\n  \"url\": \"https://media.gipity.ai/med_abc12345.mp3\",\n  \"duration_seconds\": 30,\n  \"model\": \"music-v1\",\n  \"credits_used\": 3\n}\n```\n\n### Picking a model\n\n`GET /api/<PROJECT_GUID>/services/music/models` lists what's available:\n```json\n{\n  \"data\": {\n    \"models\": [\n      { \"id\": \"music-v1\", \"label\": \"Studio\", \"description\": \"Fast, polished songs and loops.\", \"max_duration_s\": 600, \"supports_lyrics\": true }\n    ],\n    \"default_model\": \"music-v1\"\n  }\n}\n```\nUse the `id` as the `model` value on the generate call. For a model picker in the UI, render `label` + `description` and send the chosen `id`. `supports_lyrics` says whether the model can sing exact `lyrics`. Models differ in sound and cost; the platform handles where each one runs.\n\n### CLI\nFor one-off music during development (saves the clip to a local file), skip the HTTP call and use `gipity generate music`. **It writes to `./music.mp3` by default** - pass `-o <path>` to land the clip in your source tree so it deploys.\n```bash\ngipity generate music \"chill lo-fi beat for studying\" -o src/assets/audio/lofi.mp3\ngipity generate music \"epic orchestral battle theme\" --duration 60 -o src/assets/audio/theme.mp3\ngipity generate music \"80s synth-pop, female vocals\" --lyrics-file lyrics.txt --duration 120 -o song.mp3\ngipity service call music/models --get   # list available models\n```\n\n---\n\n## Audio Transcription\n\nTranscribe audio files to text, with word-level timestamps and optional speaker diarization.\n\n> **Video or a file over 100MB?** The cap is on the audio you send here, not on what you start from. Extract the audio first with ffmpeg in the sandbox (`gipity sandbox run`) - e.g. `ffmpeg -i input.mp4 -vn -ac 1 -ar 16000 -b:a 64k audio.mp3` turns hours of video into a few MB - then send that audio to this endpoint. The sandbox accepts inputs up to several GB, so the original file size is never the constraint.\n\n### Endpoint\n`POST /api/<PROJECT_GUID>/services/transcribe` (multipart/form-data)\n\n### Request\nSend as `multipart/form-data`:\n- `audio` (required): Audio file (MP3, WAV, M4A, etc.), max 100MB\n- `provider`: \"elevenlabs\" (default) or \"openai\"\n- `language`: Language code (e.g., \"en\", \"es\") - optional, auto-detected\n- `diarize`: \"true\" to identify speakers (optional)\n\n### Response\n```json\n{\n  \"text\": \"Hello, this is a transcription test.\",\n  \"words\": [{ \"text\": \"Hello\", \"start\": 0.0, \"end\": 0.5, \"type\": \"word\" }],\n  \"language\": \"en\",\n  \"duration_seconds\": 12.5,\n  \"provider\": \"elevenlabs\",\n  \"credits_used\": 5\n}\n```\n\n---\n\n## Client Code Examples\n\n```js\n// Get token first\nconst tokenRes = await fetch('https://a.gipity.ai/api/token', {\n  method: 'POST',\n  headers: { 'Content-Type': 'application/json' },\n  body: JSON.stringify({ app: '<PROJECT_GUID>' })\n});\nconst { data: { token } } = await tokenRes.json();\n\n// Sound effect\nconst soundRes = await fetch('https://a.gipity.ai/api/<PROJECT_GUID>/services/sound', {\n  method: 'POST',\n  headers: { 'Content-Type': 'application/json', 'X-App-Token': token },\n  body: JSON.stringify({ text: 'door creaking open slowly' })\n});\nconst sound = await soundRes.json();\nnew Audio(sound.url).play();\n\n// Music\nconst musicRes = await fetch('https://a.gipity.ai/api/<PROJECT_GUID>/services/music', {\n  method: 'POST',\n  headers: { 'Content-Type': 'application/json', 'X-App-Token': token },\n  body: JSON.stringify({ prompt: 'calm ambient piano', duration_seconds: 60 })\n});\nconst music = await musicRes.json();\nnew Audio(music.url).play();\n\n// Transcription (from file input)\nconst formData = new FormData();\nformData.append('audio', fileInput.files[0]);\nformData.append('diarize', 'true');\nconst transRes = await fetch('https://a.gipity.ai/api/<PROJECT_GUID>/services/transcribe', {\n  method: 'POST',\n  headers: { 'X-App-Token': token },\n  body: formData\n});\nconst transcript = await transRes.json();\nconsole.log(transcript.text);\n```\n\n## Limits\n- **Rate limit**: 600 requests per 5-minute window (per IP, all audio endpoints)\n- **Sound text**: max 1,000 chars, duration 0.5-30s, timeout 60s\n- **Music prompt**: max 2,000 chars, duration 3-600s, timeout 120s\n- **Transcription**: max 100MB file, timeout 120s\n- Standard `RateLimit-*` headers included in responses"
}