{
  "name": "app-tts",
  "title": "App API - Text-to-Speech Service",
  "description": "Text-to-speech service for deployed apps: Gipity Speech (30 voices, multi-speaker, 60+ languages)",
  "guid": "sk_plat_atts",
  "category": "App services",
  "requiredTools": [
    "project_settings"
  ],
  "content": "# App API - Text-to-Speech Service\n\n> This is the **app service** a deployed app calls over HTTPS. To generate a speech file while you build, use `gipity generate speech`.\n\nTTS is available for every project with no setup needed (user_pays billing by default).\n\n> **Plan availability:** Speech and sound-effect generation share one **3-uses-per-month** allowance on the Free plan (combined), then unlimited on Pro. When a Free-plan owner exceeds the cap, this endpoint returns **403 FORBIDDEN** with an upgrade message - handle that response in the app's UI.\n\nUse `project_settings` to customize (optional):\n- Switch billing mode (owner_pays ↔ user_pays)\n- Set default provider and voice\n\n> To make the billing choice ship with the app (reproduced on every deploy instead of living as out-of-band server state), declare it in a `services` deploy phase in `gipity.yaml` instead of only via `project_settings` - e.g. `{ service: tts, billing_mode: owner_pays }`. See [deploy](deploy.md) and [app-llm](app-llm.md).\n\n> **Billing modes only govern the deployed app's runtime calls.** Direct generation during development - `gipity generate speech`, `gipity service call tts`, or the agent's own speech tools - always bills the caller (you), whatever the service's `billing_mode` says. **Never flip a service to `owner_pays` just to generate assets**: it's unnecessary, and while flipped the live app accepts anonymous generation on your credits.\n\n## Providers\n- **elevenlabs**: ElevenLabs (many premium voices)\n- **openai**: OpenAI (alloy, ash, ballad, coral, echo, fable, nova, onyx, sage, shimmer, verse)\n- **gemini**: Gemini (30 voices: Kore, Puck, Zephyr, Charon, Fenrir, Leda, Orus, Aoede, and 22 more). Multi-speaker (up to 2) and 60+ languages\n\n## Endpoints\n- `GET /api/<PROJECT_GUID>/services/tts/voices` - list available voices\n- `POST /api/<PROJECT_GUID>/services/tts` - generate speech audio\n\n## Listing Voices\n```\nGET /api/<PROJECT_GUID>/services/tts/voices?provider=elevenlabs\n```\nReturns `{ data: { voices: [...], provider, available_providers } }`\n\n## Request Format (POST /tts)\n```json\n{\n  \"text\": \"Hello, welcome to our app!\",\n  \"voice_id\": \"JBFqnCBsd6RMkjVDRZzb\",\n  \"provider\": \"elevenlabs\",\n  \"model\": \"eleven_flash_v2_5\"\n}\n```\n\nFields:\n- `text` (required): Text to speak. **Per-provider cap** (see Limits): OpenAI **4096** chars, ElevenLabs/Gemini 5,000. Over the cap you get a clear 400 - for long text, **chunk by sentence** and synthesize each (see \"Reading long text\" below), don't send one giant request.\n- `voice_id`: Voice to use (default depends on provider - OpenAI `alloy`, ElevenLabs `dfeOmy6Uay63tNhyO99j` / Kristen)\n- `provider`: \"openai\" (default - cheapest), \"elevenlabs\", or \"gemini\"\n- `model`: Provider-specific model ID (optional)\n- `language`: BCP-47 language code (Gemini only, e.g. \"ja-JP\", \"es-ES\"). 60+ languages\n- `speakers`: Multi-speaker config (Gemini only, up to 2). Array of `{ name, voice }`. Text must use \"Name: dialogue\" format per line\n- `include_timestamps`: `true` to also get **word-level timing** for read-along / word highlighting (see \"Read-along\" below). Routes to ElevenLabs automatically (only it returns timing); pinning `provider:\"openai\"` + `include_timestamps` is a 400.\n\n## Gemini TTS Details\n\n**30 voices**: Zephyr, Puck, Charon, Kore, Fenrir, Leda, Orus, Aoede, Callirrhoe, Autonoe, Enceladus, Iapetus, Umbriel, Algieba, Despina, Erinome, Algenib, Rasalgethi, Laomedeia, Achernar, Alnilam, Schedar, Gacrux, Pulcherrima, Achird, Zubenelgenubi, Vindemiatrix, Sadachbia, Sadaltager, Sulafat\n\n**Multi-speaker example** (up to 2 speakers):\n```json\n{\n  \"text\": \"Joe: Hey, how are you?\\nJane: Great, thanks!\",\n  \"provider\": \"gemini\",\n  \"speakers\": [\n    { \"name\": \"Joe\", \"voice\": \"Charon\" },\n    { \"name\": \"Jane\", \"voice\": \"Leda\" }\n  ]\n}\n```\n\n**Language example** (Japanese):\n```json\n{\n  \"text\": \"こんにちは世界\",\n  \"provider\": \"gemini\",\n  \"voice_id\": \"Kore\",\n  \"language\": \"ja-JP\"\n}\n```\n\nOutput format is raw PCM audio (audio/L16, 24kHz). The platform converts and serves as MP3.\n\n## Response Format\n```json\n{\n  \"url\": \"https://media.gipity.ai/med_abc12345.mp3\",\n  \"voice_id\": \"JBFqnCBsd6RMkjVDRZzb\",\n  \"provider\": \"elevenlabs\",\n  \"credits_used\": 5\n}\n```\n\nThe `url` is a permanent public CDN URL to an MP3 file.\n\n### CLI\nFor one-off speech during development (downloads the result to a local file), skip the HTTP call and use `gipity generate speech`. **It writes to `./speech.mp3` by default** - pass `-o <path>` to land the file in your source tree so it deploys.\n```bash\ngipity generate speech \"Welcome to Gipity!\" -o src/assets/sounds/intro.mp3\ngipity generate speech \"こんにちは世界\" --provider gemini --voice Kore --language ja-JP -o src/assets/sounds/greeting.mp3\n```\n\n## Client Code Example\n```js\nconst tokenRes = await fetch('https://a.gipity.ai/api/token', {\n  method: 'POST',\n  headers: { 'Content-Type': 'application/json' },\n  body: JSON.stringify({ app: '<PROJECT_GUID>' })\n});\nconst { data: { token } } = await tokenRes.json();\n\nconst res = await fetch('https://a.gipity.ai/api/<PROJECT_GUID>/services/tts', {\n  method: 'POST',\n  headers: { 'Content-Type': 'application/json', 'X-App-Token': token },\n  body: JSON.stringify({ text: 'Welcome to the future of AI!' })\n});\nconst data = await res.json();\n\n// Play audio\nconst audio = new Audio(data.url);\naudio.play();\n```\n\n## Always show a loading state\n\nSynthesis takes a second or more, and reading long text means several requests. **Never leave the button dead** - flip it to a spinner / \"Loading…\" the moment it's tapped and back when audio starts. A silent delay reads as \"broken.\" (Same goes for the LLM call that produced the text.)\n\n## Reading long text (chunk by sentence)\n\nA single request is capped per provider (OpenAI 4096, ElevenLabs/Gemini 5,000). Anything longer - a story, an article, a multi-paragraph reply - must be **split into sentence-sized chunks**, each synthesized and played in order. This removes the ceiling entirely and lets playback start on the first chunk while later ones synthesize:\n\n```js\n// Split into chunks that stay under the cap (sentence boundaries, not mid-word).\nfunction chunkForTts(text, max = 600) {\n  const sentences = text.match(/[^.!?]+[.!?]+|\\s*\\S+\\s*$/g) || [text];\n  const chunks = []; let cur = '';\n  for (const s of sentences) {\n    if ((cur + s).length > max) { if (cur) chunks.push(cur.trim()); cur = s; }\n    else cur += s;\n  }\n  if (cur.trim()) chunks.push(cur.trim());\n  return chunks;\n}\n\nasync function readAloud(text, appGuid, token) {\n  for (const chunk of chunkForTts(text)) {\n    const res = await fetch(`https://a.gipity.ai/api/${appGuid}/services/tts`, {\n      method: 'POST',\n      headers: { 'Content-Type': 'application/json', 'X-App-Token': token },\n      body: JSON.stringify({ text: chunk }),\n    });\n    const { url } = await res.json();\n    const audio = new Audio(url);\n    await new Promise((resolve, reject) => { audio.onended = resolve; audio.onerror = reject; audio.play().catch(reject); });\n  }\n}\n```\n\n## Read-along (word highlighting)\n\nFor karaoke-style highlighting of the word currently being spoken, pass `include_timestamps: true`. The response adds a `words` array of `{ word, start, end }` (seconds). Highlight by matching the audio's `currentTime` against the ranges - no extra processing, no GPU job:\n\n```json\n{\n  \"url\": \"https://media.gipity.ai/med_abc12345.mp3\",\n  \"words\": [ { \"word\": \"Once\", \"start\": 0.0, \"end\": 0.31 }, { \"word\": \"upon\", \"start\": 0.31, \"end\": 0.52 } ],\n  \"voice_id\": \"dfeOmy6Uay63tNhyO99j\",\n  \"provider\": \"elevenlabs\",\n  \"credits_used\": 5\n}\n```\n\n```js\nconst { url, words } = await (await fetch(ttsUrl, { /* ...include_timestamps: true... */ })).json();\nconst audio = new Audio(url);\naudio.ontimeupdate = () => {\n  const t = audio.currentTime;\n  const i = words.findIndex(w => t >= w.start && t < w.end);\n  highlightWord(i); // toggle a class on your per-word <span>s\n};\naudio.play();\n```\n\nCombine with chunking for long text: request each chunk with `include_timestamps: true` and offset each chunk's word times by the running audio duration.\n\n**Word highlighting: which tool?** Use `include_timestamps` (above) for **on-demand TTS** you generate in the app - it's instant, free, and exact for synthesized speech. The separate `audio-align` **kit** is different: it's a Gipity Jobs GPU **batch job** (~tens of seconds + cost per run) that force-aligns lyrics to a **pre-existing** audio file (e.g. a song you uploaded). Reach for the kit only when you have audio you didn't synthesize here; for read-aloud of generated text, `include_timestamps` is the right call.\n\n## Limits\n- **Rate limit**: 600 requests per 5-minute window (per IP)\n- **Max text length (per request)**: OpenAI **4096** chars, ElevenLabs/Gemini 5,000. Over the cap → 400 with the exact limit; chunk by sentence for longer content.\n- **Timeout**: 60s\n- Standard `RateLimit-*` headers included in responses"
}