{
  "name": "app-llm",
  "title": "App API - LLM Service",
  "description": "LLM service for deployed apps: request/response format, code examples, and billing config",
  "guid": "sk_plat_allm",
  "category": "App services",
  "requiredTools": [
    "project_settings"
  ],
  "content": "# App API - LLM Service\n\n> This is the **app service** a deployed app calls. For a one-off model call while you build, use `gipity ask`.\n>\n> **Building a chatbot/assistant UI?** Don't hand-roll the message thread, streaming, and input box on top of this service - `add` the [chatbot](chatbot.md) kit. It drops in a persona, scope guardrails, static knowledge, and a streaming bubble/headless engine wired to this LLM service, no API keys. Use this raw service only when you need a non-chat LLM call.\n\nThe LLM service is available for every project with no setup (user_pays billing by default) - apps can call AI models immediately after deploy.\n\nUse `project_settings` to change settings (optional):\n- Switch billing mode (user_pays ↔ owner_pays)\n- Restrict allowed models\n- Set max token cap\n- Disable the service entirely\n\n## Billing Modes\n- **user_pays** (default): Each app user pays from their own Gipity credits, so **publishing an app never silently bills you for what your users consume** - it's safe to share. Requires the caller to be signed in (Sign in with Gipity + LLM consent), so calls must come from the browser with a logged-in user. *Load `app-auth` for the auth flow.*\n- **owner_pays**: Your credits are consumed when anyone (incl. anonymous/logged-out users, and serverless functions) calls the LLM. Simpler - no user login needed. Switch to this to subsidize usage or to call the LLM **from a serverless function** (a function has no browser session, so a user_pays call from it throws - see below).\n\n**Set it in source, not just via the tool.** `project_settings` flips billing as a one-off, but that lives as server state that does *not* travel with your app - rebuild into a fresh project or fork it and it reverts to `user_pays`. To make the choice reproducible, declare it in a `services` deploy phase in `gipity.yaml` so every deploy reproduces it (the same applies to the `image`, `tts`, `transcribe`, etc. services):\n```yaml\n- name: services\n  type: services\n  service_definitions:\n    - service: llm\n      billing_mode: owner_pays\n```\nSee [deploy](deploy.md) for the full `services` phase. Billing is project-level (one setting for dev and prod). The phase owns only the billing mode; model/token restrictions set via `project_settings` are preserved.\n\n> **Billing modes only govern the deployed app's runtime calls.** Direct calls during development - `gipity service call`, `gipity generate <image|video|speech|sound|music>`, or `gipity ask` - always bill the caller (you): a logged-in owner is an identified payer even under `user_pays`. **Never flip a service to `owner_pays` just to generate assets or test an endpoint** - it's unnecessary, and while flipped the live app accepts anonymous usage on your credits.\n\n## Configuration Options\n- `allowed_models`: Restrict which models apps can use (e.g. only allow cheap models)\n- `max_tokens`: Cap output tokens (default 4096)\n- `default_model`: Model used when app doesn't specify one (default: gpt-5.4-mini)\n\n## Choosing a Model\n\n**Recommended: use an alias, not a concrete id.** Set `model` to a capability tier and we keep it pointed at the current best model in that tier for you — when the model catalog is refreshed, your app rides along automatically with no code change:\n\n- gipity-default → claude-sonnet-5\n- small → claude-haiku-4-5\n- fast → claude-haiku-4-5\n- medium → claude-sonnet-5\n- large → claude-opus-5\n- xlarge → claude-opus-5\n- thinking → claude-opus-5\n- haiku → claude-haiku-4-5\n- sonnet → claude-sonnet-5\n- opus → claude-opus-5\n- fable → claude-fable-5\n- sol → gpt-5.6-sol\n- terra → gpt-5.6-terra\n- luna → gpt-5.6-luna\n- gpt → gpt-5.6-sol\n- codex → gpt-5.6-sol\n- deepseek → deepseek/deepseek-v4-pro\n- qwen → qwen/qwen3-coder-next\n- kimi → moonshotai/kimi-k2.7-code\n- glm → z-ai/glm-5.2\n- pro → gemini-3.1-pro-preview\n- flash → gemini-3.5-flash\n\n`small`/`fast` are the cheapest, lowest-latency tier; `medium` is the balanced default; `large`/`xlarge`/`thinking` are the strongest. Most apps should just pick `fast` or `medium` and forget about it.\n\n**Or pin a concrete id** (from the list below) if you need a specific model and want to freeze its behavior. The trade-off: a pinned id is yours to maintain — if that model is later retired from the catalog, calls will return `400 Unknown model` until you update it. Aliases never have this problem.\n\n## Available Models\nclaude-fable-5, claude-opus-5, claude-opus-4-8, claude-opus-4-7, claude-opus-4-6, claude-opus-4-5, claude-sonnet-5, claude-sonnet-4-6, claude-sonnet-4-5, claude-haiku-4-5, gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna, gpt-5.5, gpt-5.5-pro, gpt-5.4, gpt-5.4-pro, gpt-5.4-mini, gpt-5.4-nano, gemini-3.6-flash, gemini-3.5-flash, gemini-3.1-pro-preview, gemini-3.1-flash-lite, gemini-3-flash-preview, gemini-2.5-flash, deepseek/deepseek-v4-pro, deepseek/deepseek-v4-flash-0731, qwen/qwen3-coder-next, qwen/qwen3.7-flash, moonshotai/kimi-k2.7-code, z-ai/glm-5.2\n\n## Endpoints\n- `GET /api/<PROJECT_GUID>/services/llm/models` - list available models\n- `POST /api/<PROJECT_GUID>/services/llm` - call the LLM\n\n## Request Format\n\nThe endpoint accepts OpenAI-compatible messages:\n\n- `messages`: Array of `{role, content}`. Roles: `system`, `user`, `assistant`. Max 20 messages.\n- `prompt`: Shorthand for a single user message (alternative to `messages`). Max 32,000 chars.\n- `image`: Base64 image with prompt (`{ data, media_type }`). Max 5 MB.\n- `model`: Override default model. Prefer an alias (`fast`, `medium`, `large`, …) over a concrete id — see [Choosing a Model](#choosing-a-model).\n- `system_prompt`: Custom system instructions (top-level field, takes precedence over `system` role messages).\n- `temperature`: 0-2 (creativity).\n- `max_tokens`: Output limit (capped at 4096).\n- `stream`: true for SSE streaming, false (default) for JSON response.\n\n### Image Support\n\nBoth formats are accepted in message content arrays:\n\n```js\n// OpenAI format (image_url with data URI)\n{ type: 'image_url', image_url: { url: 'data:image/png;base64,iVBOR...' } }\n\n// Native format\n{ type: 'image', data: 'iVBOR...', media_type: 'image/png' }\n```\n\nOnly `data:` URIs are supported - external image URLs will return a 400 error.\n\n> **Testing your vision/OCR function from the CLI?** Don't hand-roll base64. `gipity fn call <name> --file image=@receipt.png` reads the file, base64-encodes it, and attaches it under `image` in the request body for you (repeatable; combine with an inline JSON body). Same flag on `gipity job run`. See [app-testing](app-testing.md).\n\n## Response Format (OpenAI-compatible)\n\nNon-streaming:\n```json\n{\n  \"id\": \"chatcmpl-abc123\",\n  \"object\": \"chat.completion\",\n  \"model\": \"gpt-5.4-mini\",\n  \"choices\": [{ \"index\": 0, \"message\": { \"role\": \"assistant\", \"content\": \"...\" }, \"finish_reason\": \"stop\" }],\n  \"usage\": { \"prompt_tokens\": 100, \"completion_tokens\": 50, \"total_tokens\": 150 },\n  \"provider\": \"anthropic\",\n  \"credits_used\": 5\n}\n```\n\nStreaming (SSE):\n- Content chunks: `data: {\"choices\":[{\"delta\":{\"content\":\"...\"}}]}`\n- Final chunk: `finish_reason: \"stop\"` with `usage` and `credits_used`\n- Terminator: `data: [DONE]`\n\n## Client Code Example (Non-Streaming) - use the SDK\n\nThe Gipity client SDK (auto-injected into every template as `<script ... data-app>`) exposes `Gipity.service('llm', body)`, which is the one right way to call the LLM from the browser. It reads the app guid + API base off its own script tag, mints and caches the app token, sends the signed-in user's session cookie (so `user_pays` resolves the payer), and returns the parsed OpenAI-compatible body - or **throws** on non-2xx with the server's message and `err.status`/`err.code` set. **Don't hand-roll `fetch('/api/token')` + `X-App-Token`** - that's the low-level path the SDK already does for you (and hardcoding `a.gipity.ai` breaks on any non-prod instance).\n\n```js\nconst data = await Gipity.service('llm', {\n  messages: [\n    { role: 'system', content: 'Answer concisely.' },\n    { role: 'user', content: 'What is the capital of France?' }\n  ],\n  model: 'fast'   // alias — rides the catalog refresh. Or pin a concrete id.\n});\nconst answer = data.choices[0].message.content; // \"The capital of France is Paris.\"\n```\n\nThe same helper reaches every app service - `Gipity.service('image', {...})`, `Gipity.service('tts', {...})`, `Gipity.service('payments/checkout', {...})` - always returning the result object (never `undefined`). To send an image for vision/OCR, pass it exactly as the [Image Support](#image-support) body shape below.\n\n> **Need raw `fetch` (e.g. SSE streaming, which the SDK doesn't unwrap)?** Read the API base and app guid off the SDK script tag - `const sdk = document.querySelector('script[data-api-base]'); const apiBase = sdk.dataset.apiBase; const app = sdk.dataset.app;` - **never hardcode `a.gipity.ai`** (correct only on prod). Mint a token with `const { data: { token } } = await (await fetch(`${apiBase}/api/token`, { method:'POST', headers:{'Content-Type':'application/json'}, body: JSON.stringify({ app }) })).json();` (the token is nested under `data`) and send it as the `X-App-Token` header. The streaming example below shows the full shape.\n\n## Handling out of credits (402) - don't fail silently\n\nWhen the paying user has no credits left, the call returns **HTTP 402** with `{ error: { code: \"INSUFFICIENT_CREDITS\", message } }` - the message carries a top-up link. `Gipity.service` surfaces this as a thrown error with `err.status === 402`, so **wrap the call in try/catch** and show the user something; an LLM feature that just does nothing on 402 is the #1 cause of a confusing \"it broke and I don't know why\" wall.\n\n```js\ntry {\n  const data = await Gipity.service('llm', { prompt: 'Summarize this note', model: 'fast' });\n  // …use data.choices[0].message.content\n} catch (err) {\n  if (err.status === 402) {\n    // Out of credits. Under user_pays (the default) it's THIS signed-in user who's\n    // out - err.message includes the top-up link. Show it instead of a blank.\n    showNotice(err.message); // e.g. \"Insufficient credits. Buy more at https://prompt.gipity.ai/pricing\"\n  } else if (err.status === 401) {\n    // Not signed in - send them through Sign in with Gipity (app-auth).\n    await Gipity.auth.signIn();\n  } else {\n    throw err;\n  }\n}\n```\n\n- **Streaming requests get a clean 402 too.** The credit check runs *before* the stream opens, so an out-of-credits streaming call comes back as a normal 402 JSON body - not a mid-stream error chunk. Check `res.ok` before you start reading the SSE body (streaming uses raw `fetch`, not `Gipity.service`).\n- **Who's out of credits depends on billing mode.** Under `user_pays` (default) it's the signed-in end user, who tops up their own Gipity credits at the link. Under `owner_pays` it's *you*, the app owner - run `gipity credits buy` (or top up at the pricing page) to refill.\n- **What still works at zero credits:** `location` and `notify` are not credit-gated, so those keep working - only the AI/media services (llm, image, video, tts, audio, transcribe) stop. So \"out of credits\" degrades one feature, it doesn't break the whole app.\n\n## Calling from a serverless function (server-side)\n\nInside a function, **don't** `fetch` this endpoint - `a.gipity.ai` is blocked from the sandbox. The LLM is injected as a capability on the function's second argument, called server-to-server with no token and no `fetch`:\n\n```js\nexport default async function ask(ctx, { llm }) {\n  const { text, usage, credits_used } = await llm({\n    model: 'fast',   // alias — rides the catalog refresh. Or pin a concrete id.\n    messages: [{ role: 'user', content: ctx.body.question }],\n    // same fields as the HTTP body: prompt | messages, system_prompt, temperature, max_tokens, image\n  });\n  return { answer: text };\n}\n```\n\n```yaml\n- name: ask\n  auth: public\n  services: ['llm']      # required - without it, llm() throws\n```\n\n- Returns `{ text, model, provider, usage: { input_tokens, output_tokens }, credits_used }` (non-streaming only - functions return JSON, so there's no `stream` option here).\n- **Calling the LLM from a function requires owner_pays.** The default is now `user_pays`, and a function has no browser session for user_pays consent, so an LLM call from a function **throws** until you switch the `llm` service to `owner_pays` (via `project_settings`). Either set owner_pays, or make the LLM call from the browser with a signed-in user.\n- Capped at `max_llm_calls` per invocation (default 5). See [app-development](app-development.md) for the full injected-services list.\n\n## Client Code Example (Streaming)\n\nStreaming needs raw `fetch` (the SDK's `Gipity.service` reads the whole JSON body, so it can't stream). Read the API base + app guid off the SDK tag and mint a token as shown in the note above:\n\n```js\nconst sdk = document.querySelector('script[data-api-base]');\nconst apiBase = sdk.dataset.apiBase, app = sdk.dataset.app;   // never hardcode a.gipity.ai\nconst { data: { token } } = await (await fetch(`${apiBase}/api/token`, {\n  method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ app })\n})).json();\n\nconst res = await fetch(`${apiBase}/api/${app}/services/llm`, {\n  method: 'POST',\n  headers: { 'Content-Type': 'application/json', 'X-App-Token': token },\n  body: JSON.stringify({\n    messages: [{ role: 'user', content: 'Write a story' }],\n    stream: true\n  })\n});\nconst reader = res.body.getReader();\nconst decoder = new TextDecoder();\nlet buffer = '';\nwhile (true) {\n  const { done, value } = await reader.read();\n  if (done) break;\n  buffer += decoder.decode(value, { stream: true });\n  const lines = buffer.split('\\n\\n');\n  buffer = lines.pop();\n  for (const line of lines) {\n    if (!line.startsWith('data: ')) continue;\n    const raw = line.slice(6);\n    if (raw === '[DONE]') break;\n    const chunk = JSON.parse(raw);\n    const content = chunk.choices?.[0]?.delta?.content;\n    if (content) process.stdout.write(content);\n    if (chunk.choices?.[0]?.finish_reason === 'stop') {\n      console.log('\\nUsage:', chunk.usage);\n    }\n  }\n}\n```\n\n## Image Description / OCR Example (vision)\n\nSend an image from the browser with `Gipity.service('llm', ...)` - the same one-liner, with an image block in the message content (a base64 `data:` URI; external image URLs 400):\n\n```js\nconst data = await Gipity.service('llm', {\n  messages: [{\n    role: 'user',\n    content: [\n      { type: 'text', text: 'Extract all the text in this image, verbatim.' },\n      { type: 'image_url', image_url: { url: 'data:image/png;base64,iVBOR...' } }\n    ]\n  }]\n});\nconst text = data.choices[0].message.content;\n```\n\n## Limits\n- **Rate limit**: 600 requests per 5-minute window (per IP)\n- **Max messages**: 20 per request\n- **Max prompt length**: 32,000 chars\n- **Max output tokens**: 4096\n- **Max image size**: 5 MB (base64)\n- **Timeout**: 60s\n- Standard `RateLimit-*` headers included in responses\n\n## Testing\nThe LLM service is tested end-to-end: an E2E test asks the agent to build an app that calls the LLM, deploys it, then verifies the page renders the correct AI response in a headless browser."
}