Gipity Login

docs.gipity.ai › App services › app-llm

App API - LLM Service

This is the app service a deployed app calls. For the agent making one-off cross-model queries during a chat, see query-llm.

Building a chatbot/assistant UI? Don't hand-roll the message thread, streaming, and input box on top of this service - add the chatbot kit. It drops in a persona, scope guardrails, static knowledge, and a streaming bubble/headless engine wired to this LLM service, no API keys. Use this raw service only when you need a non-chat LLM call.

The LLM service is available for every project with no setup (user_pays billing by default) - apps can call AI models immediately after deploy.

Use project_settings to change settings (optional):

Billing Modes

Set it in source, not just via the tool. project_settings flips billing as a one-off, but that lives as server state that does not travel with your app - rebuild into a fresh project or fork it and it reverts to user_pays. To make the choice reproducible, declare it in a services deploy phase in gipity.yaml so every deploy reproduces it (the same applies to the image, tts, transcribe, etc. services):

- name: services
  type: services
  service_definitions:
    - service: llm
      billing_mode: owner_pays

See deploy for the full services phase. Billing is project-level (one setting for dev and prod). The phase owns only the billing mode; model/token restrictions set via project_settings are preserved.

Billing modes only govern the deployed app's runtime calls. Direct calls during development - gipity service call, gipity generate <image|video|speech|sound|music>, gipity chat, or the agent's own tools - always bill the caller (you): a logged-in owner is an identified payer even under user_pays. Never flip a service to owner_pays just to generate assets or test an endpoint - it's unnecessary, and while flipped the live app accepts anonymous usage on your credits.

Configuration Options

Choosing a Model

Recommended: use an alias, not a concrete id. Set model to a capability tier and we keep it pointed at the current best model in that tier for you — when the model catalog is refreshed, your app rides along automatically with no code change:

small/fast are the cheapest, lowest-latency tier; medium is the balanced default; large/xlarge/thinking are the strongest. Most apps should just pick fast or medium and forget about it.

Or pin a concrete id (from the list below) if you need a specific model and want to freeze its behavior. The trade-off: a pinned id is yours to maintain — if that model is later retired from the catalog, calls will return 400 Unknown model until you update it. Aliases never have this problem.

Available Models

claude-fable-5, claude-opus-5, claude-opus-4-8, claude-opus-4-7, claude-opus-4-6, claude-opus-4-5, claude-sonnet-5, claude-sonnet-4-6, claude-sonnet-4-5, claude-haiku-4-5, gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna, gpt-5.5, gpt-5.5-pro, gpt-5.4, gpt-5.4-pro, gpt-5.4-mini, gpt-5.4-nano, gemini-3.6-flash, gemini-3.5-flash, gemini-3.1-pro-preview, gemini-3.1-flash-lite, gemini-3-flash-preview, gemini-2.5-flash, deepseek/deepseek-v4-pro, deepseek/deepseek-v4-flash-0731, qwen/qwen3-coder-next, qwen/qwen3.7-flash, moonshotai/kimi-k2.7-code, z-ai/glm-5.2

Endpoints

Request Format

The endpoint accepts OpenAI-compatible messages:

Image Support

Both formats are accepted in message content arrays:

// OpenAI format (image_url with data URI)
{ type: 'image_url', image_url: { url: 'data:image/png;base64,iVBOR...' } }

// Native format
{ type: 'image', data: 'iVBOR...', media_type: 'image/png' }

Only data: URIs are supported - external image URLs will return a 400 error.

Testing your vision/OCR function from the CLI? Don't hand-roll base64. gipity fn call <name> --file image=@receipt.png reads the file, base64-encodes it, and attaches it under image in the request body for you (repeatable; combine with an inline JSON body). Same flag on gipity job run. See app-testing.

Response Format (OpenAI-compatible)

Non-streaming:

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "model": "gpt-5.4-mini",
  "choices": [{ "index": 0, "message": { "role": "assistant", "content": "..." }, "finish_reason": "stop" }],
  "usage": { "prompt_tokens": 100, "completion_tokens": 50, "total_tokens": 150 },
  "provider": "anthropic",
  "credits_used": 5
}

Streaming (SSE):

Client Code Example (Non-Streaming) - use the SDK

The Gipity client SDK (auto-injected into every template as <script ... data-app>) exposes Gipity.service('llm', body), which is the one right way to call the LLM from the browser. It reads the app guid + API base off its own script tag, mints and caches the app token, sends the signed-in user's session cookie (so user_pays resolves the payer), and returns the parsed OpenAI-compatible body - or throws on non-2xx with the server's message and err.status/err.code set. Don't hand-roll fetch('/api/token') + X-App-Token - that's the low-level path the SDK already does for you (and hardcoding a.gipity.ai breaks on any non-prod instance).

const data = await Gipity.service('llm', {
  messages: [
    { role: 'system', content: 'Answer concisely.' },
    { role: 'user', content: 'What is the capital of France?' }
  ],
  model: 'fast'   // alias — rides the catalog refresh. Or pin a concrete id.
});
const answer = data.choices[0].message.content; // "The capital of France is Paris."

The same helper reaches every app service - Gipity.service('image', {...}), Gipity.service('tts', {...}), Gipity.service('payments/checkout', {...}) - always returning the result object (never undefined). To send an image for vision/OCR, pass it exactly as the Image Support body shape below.

Need raw fetch (e.g. SSE streaming, which the SDK doesn't unwrap)? Read the API base and app guid off the SDK script tag - const sdk = document.querySelector('script[data-api-base]'); const apiBase = sdk.dataset.apiBase; const app = sdk.dataset.app; - never hardcode a.gipity.ai (correct only on prod). Mint a token with const { data: { token } } = await (await fetch(${apiBase}/api/token, { method:'POST', headers:{'Content-Type':'application/json'}, body: JSON.stringify({ app }) })).json(); (the token is nested under data) and send it as the X-App-Token header. The streaming example below shows the full shape.

Handling out of credits (402) - don't fail silently

When the paying user has no credits left, the call returns HTTP 402 with { error: { code: "INSUFFICIENT_CREDITS", message } } - the message carries a top-up link. Gipity.service surfaces this as a thrown error with err.status === 402, so wrap the call in try/catch and show the user something; an LLM feature that just does nothing on 402 is the #1 cause of a confusing "it broke and I don't know why" wall.

try {
  const data = await Gipity.service('llm', { prompt: 'Summarize this note', model: 'fast' });
  // …use data.choices[0].message.content
} catch (err) {
  if (err.status === 402) {
    // Out of credits. Under user_pays (the default) it's THIS signed-in user who's
    // out - err.message includes the top-up link. Show it instead of a blank.
    showNotice(err.message); // e.g. "Insufficient credits. Buy more at https://prompt.gipity.ai/pricing"
  } else if (err.status === 401) {
    // Not signed in - send them through Sign in with Gipity (app-auth).
    await Gipity.auth.signIn();
  } else {
    throw err;
  }
}

Calling from a serverless function (server-side)

Inside a function, don't fetch this endpoint - a.gipity.ai is blocked from the sandbox. The LLM is injected as a capability on the function's second argument, called server-to-server with no token and no fetch:

export default async function ask(ctx, { llm }) {
  const { text, usage, credits_used } = await llm({
    model: 'fast',   // alias — rides the catalog refresh. Or pin a concrete id.
    messages: [{ role: 'user', content: ctx.body.question }],
    // same fields as the HTTP body: prompt | messages, system_prompt, temperature, max_tokens, image
  });
  return { answer: text };
}

- name: ask
  auth: public
  services: ['llm']      # required - without it, llm() throws

Client Code Example (Streaming)

Streaming needs raw fetch (the SDK's Gipity.service reads the whole JSON body, so it can't stream). Read the API base + app guid off the SDK tag and mint a token as shown in the note above:

const sdk = document.querySelector('script[data-api-base]');
const apiBase = sdk.dataset.apiBase, app = sdk.dataset.app;   // never hardcode a.gipity.ai
const { data: { token } } = await (await fetch(`${apiBase}/api/token`, {
  method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ app })
})).json();

const res = await fetch(`${apiBase}/api/${app}/services/llm`, {
  method: 'POST',
  headers: { 'Content-Type': 'application/json', 'X-App-Token': token },
  body: JSON.stringify({
    messages: [{ role: 'user', content: 'Write a story' }],
    stream: true
  })
});
const reader = res.body.getReader();
const decoder = new TextDecoder();
let buffer = '';
while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });
  const lines = buffer.split('\n\n');
  buffer = lines.pop();
  for (const line of lines) {
    if (!line.startsWith('data: ')) continue;
    const raw = line.slice(6);
    if (raw === '[DONE]') break;
    const chunk = JSON.parse(raw);
    const content = chunk.choices?.[0]?.delta?.content;
    if (content) process.stdout.write(content);
    if (chunk.choices?.[0]?.finish_reason === 'stop') {
      console.log('\nUsage:', chunk.usage);
    }
  }
}

Image Description / OCR Example (vision)

Send an image from the browser with Gipity.service('llm', ...) - the same one-liner, with an image block in the message content (a base64 data: URI; external image URLs 400):

const data = await Gipity.service('llm', {
  messages: [{
    role: 'user',
    content: [
      { type: 'text', text: 'Extract all the text in this image, verbatim.' },
      { type: 'image_url', image_url: { url: 'data:image/png;base64,iVBOR...' } }
    ]
  }]
});
const text = data.choices[0].message.content;

Limits

Testing

The LLM service is tested end-to-end: an E2E test asks the agent to build an app that calls the LLM, deploys it, then verifies the page renders the correct AI response in a headless browser.