Shipping LLM features that survive contact with real users

Building an LLM demo is easy: call an API, print the response, ship a screenshot. Building an LLM feature that survives real users, real load, and real edge cases is a different job entirely — and it's the job I've spent most of my time on since joining my current team as an AI Engineer. The gap between the two is where most "AI features" quietly fail in production.

Streaming isn't a nice-to-have, it's the UX

A model response that takes three to eight seconds feels broken if the user just sees a static spinner. The same response, streamed token-by-token as it's generated, feels instant — because something is visibly happening within a few hundred milliseconds, even though the total time is identical. This is the single highest-leverage change for making an LLM feature feel fast, and it costs almost nothing to implement on either FastAPI or a Next.js API route.

// Next.js route handler, streaming an SSE response from a backend
export async function POST(req: Request) {
  const { prompt } = await req.json();
  const upstream = await fetch(process.env.LLM_API_URL + "/chat", {
    method: "POST",
    body: JSON.stringify({ prompt }),
  });

  return new Response(upstream.body, {
    headers: { "Content-Type": "text/event-stream" },
  });
}

The model will fail. Design for it up front.

Every LLM provider has outages, rate limits, and occasional garbage output. A production feature needs to handle all three without the user ever seeing a raw stack trace:

• Timeouts and retries with exponential backoff for transient failures — but capped, so a user isn't stuck waiting 30 seconds for a retry chain to give up.
• A fallback model or provider for hard outages. If the primary API is down, falling back to a second provider (even a slightly worse one) beats showing an error.
• Output validation — if you asked for structured JSON and the model hallucinated malformed output, catch it and retry once with a stricter prompt before surfacing a failure.

None of this shows up in a demo. All of it shows up in the first week of real traffic.

Cost is a product decision, not an afterthought

LLM API calls are priced per token, and costs scale directly with usage in a way most engineers aren't used to thinking about. A few things I check on every feature before it ships:

• Is the prompt as short as it can be while still being effective? Trimmed system prompts and truncated context windows add up fast at scale.
• Is there a cap on how much a single user or session can spend? A runaway loop calling the API repeatedly is a real failure mode, not a hypothetical one.
• Would a cheaper, smaller model do the job for this specific task? Not every feature needs the most expensive model available — routing simpler tasks to a cheaper model is free money saved.

Function calling: powerful, but keep the blast radius small

Letting a model call functions — search a database, send an email, modify a record — is where LLM features get genuinely useful, and also where they get genuinely dangerous if not scoped carefully. My rule: the model decides what to call and with what arguments, but the actual execution always goes through the same validation and permission checks a human-triggered action would. The model is never trusted with unchecked side effects, no matter how good the demo looks.

# Every tool call still goes through normal authorization
def execute_tool_call(call, user):
    if call.name == "cancel_subscription":
        if not user.can(Permission.MANAGE_BILLING):
            raise PermissionError("not authorized")
    return TOOL_REGISTRY[call.name](**call.arguments)

The UI has to earn trust, not just show an answer

Users are (correctly) skeptical of AI output. A production feature should make it easy to verify the answer, not just present it with false confidence:

• Show citations or sources when the answer is grounded in retrieved data, not just asserted by the model.
• Make it obvious when something is AI-generated versus human-authored content.
• Give an easy path to regenerate, edit, or flag a bad response — silent failure erodes trust faster than an occasional visible mistake.

What actually separates a demo from a product

Streaming instead of spinners. Retries and fallbacks instead of raw errors. Cost ceilings instead of an open tab. Scoped tool execution instead of blind trust. And a UI that treats the model's output as a claim to verify, not a fact to display. None of this is visible in a five-minute demo — which is exactly why it's the part that determines whether the feature survives its first real week of traffic.