LLM API Integration — First Call to Production (2026)

LLM API integration is the first practical skill every GenAI engineer needs. This guide takes you from your first API call to a production-ready integration — covering SDK setup, streaming, error handling, cost optimization, and the multi-provider patterns that keep your application fast and your bills under $50/month.

Updated March 2026 — Covers Anthropic Messages API, OpenAI Chat Completions, streaming patterns, and the model routing strategy that cuts costs by 60-80%.

Before RAG, agents, or evaluation — you need a working LLM API call; everything else in GenAI engineering builds on this foundation.

Before you build RAG pipelines, design agents, or implement evaluation — you need to call an LLM API. Everything in GenAI engineering starts with this step.

The APIs themselves are simple: send a prompt, get a response. What makes API integration a real skill is everything around it — streaming for real-time UX, error handling for reliability, cost optimization for budget survival, and multi-provider routing for resilience.

This guide covers the practical engineering, not the theory. You will write working code.

Who this is for:

• Beginners making their first LLM API call
• Junior engineers moving from tutorials to production code
• Senior engineers setting up multi-provider architectures

Six common integration mistakes — from missing error handling to hardcoded API keys — each have a well-defined prevention pattern covered in this guide.

The good news: all of these are solved patterns. This guide covers each one.

Every LLM API call is a function with three inputs — system prompt, messages, and parameters — that returns either a complete text response or a stream of tokens.

Think of an LLM API as a function with three inputs:

1. System prompt — Who the model should be (persona, rules, constraints)
2. Messages — The conversation history (user messages + assistant responses)
3. Parameters — Temperature, max tokens, model selection

The output is either a complete text response or a stream of tokens.

Four steps take you from SDK installation to a production-ready integration: configure credentials, send your first request, add streaming, and implement error handling with retries.

# Anthropic
pip install anthropic
# Set ANTHROPIC_API_KEY in your .env file or shell profile

# OpenAI
pip install openai
# Set OPENAI_API_KEY in your .env file or shell profile

Never hardcode API keys. Use environment variables, .env files (gitignored), or a secrets manager.

Anthropic (Claude):

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from env

message = client.messages.create(
    model="claude-sonnet-4-6-20250514",
    max_tokens=1024,
    system="You are a helpful coding assistant.",
    messages=[
        {"role": "user", "content": "Explain async/await in Python in 3 sentences."}
    ]
)

print(message.content[0].text)
# Usage: message.usage.input_tokens, message.usage.output_tokens

OpenAI (GPT-4):

from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY from env

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "You are a helpful coding assistant."},
        {"role": "user", "content": "Explain async/await in Python in 3 sentences."}
    ]
)

print(response.choices[0].message.content)
# Usage: response.usage.prompt_tokens, response.usage.completion_tokens

Streaming returns tokens as they’re generated, reducing perceived latency from 3-5 seconds to under 500ms.

# Anthropic streaming
with client.messages.stream(
    model="claude-sonnet-4-6-20250514",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Explain RAG in detail."}]
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

# OpenAI streaming
stream = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Explain RAG in detail."}],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

import anthropic
import time

def call_with_retry(client, max_retries=3, **kwargs):
    for attempt in range(max_retries):
        try:
            return client.messages.create(**kwargs)
        except anthropic.RateLimitError:
            wait = 2 ** attempt  # 1s, 2s, 4s
            time.sleep(wait)
        except anthropic.APITimeoutError:
            if attempt == max_retries - 1:
                raise
        except anthropic.APIError as e:
            if e.status_code >= 500:
                time.sleep(2 ** attempt)
            else:
                raise  # Client errors (4xx) shouldn't be retried
    raise Exception("Max retries exceeded")

Routing queries to the cheapest model that meets quality requirements is the single largest cost lever, typically cutting spend 60–80%.

The biggest cost optimization is routing queries to the right model. Not every request needs the most expensive model.

A simple router:

def route_model(query: str, complexity: str = "auto") -> str:
    if complexity == "simple" or len(query) < 100:
        return "claude-haiku-4-5-20251001"
    elif complexity == "complex" or "design" in query.lower():
        return "claude-opus-4-6-20260115"
    else:
        return "claude-sonnet-4-6-20250514"

A production app processing 1,000 requests/day with 80% routed to Haiku and 20% to Sonnet costs roughly $30-50/month instead of $200+ if everything hits Sonnet.

If you send the same system prompt with every request, cache it. Anthropic’s prompt caching reduces cost by up to 90% for repeated prefixes. See the Anthropic API Guide for implementation details.

These two patterns — a streaming FastAPI endpoint and a multi-provider fallback — cover the most common production integration scenarios.

Here’s a production-ready FastAPI endpoint with streaming, error handling, and cost tracking:

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import anthropic

app = FastAPI()
client = anthropic.Anthropic()

@app.post("/chat")
async def chat(user_message: str):
    async def generate():
        with client.messages.stream(
            model="claude-sonnet-4-6-20250514",
            max_tokens=1024,
            system="You are a helpful assistant for our product.",
            messages=[{"role": "user", "content": user_message}]
        ) as stream:
            for text in stream.text_stream:
                yield text

    return StreamingResponse(generate(), media_type="text/plain")

def call_with_fallback(messages: list, max_tokens: int = 1024):
    """Try Anthropic first, fall back to OpenAI."""
    try:
        response = anthropic_client.messages.create(
            model="claude-sonnet-4-6-20250514",
            max_tokens=max_tokens,
            messages=messages
        )
        return response.content[0].text
    except Exception:
        response = openai_client.chat.completions.create(
            model="gpt-4o",
            messages=messages
        )
        return response.choices[0].message.content

Four failure modes — missing budget alerts, prompt injection, provider lock-in, and latency variance — are the most common production LLM API problems and all are preventable with upfront design.

No budget alerts: LLM APIs are pay-per-use. A bug that sends requests in a loop can run up hundreds of dollars before you notice. Set daily budget alerts on day one.

Prompt injection: User input goes directly into the prompt. Without input validation and output filtering, users can manipulate the model’s behavior. This is the LLM equivalent of SQL injection.

Provider lock-in: Each provider’s API has different message formats, tool calling conventions, and error codes. Abstract your LLM calls behind an interface from the start if you might switch providers.

Latency variance: LLM API latency ranges from 200ms to 10+ seconds depending on model, prompt length, and provider load. Design your UX for streaming, not waiting.

Two questions dominate LLM API interviews: how to handle failures in production, and how to optimize costs — both require concrete mechanisms, not vague principles.

Q: “How would you handle LLM API failures in production?”

Strong answer: “Three layers. First, retry with exponential backoff for transient errors — rate limits (429) and server errors (5xx). Second, multi-provider fallback — if Anthropic is down, route to OpenAI. Third, graceful degradation — if all providers fail, return a cached response or a helpful error message. I’d use circuit breaker pattern to avoid hammering a failing provider and alert on error rate spikes.”

Q: “How do you optimize LLM API costs?”

Strong answer: “Model routing is the biggest lever — route 80% of requests to the cheapest model that’s good enough. Second, prompt caching for repeated system prompts. Third, token budget limits per request so a single query can’t consume $5. Fourth, batch processing for non-real-time tasks — batch APIs are 50% cheaper. I’d track cost per request and set daily budget alerts.”

Before deploying any LLM integration, verify these ten checklist items — missing even one can result in key leaks, budget spikes, or silent failures under load.

Before deploying an LLM API integration to production:

• API keys in environment variables, not source code
• Separate dev and production API keys with different rate limits
• Error handling with exponential backoff and max retries
• Streaming enabled for user-facing responses
• Request timeout (30 seconds recommended)
• Cost monitoring with daily budget alerts
• Model routing by query complexity
• Input validation to prevent prompt injection
• Output validation for structured responses
• Logging of usage metrics (tokens, latency, model, cost)

Monthly cost = (requests/day × 30) × avg_tokens × cost_per_token
Example: 1,000/day × 30 × 500 tokens × $3/1M = $45/month (Sonnet)
Example: 1,000/day × 30 × 500 tokens × $0.25/1M = $3.75/month (Haiku)

• Start with one SDK — Anthropic or OpenAI, both are excellent for beginners
• Streaming reduces perceived latency from 3-5 seconds to under 500ms
• Model routing cuts costs 60-80% — Haiku for simple tasks, Sonnet for complex
• Error handling is non-negotiable — retries, backoff, timeout, fallback
• Never hardcode API keys — environment variables, gitignored .env, or secrets manager
• Set budget alerts on day one — a single bug can cost hundreds
• Abstract your LLM client early if you might switch providers

• Anthropic API Guide — Deep dive on Claude Messages API, tool use, and caching
• Prompt Engineering — System prompts, few-shot, and chain-of-thought
• LLM Tool Calling — How to make LLMs call functions
• RAG Architecture — The next step after basic API integration
• Python for GenAI — Python fundamentals for LLM development
• GenAI Engineer Roadmap — Where API integration fits in the learning path