Our guide on API and integration strategy already covers the general build-vs-buy decision for commodity capabilities like authentication, payments, search, and email — and much of that framework still applies to an AI feature. What it doesn't cover is what is genuinely different about an LLM provider specifically: a pricing model billed per token rather than per seat or per request, output that isn't deterministic the way a payments API's response is, and a vendor risk profile shaped by a pace of model change no commodity API category moves at. Our guide on data and analytics infrastructure is a different topic entirely — that guide covers tracking how your product is used, not building a feature that itself calls an AI model. This guide covers the specific mechanics of integrating an LLM-powered feature: what it actually costs, which build pattern fits which problem, and the four vendor risks worth understanding before a feature ships, not after a surprising bill or a broken integration forces the question.
What Makes This Different From a Normal API
What is genuinely different about integrating an LLM API versus a typical third-party SaaS API?
Three things: the pricing model is metered per token consumed rather than a flat per-seat or per-request fee, the output is probabilistic rather than deterministic — the same prompt can produce different answers on different calls — and the underlying models themselves are actively deprecated and replaced on a much faster cycle than a typical SaaS vendor's API surface changes. A payments API integration, once built and tested, behaves the same way indefinitely; an LLM integration is a moving target by design.
It's worth being specific about why this distinction matters beyond being an interesting technical detail. A team that treats an LLM API exactly like a payments or email API — wiring it up once, writing a handful of tests against expected output, and considering the integration “done” — is setting itself up for three separate categories of surprise that a typical API integration doesn't carry: a cost that scales with usage in a way that's much harder to predict up front than a flat subscription fee, test assertions that periodically break not because the integration code changed but because the underlying model's behavior shifted, and a migration forced by the vendor's own deprecation schedule rather than a decision the team made on its own timeline. None of this means LLM features are too risky to build — it means the risk profile is different enough to deserve its own guide, separate from the general API and integration strategy framework.
The Per-Token Pricing Model
How does LLM API pricing actually work, and why is it hard to predict?
Major LLM providers price per million tokens processed — separately for input (what you send) and output (what the model generates) — rather than per seat or per request. Because token count depends on how much text a user submits and how much the model generates in response, actual cost scales with real usage patterns in a way that's far harder to forecast from a flat subscription price, and the gap between a flagship model and a smaller model from the same provider is typically an order of magnitude per token.
As of September 2026, per each provider's own published pricing page, OpenAI lists its flagship model at $10.00 per million input tokens and $50.00 per million output tokens, alongside a smaller model at $0.20 per million input tokens and $1.20 per million output tokens — a roughly 40–50x difference in per-token cost between its most and least expensive current models. Anthropic's own pricing page lists a similar spread: its flagship model at $5 per million input tokens and $25 per million output tokens, against a smaller model at $1 per million input tokens and $5 per million output tokens. It is important to state plainly that these exact figures and model names will very likely have changed by the time this is read — both providers update pricing and their model lineups frequently enough that this guide cannot function as a permanent price reference. Anyone making a real purchasing or architecture decision should check OpenAI's own current pricing page and Anthropic's own current pricing page directly, rather than relying on any number printed here.
Why a per-token model makes cost forecasting genuinely harder
A flat, per-seat SaaS subscription is trivial to forecast: multiply the number of users by the price, and the number barely moves month to month. A per-token AI feature cost depends on variables that are much harder to pin down in advance — how long a typical user's input is, how long the model's typical response is for a given feature, how often a given user invokes the feature, and whether any part of the pipeline (a long system prompt, a large retrieved document, a lengthy conversation history resent on every turn) is quietly multiplying the token count on every single call without anyone noticing until the bill arrives. This is exactly the mechanism behind the real cost case below: the underlying per-call cost can be genuinely small, and still produce an enormous total when multiplied across enough calls at enough scale, especially in an agentic or automated workflow where calls happen without a human in the loop deciding whether each one is worth making.
A real, documented example of costs escalating past expectations
A concrete, real, named example is worth citing directly rather than describing the risk only in the abstract. Peter Steinberger — founder of PSPDFKit and creator of the open-source autonomous AI coding agent framework OpenClaw, who later joined OpenAI in February 2026 — reportedly incurred approximately $1.3 million in OpenAI usage over a 30-day period while running roughly 100 autonomous Codex agent instances, consuming an estimated 603 billion tokens across 7.6 million requests (The Next Web, “OpenClaw creator's $1.3 million monthly OpenAI bill reveals the real cost of autonomous AI coding at scale,” May 18, 2026). OpenAI reportedly treated the usage as a covered research investment rather than an unpaid bill the creator had to settle personally. It's worth being precise about the sourcing here: the article itself notes the core figures trace back to a social-media post rather than an official OpenAI statement, so treat the specific dollar figure and token count as reported by a named journalist at a named publication, not as an audited, vendor-confirmed number — and treat any more granular breakdown of how that spend was distributed as unverified beyond what the article itself states.
The lesson this case actually supports is narrower than “AI features are dangerously expensive” — it is that an automated, agentic workflow making its own decisions about when to call an LLM, without a human approving each call, removes the natural cost brake that exists in a typical user-initiated feature. A chat feature a human explicitly invokes has a natural ceiling tied to how much a human is willing to type and wait for; an autonomous agent looping on a task with no such ceiling can generate an amount of usage no team would have approved if asked about it call by call. This is precisely why the cost-control patterns covered later in this guide — particularly around monitoring and rate limits — matter disproportionately more for agentic and automated AI features than for a simple, human-initiated chat interface.
Build Patterns: Prompting, RAG, and Fine-Tuning
What are the three main patterns for building an LLM-powered feature, and how do you choose between them?
Prompting (crafting the instructions sent to a general-purpose model) is the cheapest and fastest starting point for most features. Retrieval-Augmented Generation, or RAG, adds a step that retrieves relevant external information and includes it in the prompt, closing a knowledge gap without retraining the model. Fine-tuning actually retrains a model on labeled examples, and is best reserved for a well-defined, stable task where consistent output format or behavior matters more than incorporating new knowledge.
| Pattern | What It Actually Does | Best Fit |
|---|---|---|
| Prompting | Instructs a general-purpose model directly, with no retraining or external retrieval | A first version of almost any feature; the cheapest and fastest to iterate on |
| RAG (Retrieval-Augmented Generation) | Retrieves relevant external documents/data and includes them in the prompt before generation | Closing a knowledge gap — answering questions about your own data, documents, or anything after the model's training cutoff |
| Fine-tuning | Retrains the model itself on a labeled set of examples specific to your task | A stable, well-defined task where consistent formatting or behavior matters more than new knowledge, and you have real labeled data |
RAG: where the pattern actually comes from
Retrieval-Augmented Generation has a real, specific, citable academic origin worth knowing rather than treating as an undifferentiated industry buzzword: the technique traces to Patrick Lewis and a team of co-authors at Facebook AI Research (now Meta AI), in the paper “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” first posted May 22, 2020, and later published at NeurIPS 2020 (Lewis et al., arXiv:2005.11401). The paper's core insight, still the reason the pattern is used today, is architectural: rather than relying entirely on knowledge baked into a model's weights during training, a RAG system retrieves relevant passages from an external knowledge source at the moment of the query and provides them directly in context — which means the system can incorporate information the model was never trained on, including a company's own private documents or data created after the model's training cutoff, without the cost or delay of retraining the underlying model itself.
In practice, a RAG pipeline for a product feature typically involves a vector database that stores embeddings of a company's own documents or data, a retrieval step that finds the most relevant passages for a given user query, and a generation step that includes those passages in the prompt sent to the LLM — a pattern described directly, from a vendor's own perspective, in vector-database provider Pinecone's own documentation on retrieval-augmented generation. It is worth reading that kind of vendor documentation with the ordinary caveat that it is written to sell a specific product category, but the underlying architectural description (query encoder, retriever, vector store, generator) matches the academic origin closely enough to be a reliable starting point for understanding how a real RAG system is actually built.
When fine-tuning actually makes sense
OpenAI's own documentation on fine-tuning frames it as one step in an iterative loop — build evaluations first, try prompting, fine-tune when prompting alone isn't sufficient, test against the same evaluations, and repeat — explicitly warning that fine-tuning without a real evaluation framework in place gives no reliable way to know whether it actually helped. The practical signal worth taking from this: fine-tuning is not a first resort, and a team that reaches for it before establishing how it will measure success is likely to spend real time and money without a clear way to tell if the result is actually better. It tends to make the most sense once a team has a stable, well-defined task with a real, labeled dataset of examples, and a specific reason prompting alone hasn't been sufficient — inconsistent output formatting, a specialized vocabulary the base model doesn't handle well, or a genuine need to reduce prompt length (and therefore per-call cost) by baking instructions into the model itself rather than resending them on every call.
It is worth being direct about a category of claim this guide deliberately excludes: several secondary, aggregator-style sources cite specific multipliers — a fine-tuned smaller model being some exact factor cheaper than a larger general-purpose model, or fine-tuning increasing inference cost by some exact factor — without a traceable primary source behind the specific number. Rather than repeat an unverified multiplier, the honest guidance is directional: fine-tuning can reduce per-call cost by shortening prompts and by allowing a smaller base model to perform a specific task well, but the exact savings depend entirely on the specific task and should be measured against your own evaluation framework rather than assumed from a number circulating in marketing content.
Agentic workflows: a fourth pattern worth naming separately
Beyond prompting, RAG, and fine-tuning, it's worth naming a fourth, increasingly common pattern separately, precisely because it's the pattern most directly implicated in the real cost case covered above: an agentic workflow, where a model doesn't just respond to a single prompt but plans and executes a sequence of steps on its own, often calling itself or a set of tools repeatedly until it decides a task is complete. This pattern is genuinely different from the other three in a way that matters specifically for cost and risk planning — a single user action can now trigger an unbounded, or at least much less predictable, number of underlying model calls, each one billed individually, with no natural stopping point unless the system explicitly enforces one. The OpenClaw case cited above is an agentic workflow specifically, not a simple prompting or RAG pattern, which is precisely why its cost scaled the way it did — a hundred autonomous instances, each looping through its own multi-step task, multiplies the token-metered pricing model covered earlier by a factor no single-prompt feature would ever approach.
For a small team building its first AI feature, the practical guidance is to treat an agentic pattern as a meaningfully bigger commitment than prompting, RAG, or fine-tuning, not a natural next step to reach for once those feel too simple. An agentic workflow needs its own explicit safeguards from the start — a hard ceiling on how many steps or calls a single task can take, logging detailed enough to reconstruct exactly what happened if a run goes wrong, and ideally a real-time cost or call-count check that can halt an in-progress run before it compounds into the kind of bill this guide has already covered. None of this means agentic workflows should be avoided — they are increasingly the pattern that makes an AI feature genuinely useful rather than a simple chat wrapper — but they deserve the most deliberate safeguards of the four patterns covered here, precisely because they carry the least natural cost ceiling.
Vendor Risk: Model Deprecation
How often do LLM providers actually retire models, and how much notice do they give?
Both Anthropic and OpenAI publish explicit model-deprecation policies with defined minimum notice periods — Anthropic states a minimum of 60 days for a publicly released model, while OpenAI states 6 months for generally available models, 3 months for specialized variants, and as little as 2 weeks for preview models. Real, dated deprecation announcements from both companies show they follow through on this cycle regularly, which means an LLM-powered feature should assume the specific model version it launches on will eventually need to be swapped out.
Anthropic's own model-deprecations page documents this policy directly, along with a running history of specific retirements (Anthropic, “Model deprecations”): Claude Sonnet 3.5 was announced for deprecation on August 13, 2025 and retired October 28, 2025; Claude Opus 3 was announced June 30, 2025 and retired January 5, 2026; and Claude 2, Claude 2.1, and Claude Sonnet 3 were announced January 21, 2025 and retired July 21, 2025 — a consistent pattern of several months' notice followed by an actual, enforced retirement date, not an indefinite grace period. OpenAI's own deprecations documentation states a similar structure, and a concrete real example: the Assistants API was announced for sunset on August 26, 2025, with an actual shutdown date of August 26, 2026 — a full year of transition runway in that specific case, longer than the policy's stated minimum.
| Provider | Stated Minimum Notice | Real, Dated Example |
|---|---|---|
| Anthropic | At least 60 days for a publicly released model | Claude Sonnet 3.5: announced Aug 13, 2025 → retired Oct 28, 2025 |
| Anthropic | (same policy) | Claude Opus 3: announced Jun 30, 2025 → retired Jan 5, 2026 |
| OpenAI | 6 months (GA models), 3 months (specialized), as little as 2 weeks (previews) | Assistants API: announced for sunset Aug 26, 2025 → shutdown Aug 26, 2026 |
The practical implication for a team shipping an AI feature is straightforward but frequently ignored until the first forced migration arrives: the specific model version powering a feature at launch is not a permanent foundation, and treating it as one — hardcoding a specific model identifier deep into application logic, skipping the evaluation framework that would let a team confirm a replacement model still performs acceptably, or failing to track vendor deprecation announcements at all — converts an expected, predictable event into an unplanned emergency. A team that isolates the model call behind a clear internal interface, and maintains even a lightweight evaluation set to re-run against a new model version, can treat a deprecation announcement as a scheduled task rather than a fire drill.
Vendor Risk: Rate Limits
How do LLM provider rate limits actually work, and why do they matter for planning?
Both major providers gate API rate limits behind usage tiers tied to how much a customer has already spent — not a single fixed limit for all customers — measured across dimensions like requests per minute, tokens per minute, and (for OpenAI) requests per day, with substantially higher ceilings unlocked only after a customer has paid a cumulative amount over time.
OpenAI's own rate-limits documentation describes a tiered system (Free through Tier 5) gated by cumulative dollars paid — for instance, reaching a specific paid-usage threshold unlocks a materially higher monthly usage cap than the entry tier allows — with limits enforced across requests per minute, requests per day, tokens per minute, tokens per day, and images per minute depending on the endpoint. Anthropic's own rate-limits documentation describes a similar structure with named tiers (Start, Build, Scale, and a negotiated Custom tier) tied to explicit monthly spend caps, and measures limits via requests per minute alongside separate input-token-per-minute and output-token-per-minute buckets, enforced through a token-bucket algorithm — notably, Anthropic's documentation states that cached input tokens are generally exempt from the input-token-per-minute limit, which is a real, citable mechanic worth knowing when planning both cost and throughput together.
The practical consequence for a team building its first AI feature is that the rate limit available on day one, under a new or lightly-used account, is very likely lower than the limit the feature will need once it has real usage — which means a feature that works perfectly in testing can start failing with rate-limit errors precisely when it starts succeeding with real users, unless a team proactively tracks its tier status and spend trajectory against its provider's published thresholds well before hitting them. This is a meaningfully different failure mode than a typical SaaS API rate limit, which is usually a flat, predictable ceiling communicated clearly at signup rather than one that moves as a function of how much the customer has already been billed.
Vendor Risk: Hallucination and Non-Determinism
Is LLM hallucination a real, measured problem, or mostly anecdotal?
It is real and measured, not just anecdotal. TruthfulQA, a peer-reviewed academic benchmark of 817 questions across 38 categories, found the best-performing model tested was truthful on only 58% of questions, compared to a 94% human baseline — a substantial, quantified gap between model and human reliability on questions specifically designed to surface common false beliefs and misconceptions.
This figure comes from a real, named, peer-reviewed source: Stephanie Lin (Oxford), Jacob Hilton (OpenAI), and Owain Evans (Oxford), “TruthfulQA: Measuring How Models Mimic Human Falsehoods,” first posted September 8, 2021 and published at the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022) (Lin, Hilton & Evans, arXiv:2109.07958). It is worth being precise about what this benchmark measures specifically: its 817 questions were deliberately designed to probe common human misconceptions and false beliefs a model might have absorbed from its training data, so the 58%/94% gap is a measure of resistance to a specific, adversarially chosen category of falsehood — not a general-purpose accuracy score applicable to every possible use of an LLM. It is still a genuinely useful, quantified data point for the underlying claim this section makes: LLM output is not reliably truthful by default, and a feature that presents model output directly to a user as fact, without any verification step, is making that same reliability bet on every single response.
A second real, ongoing, publicly maintained source worth knowing is Vectara's Hughes Hallucination Evaluation Model (HHEM), a continuously updated public leaderboard measuring hallucination rates specifically in grounded-summarization tasks across current commercial models, named for its lead developer Simon Hughes. Unlike a single, dated academic paper, this leaderboard is actively maintained and updated as new models are released, which makes it a more current reference point for a team specifically wanting to know how a candidate model performs on summarization-style hallucination today, as opposed to the broader, dated truthfulness question TruthfulQA addresses. It is worth being direct that this guide could not verify several other hallucination-rate statistics circulating in secondary, aggregator-style content — broad claims about hallucination rates ranging across dozens of models with a single percentage attributed to a named academic benchmark — and has deliberately excluded them rather than repeat a number that could not be traced to a specific, citable primary source.
What non-determinism means for product design, not just accuracy
Beyond outright factual hallucination, it's worth naming a related but distinct property: the same prompt sent to the same model can produce a different response on different calls, even holding every input constant. This matters for reasons beyond accuracy — it means a feature's automated tests need to check for acceptable ranges of behavior rather than exact string matches, that a user reporting “it gave me a different answer yesterday” may be describing expected behavior rather than a bug, and that a feature relying on the model producing a consistent, parseable output format (for downstream automated processing, for instance) needs explicit structure enforced in the prompt or via a provider's structured-output feature, rather than assuming the model will spontaneously format its response the same way every time. Designing around this property from the start — building verification steps for anything presented as fact, and tolerance for variation into anything checking model output programmatically — is meaningfully cheaper than discovering the need for it after a feature has already shipped and users have already been served an inconsistent or incorrect response.
Vendor Risk: Switching Costs
Is it easy to switch between LLM providers once a feature is built on one of them?
No, not typically, despite providers often exposing broadly similar-looking APIs. Real, documented technical friction includes different preferred prompt formatting conventions between providers, different tokenizers that make direct per-token price comparisons misleading, and reported differences in how model quality holds up as context length grows — meaning a prompt carefully tuned for one provider often needs real rework, not a simple API-endpoint swap, to perform comparably on another.
A real, named technical account of this friction is worth citing directly: VentureBeat published “Swapping LLMs isn't plug-and-play: Inside the hidden cost of model migration” on April 16, 2025 (VentureBeat, April 16, 2025), documenting specific, concrete sources of friction: OpenAI's models reportedly respond better to markdown-formatted prompts while Anthropic's favor XML-tag structuring, meaning a prompt optimized for one provider's conventions may need genuine rewriting, not just a find-and-replace, to perform as well on the other. The article's byline is credited to an anonymized “Guest Contributor” rather than a named individual, which is worth disclosing directly — the publication and its specific, technically detailed claims are real and citable, but the individual author's identity isn't. The article also reports specific, first-hand testing suggesting output quality degradation above certain context lengths for one specific model version tested against another's much larger context window — a claim worth attributing to that article's own testing specifically, since it reflects one outlet's specific test conditions at a specific point in time rather than an independently reproduced, universal finding.
The practical lesson for a team building its first AI feature is to treat prompt engineering as provider-specific work from the start, rather than assuming a prompt is a portable asset that will transfer cleanly if the underlying model is swapped later. This doesn't mean a team should avoid committing to a provider — every provider carries this same risk, and indefinite hedging against it has its own real cost in complexity — but it does mean budgeting real time for prompt rework, not just a configuration change, if a future model deprecation or a deliberate cost or quality decision eventually requires a provider switch.
Cost-Control Patterns
A small number of real, vendor-documented patterns account for most of the practical cost control available to a team running an AI feature in production, and it's worth understanding each one directly from the source rather than through secondhand summary.
- 1
Use prompt caching for repeated context
Both OpenAI's and Anthropic's own documentation describe automatic or explicit caching of repeated prompt prefixes, billed at a meaningful discount off the standard input rate — check each provider's current documentation directly for the exact discount percentage, since this guide found inconsistent figures cited across different sources and does not want to assert one as fixed.
- 2
Route simple tasks to a smaller, cheaper model
Both major providers design their own pricing structure around a clear tier of models at meaningfully different price points — a strong signal, built into the pricing structure itself, that not every call needs the most capable (and most expensive) model available.
- 3
Treat prompt length as a direct cost lever
Because pricing is metered per token, anything that grows the prompt on every call — a long system prompt, an entire conversation history resent each turn, an oversized retrieved document in a RAG pipeline — is a direct, ongoing cost multiplier, not a one-time convenience.
- 4
Put a human decision point in front of any agentic or looping call pattern
As the real cost case above illustrates, usage without a human approving each individual call can scale far faster than a team expects — a monitored spend cap or an approval gate on automated, multi-step AI workflows is a direct mitigation, not an optional safeguard.
Anthropic's own rate-limit documentation adds one further, easy-to-miss detail worth knowing directly: cached input tokens are generally exempt from the input-token-per-minute rate limit, which means prompt caching functions as a throughput lever as well as a cost lever — a detail worth factoring into rate-limit capacity planning specifically, not just cost forecasting, for any feature expecting meaningful concurrent usage.
Monitoring as a cost-control practice, not just an operational one
It's worth connecting this section directly back to our incident response and on-call guide: the same alerting philosophy covered there — actionable, symptom-based alerts rather than noisy, cause-based ones — applies directly to AI feature spend, just with a different symptom being monitored. A team that only discovers a cost spike when the monthly invoice arrives is, in effect, running with no alerting at all on one of its most variable-cost dependencies. A daily or even hourly spend check against an expected baseline, with an actual alert firing when spend crosses a defined threshold, converts a once-a-month surprise into a same-day catch — the same logic that guide applies to production outages applies just as directly to a runaway AI feature quietly consuming far more budget than expected before anyone happens to look at the bill.
This is a genuinely inexpensive safeguard relative to the risk it addresses. Most providers expose usage and spend data via their own dashboards or APIs, which means a small team can build a simple daily check — comparing current spend against the same period a week or a month prior, and flagging anything that deviates sharply — without needing a dedicated observability platform. The specific mechanism matters less than the underlying discipline: an AI feature's cost is a live, moving number in a way a flat SaaS subscription never is, and treating it as something to check only when the bill arrives is the single most common way a team ends up surprised by a number it could have caught days or weeks earlier.
A Practical Framework
Bringing the research above together into an actual sequence for a team building its first AI feature:
Begin with prompting against a general-purpose model
Prompting is the cheapest, fastest way to validate whether a feature is worth building at all — reach for RAG or fine-tuning only once prompting alone has demonstrated a specific, real gap.
Abstract the model call behind a clear internal interface
Treat the specific model identifier as a configuration detail, not a hardcoded assumption throughout the codebase — this is what turns an eventual forced deprecation into a scheduled task rather than an emergency.
Build a real evaluation set before trusting output as fact
Given the measured gap between model and human truthfulness, any feature presenting model output as fact to a user needs an explicit verification step, not an assumption of default reliability.
Put a spend cap or approval gate on any automated, looping call pattern
The single clearest lesson from the real cost case in this guide: usage without a human decision point per call can scale far beyond what any team would knowingly approve.
None of this framework requires a large team or a dedicated AI infrastructure specialist to follow — it requires treating an LLM integration with the specific caution its specific risk profile warrants, rather than the generic caution appropriate to a typical third-party API. A feature built with an isolated model interface, a real evaluation set, and a spend gate on anything automated will absorb a future deprecation, a rate-limit ceiling, or an unexpected usage spike far more gracefully than one built as if the underlying model, and the vendor providing it, would simply stay the same indefinitely.
Frequently Asked Questions
What is genuinely different about an LLM API compared to a typical SaaS API?
The pricing model is metered per token rather than per seat or per request, output is probabilistic rather than deterministic, and the underlying models are actively deprecated and replaced on a much faster cycle than a typical SaaS API surface changes — three risk categories a standard API integration checklist does not cover.
How does LLM API pricing actually work?
Major providers charge per million tokens processed, priced separately for input and output, with a roughly 40–50x spread between a flagship and a smaller model from the same provider as of the pricing this guide checked in September 2026. Exact figures change frequently — verify current pricing directly on the provider's own pricing page before making a decision.
Is there a real, documented example of an AI feature's costs escalating unexpectedly?
Yes — Peter Steinberger, creator of the OpenClaw autonomous coding agent framework, reportedly incurred roughly $1.3 million in OpenAI usage over 30 days running about 100 autonomous agent instances (The Next Web, May 18, 2026). The core lesson is that automated, agentic call patterns without a human approving each call remove the natural cost brake a human-initiated feature has.
What is the difference between prompting, RAG, and fine-tuning?
Prompting instructs a general-purpose model directly and is the cheapest starting point. RAG (Retrieval-Augmented Generation, from Lewis et al.'s 2020 paper) retrieves relevant external data and includes it in the prompt, closing a knowledge gap without retraining. Fine-tuning retrains the model itself, best reserved for a stable, well-defined task with real labeled data.
How much notice do LLM providers give before retiring a model?
Anthropic states a minimum of 60 days for a publicly released model; OpenAI states 6 months for generally available models, 3 months for specialized variants, and as little as 2 weeks for previews. Both companies have real, dated deprecation histories showing they follow through on this cycle regularly.
How do LLM provider rate limits work?
Both major providers gate rate limits behind usage tiers tied to how much a customer has already spent, not a single fixed limit for everyone — measured across requests per minute, tokens per minute, and similar dimensions. This means a feature's available rate limit on day one is often lower than what it will need once it has real usage.
Is LLM hallucination a real, measured problem?
Yes. TruthfulQA, a peer-reviewed 2021/2022 academic benchmark (Lin, Hilton & Evans), found the best-performing model tested was truthful on only 58% of its 817 questions, versus a 94% human baseline. Vectara's continuously updated Hughes Hallucination Evaluation Model tracks a related, more current measure for summarization tasks specifically.
Is it easy to switch between LLM providers once a feature is built?
No, typically not, despite broadly similar-looking APIs. Documented friction includes different preferred prompt formatting between providers (e.g., markdown versus XML-tag conventions), different tokenizers that complicate direct price comparisons, and reported quality differences at longer context lengths — meaning a provider switch usually requires real prompt rework, not just a configuration change.
What are the most effective real cost-control patterns for an AI feature?
Prompt caching for repeated context, routing simple tasks to a smaller/cheaper model in the same provider's lineup, treating prompt length as a direct ongoing cost lever, and — most importantly for automated workflows — putting a human decision point or spend cap in front of any agentic or looping call pattern.
How is this guide different from your API and Integration Strategy guide?
Our API and Integration Strategy guide covers the general build-vs-buy decision for commodity capabilities like auth, payments, and search. This guide covers what is specifically different about LLM/AI vendor integration — per-token pricing, non-deterministic output, and a model-deprecation cycle no commodity API category moves at.
How is this guide different from your Data and Analytics Infrastructure guide?
Our Data and Analytics Infrastructure guide covers tracking how your product is actually used — events, warehouses, data quality. This guide covers building a feature that itself calls an AI model. The two are unrelated in scope even though both involve handling data.
Does non-deterministic output mean LLM features can't be tested reliably?
It means they need to be tested differently — checking for acceptable ranges of behavior rather than exact string matches, and using a provider's structured-output features when downstream code needs to parse the response reliably, rather than assuming the model will spontaneously format its output the same way on every call.
What is an agentic workflow, and why does it carry more cost risk than a simple prompting feature?
An agentic workflow has a model plan and execute a sequence of steps on its own, often calling itself or tools repeatedly, rather than responding once to a single prompt. Because each step is a separately billed call with no natural stopping point, a single user action can trigger far more usage than a simple prompting feature — exactly the pattern behind the real $1.3 million cost case covered in this guide, which makes explicit step limits and cost monitoring especially important for this pattern specifically.
None of the risks covered in this guide are a reason to avoid building AI-powered features — they are a reason to build them with the same deliberateness a team would apply to any dependency whose pricing, behavior, and lifecycle it doesn't fully control. The teams that get burned by an AI integration are rarely the ones who researched these mechanics upfront; they're the ones who treated an LLM API as just another item on the same checklist as their payments and email providers, and discovered the differences only once a bill, a deprecation notice, or an inconsistent response forced the question.