app.dataengineerhub.blog

Tag: llm

  • Why Larger LLMs Give Incorrect Answers in Production

    Why Larger LLMs Give Incorrect Answers in Production

    A team I worked with upgraded their support-ticket triage pipeline to a bigger, newer model expecting fewer mistakes. Benchmarks said it was smarter across the board. Two weeks in, the error rate on a specific ticket category had gotten worse, not better — and worse in a particular way: the new model was wrong just as often as the old one, but its wrong answers now came with confident, detailed justifications instead of the old model’s hedgy “I’m not entirely sure, but…” The support team started trusting the wrong answers more, because they sounded more certain. Nobody had budgeted for that.

    That’s the part the “AI hallucinates” conversation usually skips. Bigger models are not simply less wrong — in several measured respects they’re differently wrong, and the difference that matters most in production is confidence, not accuracy. There’s real, recent research behind this, and it points at three separate mechanisms: how models are trained to answer instead of abstain, how their reliability quietly degrades as the input grows even inside benchmarks-passing context windows, and how production conditions differ from the clean single-turn evals every leaderboard measures. None of that means larger models are worse. It means “larger” doesn’t buy you the thing production actually needs, which is knowing when to say “I don’t know.”

    TL;DR

    • → OpenAI’s 2025 research reframes hallucination as an incentive problem: next-token training and standard evals reward confident guessing over calibrated “I don’t know,” so models learn to bluff.
    • → Pretrained models are reasonably well-calibrated; RLHF alignment measurably degrades that calibration, making models more confident without making them more correct — an effect researchers call the alignment tax.
    • → Scaling model size doesn’t eliminate this; multiple studies describe larger models producing “confident nonsense” at a similar or greater rate, just more fluently.
    • → Chroma’s 2025 “Context Rot” study tested 18 frontier models and found every one degrades as input length grows — well before the context window is full, even on simple tasks.
    • → Distractors (content that’s topically close but doesn’t answer the question) hurt accuracy more than irrelevant filler, and the effect compounds as input grows.
    • → In Chroma’s tests, Claude models abstained more under ambiguity while GPT models more often produced confident, incorrect answers — model families differ meaningfully in this failure mode.
    • → Production adds failure modes benchmarks don’t test: retrieval quality in RAG, long accumulated agent context, and non-deterministic sampling — so a benchmark-topping model can still underperform in your actual pipeline.

    The training incentive: models are rewarded for guessing

    Start with the most fundamental mechanism, because it explains why this doesn’t go away as models get bigger. OpenAI’s 2025 paper, Why Language Models Hallucinate, argues that the standard training and evaluation setup structurally rewards confident guessing over calibrated uncertainty. A model is trained to predict the next token, and it’s evaluated on benchmarks that score right-or-wrong with no separate credit for a correct “I don’t know.” A model that always guesses will, on average, score higher than one that abstains when unsure — even if the abstaining model is more trustworthy. The behavior isn’t a bug that better data fixes; it’s the predictable output of an incentive structure, and scaling the model up scales the same incentive with it.

    This gets compounded at the alignment stage. Research tracing model calibration through the training pipeline — from Kadavath et al.’s foundational 2022 work through more recent studies on the “alignment tax” — has found that base, pretrained models are reasonably well-calibrated: when a pretrained model says it’s 80% confident, it’s right roughly 80% of the time. RLHF, the fine-tuning stage that makes a model helpful and fluent, measurably degrades that calibration. The model comes out more confident and more polished, but not more accurate — confidence and correctness get pulled apart at exactly the stage designed to make the model pleasant to use.

    Each stage of the standard training pipeline independently pushes toward confident answers. None of them individually looks like a bug — the compounding is the problem.

    Scaling doesn’t fix this because scaling doesn’t change the incentive. A survey published in Frontiers in AI in 2025 makes the point directly: larger models remain capable of “confident nonsense,” and model scaling alone amplifies rather than eliminates hallucination in certain contexts. A bigger model has more capacity to construct a fluent, internally consistent, wrong answer — which is a worse failure mode for a downstream system to catch than an obviously garbled one.

    Context rot: accuracy degrades long before the window fills

    The second mechanism is specific to production because it’s about what happens once you start feeding a model real, long inputs — RAG context, chat history, tool outputs, agent state — rather than the short, clean prompts most benchmarks use. A 2025 Chroma technical report, deliberately titled Context Rot, tested 18 frontier models including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3, and found that every one of them degrades as input length increases — often well before the context window is close to full, and even on tasks as simple as finding one fact in a document or exactly repeating a string back.

    Chroma’s 2025 study found this pattern held across all 18 models tested — accuracy declines with input length even when the task itself doesn’t get harder.

    Three details from that report matter for anyone building production RAG or agent systems. First, ambiguity compounds the effect: when the needed fact doesn’t closely match the wording of the question — the realistic case, since users rarely phrase things the way a document does — performance degrades faster as input grows. Second, distractors hurt more than plain irrelevant filler; content that’s topically related but doesn’t actually answer the question pulls accuracy down non-uniformly, and the effect gets worse, not better, as more distractors are added. Third — and this is the one worth sitting with — the researchers found that structurally coherent context (well-organized, logically flowing text) degraded model performance more than shuffled, incoherent text did. That’s the opposite of the intuitive assumption that cleaner input is always easier for a model to use.

    There’s also a genuinely useful, model-specific finding buried in that report: under ambiguity, Claude models more often abstained — explicitly stating that an answer couldn’t be determined from the given context — while GPT models more often produced a confident, incorrect answer instead. That’s not a claim that one vendor is unconditionally more accurate; it’s evidence that how a model handles uncertainty under long, messy context is a real, measurable, model-specific property worth testing before you pick what runs in production, not something you can assume from a benchmark leaderboard alone.

    Why this specifically bites in production and not in your evaluation

    Put those two mechanisms together and the production gap makes sense. Most public benchmarks are short, clean, single-turn, and score right-or-wrong with no credit for calibrated abstention — which is exactly the setup that rewards confident guessing and doesn’t test context rot at all. Production is the opposite on every count: it’s long (RAG context, chat history, tool outputs), ambiguous (real user phrasing rarely matches document wording), and cumulative (an agent’s context grows with every tool call it makes). A model can genuinely top the leaderboard and still be the wrong choice for a pipeline that hands it 40,000 tokens of retrieved documents and expects a single correct fact back.

    This is also where architecture choices you already know matter start to compound the problem instead of solving it. Retrieving too many chunks “to be safe” in a RAG pipeline doesn’t just cost more — per the context rot findings, it actively degrades accuracy, especially when the extra chunks are distractors rather than clean irrelevant filler. An agent that accumulates tool results across a long-running task is accumulating exactly the kind of long, structurally coherent context Chroma found hurts performance most. And a model under uncertainty defaulting to a confident guess rather than an abstention is the last-mile version of the training-incentive problem — it’s not a separate bug, it’s the same one showing up downstream.

    What actually helps

    None of this is a reason to avoid larger models — it’s a reason to design around known failure modes instead of assuming scale solves them. A few things follow directly from the research above. Keep retrieved context tight rather than generous: fewer, more relevant chunks measurably outperform “retrieve broadly and let the model sort it out,” because more chunks means more distractors and distractors compound with length. Test for abstention behavior specifically, not just accuracy — a model that says “I can’t determine this from the given context” on an ambiguous case is behaving correctly even though a naive eval scores it as a miss; the more dangerous system is the one that never abstains. And treat long-running agent context as a liability to actively manage — summarizing or pruning accumulated state periodically rather than letting it grow unbounded, since the Chroma findings suggest that coherent, well-organized accumulated context degrades performance rather than protecting it.

    It’s also worth remembering that non-determinism sits underneath all of this: the same prompt can produce a different answer on a different run, for reasons rooted in how these models sample tokens — I’ve written separately about why LLMs give different answers to the same question, and that variance means a single spot-check of a prompt tells you less than it feels like it does. If you’re using AI to generate or review structured output like SQL, the same discipline applies: never trust a green run as proof of correctness, a point I’ve made about why passing tests still ship bad data. And if agents are reading your pipeline’s metadata to answer questions, the context rot findings are a direct argument for curating what reaches them rather than dumping everything and hoping — the same principle behind giving agents metadata access deliberately rather than broadly.

    The gotchas nobody warns you about

    A benchmark win doesn’t transfer to your pipeline. Public leaderboards are short, clean, single-turn tests. If your production input is long, ambiguous, or accumulated over many turns, the benchmark isn’t measuring the failure mode you’ll actually hit.

    More retrieved context is not a safety margin. The instinct to retrieve generously “in case the model needs it” directly works against the research: more chunks means more distractors, and distractors degrade accuracy faster as volume grows.

    A model that never says “I don’t know” is a red flag, not a feature. If your eval only scores right/wrong, you can’t distinguish a model correctly abstaining from one confidently guessing wrong — and the second one is more dangerous precisely because it’s harder to catch downstream.

    Well-organized context can hurt more than messy context. Counterintuitively, Chroma found coherent, logically flowing input degraded performance more than shuffled text with the same content. Don’t assume tidy prompt construction is automatically safer.

    Model families differ on this specific behavior. Whether a model abstains or guesses under ambiguity is a measurable, model-specific property. If reliability under uncertainty matters for your use case, test for it directly rather than assuming it from general capability scores.

    The one principle

    A larger model gives you more capability, not more honesty about its limits — those are trained in separately, and mostly not trained in at all. The fix isn’t a bigger model or a longer context window; it’s designing the system around the specific, now well-documented ways models fail — tight retrieval instead of generous, explicit testing for abstention, and active management of anything that accumulates context over time. Production doesn’t need a model that’s never wrong. It needs one — and a system around it — that’s honest about when it doesn’t know.

    Related reading: Why LLMs give different answers to the same question · Why passing tests still ship bad data · Giving agents metadata access safely · It’s not AI you should worry about — it’s automation · Chroma: Context Rot research report · Why Language Models Hallucinate (OpenAI, 2025)

  • Why LLMs give different answers to the same question

    Why LLMs give different answers to the same question

    The bug report said: “The model is broken. It gives a different answer every time I ask the same question.” I’ve gotten some version of this from three different engineers now, and each time the fix is the same — not a code change, but a change in how they think about what a language model actually is. Because the model isn’t broken. It’s doing exactly what it was built to do. The expectation is what’s broken.

    Traditional software is a vending machine: press B4, get the same chips every time. Same input, same output, forever. That determinism is so deeply baked into how engineers think that when an LLM returns “Sure, here’s an email…” one moment and “I’d be happy to help you draft that…” the next — same prompt, same model, same settings — it feels like a defect. It isn’t. A language model doesn’t retrieve answers. It rolls them, one token at a time, from a set of loaded dice it learned during training. This is the guide to why that happens, how to control it, and the surprising truth that you can’t fully turn it off.

    TL;DR

    → LLMs don’t store answers — they predict the next token as a probability distribution over the whole vocabulary, then sample one token from it, append it, and repeat. Different samples → different answers.

    → At each step the model outputs raw scores (logits) for every possible token. Softmax turns those into probabilities. A decoder picks one. That pick is where variation enters.

    → Temperature reshapes the probability distribution before sampling. Low temperature (→0) sharpens it toward the single most likely token (predictable, repetitive). High temperature (0.8–1.2) flattens it (diverse, creative, riskier).

    → top_p (nucleus sampling) and top_k limit which tokens are even eligible — they cut the long tail of unlikely tokens so the model can’t wander into nonsense.

    → The counterintuitive part: even at temperature 0 (greedy decoding), you are not guaranteed identical output. Floating-point rounding and how the server batches your request with others introduce tiny variations that can cascade into different tokens.

    → This is a feature, not a bug. If the model always picked the single highest-probability token, every answer would collapse into the same bland, repetitive text. Sampling is what gives it range.

    → To maximize reproducibility: pin a dated model version (not “latest”), set temperature 0 and top_p 1, use a seed if the API offers one, and design your tests to accept semantic equivalence — not byte-for-byte matches.

    The core idea: the model predicts, it doesn’t retrieve

    Here’s the mental shift that fixes the “it’s broken” reaction. When you ask an LLM a question, it does not look up a stored answer. Before it writes a single word, it scores every token in its vocabulary — tens of thousands of possible next pieces of text — by how well each would continue what’s been written so far. Those raw scores are called logits. They’re just the model’s unnormalized confidence in each candidate token.

    Then a function called softmax converts those scores into a proper probability distribution: numbers between 0 and 1 that sum to 1. Maybe “Sure” gets 18%, “I’d” gets 15%, “Happy” gets 9%, and a long tail of thousands of other tokens splits the rest. A decoder then samples one token from that distribution, appends it to the context, and the whole loop runs again for the next token. And the next. Hundreds of times.

    The key realization: for almost any interesting prompt, there is no single correct next token. There are thousands of valid continuations. An email can open with a greeting, a question, a bold statement, an apology — all reasonable. The model has learned that they’re all plausible, and it assigns each a probability. When it samples, it might pick “Sure” this time and “I’d” the next. From that one different first token, the entire rest of the response can diverge. That’s not the model malfunctioning. That’s the model exploring the space of good answers.

    Temperature: the dial that reshapes the dice

    Temperature doesn’t add randomness — it reshapes the probability distribution the model samples from. Low = sharp spike, one clear winner. High = flattened, many contenders.

    People call temperature “the creativity slider.” That’s directionally right but explains nothing about what’s actually happening. Mechanically, temperature is a number the logits are divided by before softmax converts them to probabilities.

    Divide by a small number (temperature near 0) and the differences between logits get exaggerated. The most likely token’s probability balloons toward 100% and everything else shrinks toward 0. The distribution becomes a single tall spike. The model has almost no choice but to pick that top token every time — predictable, consistent, and at the extreme, repetitive and a little robotic.

    Divide by a larger number (temperature around 1) and the differences compress. The gap between the top token and the runners-up narrows, so more tokens become live options. The distribution flattens. Now the model genuinely might pick the second or fifth most likely token, which is where variety, surprise, and “creativity” come from — along with a higher chance of an odd or wrong choice.

    My rule of thumb from production use: temperature 0–0.3 for anything where correctness and consistency matter (classification, extraction, structured output, factual Q&A). 0.7–0.9 for drafting, brainstorming, and copy where you want variety. Above 1.0 only when you’re deliberately chasing unusual output and can tolerate the misfires.

    top_p and top_k: fencing off the nonsense

    Temperature reshapes the whole distribution, but two other levers control which tokens are even allowed into the drawing.

    top_k is the blunt version: keep only the k most likely tokens, discard the rest, sample from what’s left. top_k = 40 means “only ever consider the 40 best options.” It stops the model from occasionally grabbing a bizarre low-probability token from the tail.

    top_p (nucleus sampling) is smarter and more common. Instead of a fixed count, it keeps the smallest set of top tokens whose probabilities add up to p. top_p = 0.9 means “keep adding tokens from most-likely down until we’ve accounted for 90% of the probability mass, then sample only from that set.” When the model is confident (one token has most of the mass), the nucleus is tiny. When it’s uncertain (mass spread over many tokens), the nucleus is larger. It adapts to how sure the model is.

    In practice you usually tune temperature or top_p, not both aggressively. A common safe setting for consistent output is temperature 0 with top_p 1; a common creative setting is temperature 0.8 with top_p 0.9.

    The part that surprises even experienced engineers

    Temperature 0 gets you close to deterministic, not all the way. Floating-point rounding and server-side batching introduce tiny variations that can tip a near-tie to a different token.

    Here’s the thing almost everyone gets wrong, including people who’ve shipped LLM features: setting temperature to 0 does not guarantee you’ll get the same answer twice. Temperature 0 means greedy decoding — always take the single highest-probability token — so in theory there’s only one path. In practice, reproducibility still breaks, and it’s worth understanding why because it will bite you during evaluation and debugging.

    The first reason is floating-point arithmetic. A forward pass through a large model is billions of arithmetic operations, and computers represent numbers with finite precision. Tiny rounding errors accumulate. When the top two candidate tokens are nearly tied — say 42.7% versus 42.6% — a rounding difference of a hair can flip which one “wins” the greedy pick. That one flipped token cascades: the next context is now different, so the whole rest of the output can diverge.

    The second reason is subtler and more modern: batch variance. When you send a prompt to a hosted model, the server doesn’t process it alone — it batches your request with other users’ requests for GPU efficiency. The exact composition of that batch changes the order and grouping of the underlying matrix operations, and because floating-point addition isn’t perfectly associative (a + b + c can differ slightly from c + b + a at the bit level), the logits come out microscopically different depending on who else you were batched with. You changed nothing; the server’s batching did.

    How bad is it? One 2026 benchmark sent the same prompt ten times at temperature 0 across several models and measured byte-for-byte identical responses. On an open-ended prompt, results ranged from around 70% identical on one model down to essentially 0% on others — the longer and more open-ended the prompt, the faster determinism collapsed. Another team ran a single prompt a thousand times at temperature 0 and got around 80 distinct outputs. It’s fixable with special deterministic inference kernels, but that’s an infrastructure choice most hosted APIs don’t make by default because it’s slower.

    Why this is a feature, not a bug

    It’s tempting to see all this as a flaw to be stamped out. But step back: if a model always emitted the single most probable token, it would be nearly useless for most of what we use it for. Ask it to write three taglines and you’d get the same one three times. Ask for brainstorming and it would give one rigid answer. The probabilistic sampling is precisely what lets a model produce a greeting one way and a different, equally-good way the next — what makes it feel like it has range instead of a single canned response per prompt.

    The variation isn’t randomness in the “anything goes” sense. It’s controlled exploration of a space of good answers, bounded by the probabilities the model learned. Turn the temperature down when you need the boundaries tight; turn it up when you want the model to roam. The dial is the point.

    The gotchas nobody warns you about

    “Same answer” is the wrong test. If your evaluation checks whether the model returns byte-identical output across runs, it will fail for reasons that have nothing to do with quality. Test for semantic equivalence — does the answer mean the same thing, contain the same facts, pass the same downstream parse — not exact string match.

    Structured output is where variation actually hurts. A human reading two differently-worded but equivalent answers doesn’t care. A downstream system parsing the model’s output with a regex absolutely does. If run one returns {"status": "approved"} and run two returns The status is approved., your parser breaks. This is why low temperature plus a strict output schema (or structured-output / JSON mode) matters so much for anything programmatic.

    “latest” is a moving target. If you pin your app to a model alias like “latest” or an undated name, the provider can update the underlying model and your outputs shift overnight — a different kind of non-determinism entirely, at the version level. Pin a specific dated model identifier so you control when the model changes.

    Reasoning models add a hidden layer. Models with extended thinking generate a hidden chain-of-thought before the final answer. That internal reasoning is itself sampled, so even more variation can accumulate before you see the first visible token. Same principles, more surface area.

    How to get the most reproducible output you can

    You can’t make a hosted LLM perfectly deterministic, but you can get close enough for most needs. Pin a specific dated model version rather than a moving alias. Set temperature to 0 and top_p to 1. Use the API’s seed parameter if it offers one, and record any response fingerprint the provider returns so you can tell when the underlying system changed. For self-hosted models, pin the inference engine version, the numeric precision (bf16 vs fp16), and the batch settings — and if you truly need bit-for-bit reproducibility (for audits, evaluation, or RL training), look into the batch-invariant deterministic kernels that some inference stacks now support, accepting that they run a bit slower.

    Then, most importantly, build your evaluation to tolerate the residual variation. Assert on meaning, structure, and facts — not on exact wording. The teams that fight non-determinism with string equality lose; the teams that design around semantic equivalence ship.

    The one principle

    A language model is a probability engine, not a lookup table. Different answers to the same prompt aren’t a malfunction — they’re the visible result of sampling from a distribution of good continuations. Control the spread with temperature and top_p, pin your versions, and test for meaning rather than exact text. Once you stop expecting a vending machine and start treating it like a set of well-trained dice, almost everything about its behavior makes sense.

    Related reading: Why AI Agents Forget: Memory Architecture in AI Agents · Why Your RAG Pipeline Is Failing Silently · OpenAI API: temperature, top_p, and seed parameters · Anthropic Claude API reference