The reliability debate around generative AI is often framed as a contrast: traditional software is deterministic; language models are probabilistic. Run ordinary code twice and expect the same output. Ask an LLM twice and the wording—or even the conclusion—may change.
This difference matters for testing and operations. But determinism is not the same as correctness.
A deterministic program can calculate the wrong tax, apply a discriminatory rule or repeat a corrupted assumption millions of times. Repetition makes the error reproducible. It does not make the result true.
The deeper epistemic question is whether a claim can be falsified.
Reproducibility Is a Property of the Process
Determinism helps engineers reconstruct a failure. If the same inputs produce the same outputs, it is easier to isolate a line of code or state transition. This is a major operational advantage.
Yet knowledge requires more than reconstruction. A result must also encounter something capable of proving it inadequate: contradictory evidence, a failed prediction, an invalid proof step, a violated requirement or a counterexample.
Without such a mechanism, a system may be perfectly reproducible and epistemically closed. It repeats, but it cannot learn why it is wrong.
Probabilistic systems create the opposite risk. They can vary enough to expose alternatives, but the path from proposal to accepted claim may remain implicit. A user receives a polished answer without knowing which evidence could overturn it.
What Makes a Claim Falsifiable?
In an operational AI system, falsifiability is not an abstract declaration. It must be implemented through controls.
A claim becomes meaningfully challengeable when the system records:
- Scope: the context in which the claim is intended to hold.
- Provenance: the sources, observations and transformations behind it.
- Assumptions: the premises that must remain true.
- Test conditions: observations that would support or contradict it.
- Status: whether it is proposed, validated, contested or rejected.
- Ownership: the person or role responsible for accepting and correcting it.
Consider the statement, “The customer is likely to cancel.” On its own, this is difficult to falsify. Over what period? Based on which behavior? At what probability threshold? For which customer segment? What outcome would count as a failed prediction?
Once those boundaries are explicit, the claim becomes testable rather than merely plausible.
Data Can Be Correct While Knowledge Is Wrong
A system may retrieve an accurate number from an authoritative source and still form the wrong conclusion. The data can be current, correctly parsed and faithfully quoted while the interpretation ignores context or applies the wrong category.
This is why data governance and knowledge strategy solve different problems.
Data governance asks who owns the data, who may access it, whether it is complete and how long it should be retained. Knowledge strategy asks what a claim means, when it is valid, which evidence is sufficient and how it can be rejected.
An AI system needs both. Clean data does not prevent a model from forming an invalid inference. A model with strong benchmark performance does not define which claims an organization is prepared to own.
Verification Must Be External to Fluency
An LLM can criticize its own answer, and this can be useful for exploration. But self-critique is not automatically independent verification. The critique is generated by the same statistical machinery and may preserve the original hidden assumption.
Stronger validation connects the proposal to an external constraint:
- code runs against tests;
- a proof passes a formal checker;
- a factual claim resolves to a source;
- a causal hypothesis predicts an observation;
- a policy interpretation is reviewed by an accountable expert;
- a recommendation remains inside a documented safety boundary.
Research on autoformalization illustrates the principle. A model translates an informal quantitative problem into a formal representation that can be checked for consistency. Candidate answers can then be rejected by a verifier rather than selected only because the model sounds confident or because several samples agree.
The relevant architecture is not “AI versus deterministic software.” It is a generator connected to a falsification mechanism appropriate to the domain.
Local Correction Is Part of Falsifiability
A claim is not operationally falsifiable if discovering an error does not provide a practical route to correction.
In an opaque model, fixing one failure may require prompt changes, fine-tuning or retraining whose effects are non-local. A correction intended for one domain can alter behavior elsewhere. The original error may disappear without leaving a clear account of what changed.
A knowledge-centric system keeps production claims outside the model in an inspectable representation. Sources, policies, interpretations and validation states can be versioned. A false claim can be rejected locally while the history of its creation and use remains visible.
The LLM becomes replaceable. The organizational knowledge and responsibility structure remain stable.
From Confidence to Contestability
AI products often communicate reliability through confidence: a fluent tone, a score or a single recommended answer. Falsification suggests a different interface.
Instead of asking only, “How confident is the system?”, ask:
- Can the affected person challenge the result?
- Can an auditor reconstruct the evidence path?
- Can an operator see which assumption failed?
- Can the organization correct the claim without erasing the record?
- Is responsibility assigned to a human role rather than to “the model”?
These are measures of contestability. They do not guarantee that every decision will be correct. They ensure that errors can become visible, actionable and attributable.
The Better Boundary
Determinism remains valuable for testing, replay and forensic analysis. But it is not the boundary between knowledge and error.
The better boundary is whether the system can distinguish a proposal from an accepted claim, expose what would invalidate that claim and preserve responsibility for the decision to use it.
An LLM may generate the hypothesis. Evidence must be allowed to defeat it.