Generative AI has changed the economics of possibility. A model can produce another hypothesis, image, design or implementation almost instantly. What was once scarce—variation—can now be generated in abundance.
But abundance does not remove uncertainty. It creates more candidates about which we can be uncertain.
That is where entropy becomes useful. It provides a language for connecting the expansion of a possibility space to the work required to make that space navigable. Generation distributes probability across alternatives. Selection uses constraints, evidence and evaluation to concentrate attention on what survives.
The important question is no longer only how many possibilities a system can create. It is how much relevant uncertainty it can remove—and what that reduction costs.
One Word, Three Different Questions
Entropy appears in information theory, epistemology and thermodynamics, but it does not mean exactly the same thing in each domain.
| Domain | What entropy helps describe |
|---|---|
| Information theory | Uncertainty in a probability distribution or information source |
| Knowledge work | Uncertainty among claims, explanations or courses of action |
| Thermodynamics | The multiplicity and dispersal of physical states in a system |
Claude Shannon defined information entropy without needing to explain what a message means. The measure concerns the uncertainty of symbols drawn from a distribution. A source whose next symbol is difficult to predict has higher entropy than one whose output is nearly fixed.
Epistemic uncertainty is related but richer. A team may be uncertain because evidence is missing, concepts are ambiguous, models conflict or the future remains open. It cannot always be reduced to one probability distribution.
Thermodynamic entropy concerns physical systems. It becomes relevant because information processing is never performed by abstraction alone. Bits are represented, moved and erased by physical machinery.
These meanings should not be collapsed into a single equation. Their connection is architectural: an AI system manipulates informational uncertainty through physical processes in order to support knowledge and action.
Generation Expands the Distribution
The entropy of a discrete distribution is commonly written as H(X) = -Σ p(x) log p(x). The formula captures how uncertain we are about which outcome will occur. It does not say whether an outcome is beautiful, useful or true.
For generative systems, probability is productive. Sampling can expose different regions of a model’s learned distribution. Temperature, seeds, prompts and other controls can preserve or suppress variation. The site’s earlier work on entropic shaping treats this uncertainty as creative material: an artist shapes a field of possibilities instead of determining every detail directly.
The same principle applies beyond art. Multiple candidate explanations reduce the risk of committing to the first plausible story. Alternative implementations reveal trade-offs. Competing forecasts make hidden assumptions visible.
High candidate entropy can therefore be valuable. A system with too little variation repeats familiar answers and converges prematurely.
But variation is only potential. Ten plausible answers may create more work than one if the system cannot distinguish among them.
Selection Turns Possibility into Structure
The two engines of discovery are generation and selection. Entropy clarifies their relationship.
Generation spreads effort across a wider possibility space. Selection concentrates it again. Tests eliminate implementations. Measurements weaken hypotheses. Formal verification removes invalid proof paths. Human judgment rejects outcomes that satisfy a metric while violating the actual purpose.
This resembles a reduction in uncertainty, but it is always conditional. A candidate survives relative to specific evidence, criteria and scope. Selection cannot turn a weak evaluator into truth. It can only produce confidence justified by the distinctions the evaluator is capable of making.
The result of good selection is not merely fewer options. It is structure:
- a record of which paths failed;
- evidence distinguishing the survivors;
- constraints that can guide another generation;
- a clearer account of what remains unknown.
This structure feeds back into the next search cycle. Entropy is not driven monotonically toward zero. Productive systems alternate between opening the space and reducing it.
Every Reduction Has a Budget
Selection requires work. A candidate program must be executed. A scientific hypothesis needs an experiment. A factual claim must be checked against evidence. A design needs comparison, simulation or review.
Every one of these operations consumes resources: compute, memory, network capacity, energy, time and human attention.
Landauer’s principle established a physical connection between logically irreversible information processing and heat generation. It defines a theoretical lower bound associated with erasing information. Present-day AI hardware operates far above that minimum, so the principle should not be used as a direct estimate of a model’s electricity consumption.
Its deeper lesson is enough: information processing has a physical implementation. Selecting among abstract possibilities eventually becomes switching, moving, storing, cooling and discarding physical states.
The entropy budget of intelligence is therefore the resource envelope within which a system can explore possibilities and reduce uncertainty. A larger budget may support more samples, tests and simulations. A better architecture may achieve the same useful reduction with less.
A Fitness Function for Useful Intelligence
This suggests a conceptual ratio:
useful intelligence ≈ relevant uncertainty removed / total resources consumed
It is not a universal metric. “Relevant uncertainty” depends on purpose, and resources include qualities that cannot be reduced to electricity alone: time, capital, expert attention, environmental impact and risk.
Still, the ratio corrects a common category error. Compute is an input, not the outcome.
Users do not ultimately need tokens, parameters or floating-point operations. They need a support case resolved, a safe decision made, a proof completed or a design that works. A system that produces more tokens while leaving the decision equally uncertain has consumed resources without delivering equivalent informational value.
The evaluator matters as much as the generator. A decisive cheap test can be more valuable than thousands of additional samples. Conversely, a poor evaluator can spend enormous compute selecting candidates that merely exploit its proxy.
Efficiency Selects the Architecture
Once systems compete on useful outcomes rather than raw computation, efficiency becomes a selection pressure on AI itself.
Models, algorithms, memory systems and chips become variations within a larger search. Some achieve better results through scale. Others reduce the search space earlier, represent knowledge more compactly, reuse verified work or exploit a physical architecture suited to the operation.
AlphaEvolve makes the recursive character of this process unusually visible. It combines LLM-generated program variations with automated evaluators and evolutionary selection. Among its reported applications are improvements to data-center scheduling, chip design and AI training—the discovery loop acting on parts of its own computational substrate.
This is more than software optimization. It shows how an evaluator can redirect generative abundance toward lower resource cost. When energy, latency or hardware utilization enters the fitness function, the search begins selecting the machinery of search.
From Information to Infrastructure
AI infrastructure forecasts often begin with demand and extrapolate the compute required to serve it. The entropy perspective reveals a hidden assumption: that tomorrow’s systems will need roughly the same resources to remove a given amount of relevant uncertainty.
That relationship is not fixed.
Better algorithms can reduce the candidates that require evaluation. Better memory can prevent the system from repeating failed searches. Specialized hardware can perform important operations with less data movement. New computational paradigms may use physical dynamics that conventional architectures spend energy suppressing.
At the same time, efficiency does not guarantee lower total energy consumption. Cheaper intelligence may expand demand and produce a rebound effect. The International Energy Agency already expects strong growth in data-center electricity use while accounting for continuing efficiency improvements.
The narrower conclusion is more important: the physical infrastructure of AI is exposed to changes in the cost of uncertainty reduction.
The future may not belong to the system that can generate the most possibilities. It may belong to the system that can eliminate the most irrelevant possibilities per unit of energy, time and capital.
Once that becomes the selection criterion, the machine begins searching for ways to make its own infrastructure obsolete.