Science fiction may have accidentally gotten one part of the future right.
Not flying cars. Not faster-than-light travel.
The chunky hardware.
Cyberpunk 2077 is full of terminals, physical interfaces, cables and industrial controls. Star Wars, despite having artificial intelligence sophisticated enough to produce autonomous droids, imagines code cylinders, isolated terminals and an almost comically physical approach to information.
For decades this looked like retrofuturism: storytellers imagined advanced societies using the technological vocabulary available when their worlds were created.
Perhaps there is another interpretation. Maybe sufficiently powerful software eventually makes physical boundaries desirable again.
We Made Everything Programmable
For the last twenty years, software architecture has moved in one direction: make everything programmable.
Infrastructure became code. Networks became software-defined. Servers became virtual machines and containers. Authentication became APIs. Cars became software-defined vehicles. Physical controls became touchscreens. Administration became remote. Hardware boundaries became software policies.
This made sense while the actor operating these systems was usually a human:
human → decision → software → API → system
Agents change that assumption.
The Tireless Operator
Traditional automation follows a rule: if X, then Y. An agent instead operates a loop:
observe → interpret → hypothesize → act → observe again
It can continue that loop for a long time. It does not become tired after two hours of debugging or naturally pause after changing five resources. It can inspect logs, modify infrastructure, redeploy services, change configuration and investigate the consequences of its own actions.
This is extraordinarily useful. It also changes the risk model.
risk ≈ probability of error × speed × autonomy × blast radius
The formula is not a quantitative standard. It is a useful design heuristic: a mostly reliable agent with broad authority can create more damage than an unreliable tool that has almost none. NIST’s work on agent tool use similarly distinguishes read-only, constrained-write and broad-write access because an agent’s risk depends not only on its model but on the actions its environment permits.
The Dangerous Agent May Be the Helpful One
The obvious fear is a malicious or compromised agent. The more ordinary failure mode may be an enthusiastic agent trying very hard to solve the problem.
Imagine a failed deployment. The agent forms a plausible hypothesis, changes a resource, encounters another error, adjusts permissions, recreates something, rotates a secret, updates the deployment and continues.
Every individual action may be locally reasonable. The original hypothesis may simply have been wrong.
competent execution ≠ correct direction
An agent can move competently and quickly through the wrong part of the state space. A human operator often has a different advantage. An experienced engineer may look at a failure and immediately think, “That is the wrong log,” or, “Why is that architectural boundary missing?”
The machine searches faster. The human may still recognize the wrong search space faster.
The Missing Falsification Loop
This is especially dangerous because generative models are good at constructing coherent explanations. Suppose an agent observes an error and forms hypothesis H.
A robust engineering process tries to falsify it:
H → prediction → test → reject or retain
An autonomous agent has another option:
H → modify environment → observe modified environment
The distinction matters. The agent can begin changing the system according to an incorrect model and then spend considerable effort debugging the world it has just created.
The problem is not stupidity. It is competent action without a sufficiently strong falsification loop. This is why falsification must be a system capability , not another sentence in the prompt.
Authorization Is Not Enough
Modern security architecture is good at asking: Is this identity authorized to perform this operation?
Agents introduce another question: Should this authorized operation actually be happening?
Those questions are not equivalent. An agent using legitimate credentials can delete the wrong resource with a valid API call. It can change an architecture while believing it is simplifying it. It can rotate secrets, rebuild infrastructure or alter policies entirely within its assigned permissions.
Zero Trust and IAM remain necessary, but they do not solve this by themselves. The agent may be exactly who it claims to be. The problem is its authority combined with an incorrect model of the world.
NIST defines least privilege as granting only the resources and authorizations required for a function. Its current work on AI-agent identity goes further by asking how an agent can convey intent, how agent actions bind back to human authorization and how least privilege can work when required actions are not fully predictable. That is the emerging design problem: identity establishes who acts; architecture must also constrain what consequences can follow.
We Virtualized the Boundaries
Over several decades we deliberately removed physical boundaries.
A switch became software. A server became a VM. A network became an overlay. A security boundary became a policy. A deployment became an API call. A credential became a file. A production environment became another profile in ~/.aws/config.
For human operators, this was enormously convenient. For autonomous software, it means that many boundaries exist inside the same programmable world the agent can manipulate.
There is an enormous difference between these two statements:
The agent must not access production.
The agent cannot access production.
Two Science-Fiction Images
One of the strangest features of Star Wars is its deliberately physical information technology. Droids reason, navigate and repair spacecraft, yet important information and authority frequently appear as objects that must be carried to a place.
Production history is the obvious explanation. As a thought experiment, however, imagine a civilization that learned a different lesson:
advanced AI + universal connectivity → catastrophic incidents
advanced AI + deliberately primitive boundaries → bounded authority
The code cylinder then becomes interesting not because the civilization cannot build cloud identity, but because some authority requires a physical object to be present. The galaxy may not have forgotten how to build better networks. Perhaps it remembers why it stopped connecting everything.
Cyberpunk 2077 offers a second visual intuition. Its world combines extraordinary software with heavy connectors, dedicated terminals, physical switches and visibly separate machines. Agentic computing may create pressure back toward that terminal—not for more compute, but for visible authority boundaries.
Imagine a workstation whose physical state reveals something important. A switch determines whether a privileged interface is connected. A production signing key lives only inside a hardware module. A recovery system has no remotely programmable deletion interface. A token physically represents a capability.
The chunky hardware is no longer just aesthetic. It says: this boundary exists in the physical world.
The Agent Computer
A simple version already makes sense today. Instead of running an agent on the workstation that contains a human administrator’s development and production credentials, give the agent its own machine:
agent computer → development credentials only
The human workstation retains broader authority. The agent machine has no production credential, no production SSO session, no backup-administration key and no path for assuming a production role.
Even a complete compromise of the agent computer then has a bounded consequence. The machine can lose everything it was explicitly allowed to control. It cannot obtain authority that was never placed there.
This is not primarily compute isolation. It is authority isolation.
Production Should Be Somewhere Else
In development, high autonomy may be desirable. Let the agent inspect logs, modify code, deploy experiments, run tests and investigate failures. Development is supposed to tolerate failure.
Production is different. A safer flow looks like this:
agent
→ proposed change
→ tests
→ reviewable change set
→ external authorization boundary
→ production
The critical property is that the agent cannot remove the final boundary itself. For sufficiently important systems, that boundary may be a separate identity domain, a hardware-backed key, a dedicated terminal, a physical switch or a second operator.
The specific mechanism matters less than its position outside the agent’s action space. A production agent should operate inside a bounded runtime ; the surrounding system, not the model, owns the legal transitions.
Backups Reveal the Principle
Backups make the problem obvious. If an autonomous production agent can also delete every backup, the backup system belongs to the same failure domain.
production → protected recovery domain
Production may cause recovery data to be created, but it should not be able to destroy protected recovery points. CISA recommends offline, encrypted backups and warns that accessible backups can be found, deleted or encrypted by ransomware. Agents reveal the same structural weakness without requiring a malicious actor: recovery is not independent if the system being recovered holds its deletion authority.
The durable rule is simple:
Recovery must exist outside the authority of the system it is intended to recover.
Small Deterministic Interfaces
The same issue extends beyond cloud infrastructure. Consider a software-defined vehicle connected to an autonomous agent:
probabilistic model → tool → vehicle API → physical world
The question is no longer only whether the software is secure. We must decide which physical capabilities should be available to general-purpose software at all.
An agent may reasonably request, “Set cabin temperature to 20°C.” That does not imply that it should receive an API for arbitrary control over safety-critical actuators.
The more appropriate architecture is:
agent → small deterministic interface → safety-critical system
Some state transitions should not be represented as general-purpose agent tools. This is the same boundary discussed in agentic workflows versus production pipelines : model judgment belongs where the next step is genuinely uncertain; deterministic control belongs where the valid transition is already known.
Reduce the State Space
Modern software security often tries to control an enormous number of possible states correctly. Another approach is to remove dangerous states.
Instead of asking, “How do we ensure the agent never performs this operation?”, ask, “Why is this operation present in the agent’s action space?”
That produces a better objective:
maximum autonomous capability
< maximum safely recoverable damage
Inside its boundary, the agent can be extremely powerful. Outside that boundary, capability should fall sharply. This is not about making AI less capable. It is about building environments in which capability does not imply unlimited authority.
The Return of Dumb Things
We may therefore see a strange consequence of increasingly intelligent software: some parts of computing may intentionally become dumber.
A mechanical switch cannot reinterpret an instruction. An unplugged cable cannot accept an API request. A hardware token cannot be discovered in ~/.aws. An isolated recovery machine does not care how persuasive the agent’s reasoning is. A physical boundary does not need a system prompt telling it to remain a boundary.
For decades, the direction of computing was clear:
hardware → software → virtualization → automation
Agents may create a partial countercurrent:
automation → autonomy → unbounded authority risk → physical constraints
Perhaps cyberpunk was right about the hardware. Perhaps space opera was right about data cylinders. Maybe the future does contain dedicated terminals, isolated machines, physical keys, chunky connectors and enormous switches that go clunk when authority changes.
Not because civilization forgot how to virtualize them, but because after putting autonomous intelligence behind every API, it rediscovered an ancient engineering principle:
Some things should be impossible, not merely forbidden.
The smarter software becomes, the dumber some boundaries should become.