AI Is Not Software
Establishes why deterministic security assumptions fail when policy compliance is an inference problem under uncertainty.
Scenario excerpt
A helpful enterprise assistant turns a plausible answer into an irreversible operational action.
Exploit-path names
- Ambiguity to wrong action — Ambiguity yields a plausible inference that becomes an irreversible operational action.
- Long-context constraint loss — Constraints degrade under length and salience competition.
- Automation bias — Operators accept plausible trajectories without verifying authority.
Root causes
- Probabilistic instruction weighting — Policy compliance is inferred, not deterministically enforced.
- Tool-calling expands blast radius — Language failure becomes system change.
Impact categories
- Financial: Fraud, overpayment, or unauthorized transactions.
- Operational: Data corruption or unintended workflow completion.
- Compliance: Mishandled regulated data without a recoverable evidence trail.
Countermeasure preview
- Gate irreversible actions with deterministic policy checks.
- Require traceable evidence for consequential execution.
- Separate inference from authority to act.