Model-centric AI safety misses system-level organoid risk

What the work claims

Vautravers, Chalkley, Downer, Kate S, and Ruck argue that current AI evaluation is model-centric and therefore cannot answer the "so what?" question: a measurable shift in a model's output does not, by itself, reveal whether that shift will cause harm once the model is embedded in a complex sociotechnical system.¹ They propose a three-stage framework that links structured hazard analysis, component-level testing, and probabilistic system modelling so that regulators and operators can trace a failure from a model's behaviour to a system-level loss.

The authors apply the framework to the UK's Real Time Gross Settlement (RTGS) system, which settles more than GBP 700 billion on an average working day. They use Systems Theoretic Process Analysis (STPA) to identify over 100 unsafe control actions and eight AI-driven loss scenarios, then test one scenario experimentally: adversarial manipulation of a large language model (LLM) used to advise trading allocations. They project the measured allocation shifts into a financial contagion model of 50 banks and find that adversarially shifted behaviour raises bank failure rates and lowers the shock threshold that triggers cascading disruption, especially under widespread or monopolistic AI adoption.

How it works

The framework has three stages. Stage A is structured hazard analysis. The authors use STPA, which treats safety as an emergent property of control and feedback rather than as a list of component failures. They define system losses, hazardous states, and constraints; model the control structure of the RTGS ecosystem, including the Bank of England, direct participants, and the SWIFT messaging layer; identify unsafe control actions; and develop loss scenarios that explain how AI adoption could contribute to a harm.

Stage B is component-level testing. The authors build a stylized LLM-based portfolio advisor that ingests four synthetic market-news articles per episode and recommends allocations across equities, mortgage-backed securities, corporate bonds, and government bonds. They test three foundation models under neutral sentiment, bearish sentiment for one asset, and bearish sentiment plus a simple adversarial prompt injection targeted at the distressed asset. They find that even simple adversarial text measurably shifts the recommended allocation, with GPT-5-mini showing the largest swing toward selling the distressed asset and Claude 3.5 Haiku producing only 50 adversarial measurements because the model was retired during testing.

Stage C is system-level modelling. The authors construct a stylized financial contagion model with 50 banks arranged in a core-periphery structure, extending the Eisenberg-Noe framework with fire-sale dynamics. They map the component-level allocation shift into a fire-sale intensity parameter and run Monte Carlo parameter sweeps over exogenous mortgage shocks. The result is a quantitative relationship between adversarially induced allocation shifts and bank failure rates.

Where a skeptic should push

The most load-bearing assumption is the mapping from component behaviour to system parameter. The authors are explicit that their financial model is illustrative and not a high-fidelity analysis of RTGS. The value of K=1.0 and the other mapping choices are modeling decisions, not empirically derived constants; changing them changes the quantitative output. The qualitative pattern - that adversarial shifts worsen systemic outcomes - is robust across parameter choices in the appendix, but the headline numbers should be treated as existence proofs, not forecasts.

The component experiment is stylized. The market-news articles are generated by foundation models, the portfolio task is a four-asset allocation, and the adversarial attacks are simple, low-sophistication prompt injections. This is deliberate: it tests whether the failure mode can be made to manifest at all. But it does not establish how often such attacks would succeed in a real trading environment with filtering, human review, and adversarial defenses. The authors note that prompt hardening was highly effective at the stylized scale, though that does not guarantee robustness against adaptive attacks.

The paper is also a preprint and has not undergone peer review. Its worked example is finance, not biotechnology, so any transfer to organoid intelligence platforms is analogical and must be justified rather than assumed.

What system-level AI harms mean for organoid governance

The non-obvious implication is that organoid intelligence platforms are exactly the kind of sociotechnical system the paper describes. They combine a biological component (living human neural tissue), a hardware component (MEAs, amplifiers, stimulators), a software component (spike sorting, feature extraction, control algorithms), and a human component (experimenters, reviewers, vendors, regulators). A failure in any one layer can propagate through feedback loops that do not exist when the layers are tested separately. An LLM that reads MEA data and proposes a stimulation pattern is not just a model; it is a controller acting on living tissue.

The opportunity is methodological. The framework gives regulators and platform operators a way to reason about organoid AI systems before a harm occurs. STPA can identify loss scenarios such as an AI controller driving the tissue into a pathological excitatory state, an opaque feature extractor hiding signs of organized activity, or correlated automation bias across multiple labs using the same vendor model. Component-level tests can then measure whether those failure modes are real, and system-level models can estimate how they would scale if the platform were adopted widely.

The threat is that the field is currently moving in the opposite direction. Most AI safety work in organoid intelligence is model-centric: benchmarks for spike prediction, metrics for stimulus-response fidelity, or evaluations of closed-loop training algorithms. These are necessary but not sufficient. A controller that scores well on a benchmark may still destabilize tissue when connected to a specific MEA, culture protocol, and experimental goal. If multiple labs adopt the same closed-loop AI vendor, the correlated behaviour that worried the Bank of England in finance becomes a correlated failure mode in living-tissue computing.

The governance consequence is that evidence-based, anticipatory oversight needs system-level tests as a condition of deployment, not just model cards and benchmark scores. For organoid platforms, that means requiring hazard analyses that span the tissue, the instrument, the algorithm, and the operator; requiring component tests that probe adversarial or out-of-distribution inputs to the control loop; and requiring transparent mappings from those component measurements to predicted system-level outcomes. Without this, the first sign of a systemic harm may be an unexpected pattern in a dish of human neurons.

The bottom line

The paper is a position and framework, not an empirical finding about organoids. It demonstrates that simple adversarial inputs can shift LLM recommendations in a stylized trading task and that, under a specific component-to-system mapping, those shifts increase systemic financial fragility. The central claim - that AI governance must move from model-centric evaluation to system-level harm quantification - is well supported by the worked example. For organoid intelligence, the framework is a blueprint for oversight rather than a finished assessment: it shows what kind of evidence would be needed to claim that a closed-loop AI platform is safe to deploy at scale.