LLM-guided search co-designs neural networks for physical hardware

What the work claims

King and Leleu introduce Unconventional Hardware Neural Architecture Search (UH-NAS), a framework that uses a large language model as an evolutionary operator to search for neural architectures that co-optimize task accuracy and inference energy.¹ The central claim is that the same search procedure can work across heterogeneous hardware because each platform is represented by a replaceable backend that exposes only an energy model, physical constraints, and a non-ideality simulator.

The authors test the claim on three backends: a CPU (Intel Xeon 8380, FP32), a GPU (NVIDIA Blackwell B200, FP16), and a simulated silicon-photonics MZI mesh optical neural network (8-bit). On MNIST, UH-NAS reaches 97.00 percent top-1 accuracy under worst-case optical non-idealities when using GPT-4.1 as the LLM backbone, and it explores 203 unique architectures out of 250 evaluations, compared with 26 for the same pipeline without LLM guidance.

How it works

The framework is built around three abstractions. First, a hardware backend object that reports the energy cost, precision, and physical constraints of each operation. Table 1 in the paper shows the spread: a linear multiply-accumulate costs 91.7 pJ on the CPU, 0.89 pJ on the GPU, and 0.02 pJ on the optical MZI mesh, while optical nonlinearities, DACs, and ADCs add overhead that partly offsets the per-operation advantage.¹

Second, a non-ideality simulator for the optical backend. The authors model phase error, thermal crosstalk, and gamma noise at three levels: realistic (phase error 0.01, crosstalk 0.01, gamma noise 0.001), medium (0.03, 0.08, 0.003), and worst case (0.05, 0.15, 0.005). Every candidate architecture is trained and scored under the relevant noise regime, not on clean accuracy.

Third, an LLM-guided evolutionary loop. At each generation the model updates a knowledge base of hardware-aware design heuristics, proposes architectural mutations or crossovers conditioned on that context, and feeds candidates into a 20-epoch training loop. NSGA-II maintains a Pareto front of accuracy and energy. The search runs for 30 generations with 8 candidates per generation, and the strongest LLM backbone, GPT-4.1, converges to roughly 97 percent accuracy within 10 to 15 generations under worst-case optical noise.

Where a skeptic should push

The experiments are on MNIST, a ten-class digit task that is far simpler than the workloads that would matter for real-world physical accelerators. A method that works on MNIST may not scale to larger models, harder datasets, or tasks with temporal structure such as spiking neural networks. The authors explicitly note that UH-NAS assumes feedforward architectures and has not been extended to spiking neuromorphic or quantum hardware.

Push also on the energy numbers. They are analytical proxies, not measured silicon energy for a fabricated chip. The optical backend is simulated, and the DAC and ADC overheads are modeled rather than bench-tested. The four-orders-of-magnitude spread in linear-MAC energy is therefore a useful design signal, not a verified system-level efficiency claim.

Finally, the LLM is doing real work, but its role is partly opaque. Stronger models outperform weaker ones, and removing the hardware-informed system prompt reduces accuracy, yet the prompt itself is constructed by the authors and the LLM's reasoning is not fully inspectable. For a field that may eventually design controllers for biological neural tissue, that opacity is a governance concern, not just an engineering one.

What hardware-agnostic NAS means for biocomputing platforms

The non-obvious implication is methodological: UH-NAS offers a way to compare unconventional computing platforms without letting any one vendor's benchmark define the race. Because the search algorithm stays fixed while the backend swaps out, the framework can expose whether a given platform's advantage is real or an artifact of an architecture tuned for a different hardware family. For organoid intelligence and biocomputing, where candidate substrates include CMOS neuromorphic chips, optical accelerators, and eventually ionic or biological interfaces, that kind of neutral comparator is badly needed.

The vendor angle is equally direct. Today, platform vendors often report peak TOPS per watt or ideal-case accuracy. UH-NAS instead forces architectures to be re-optimized per backend under that backend's own noise and precision constraints. The result is a Pareto front of accuracy versus energy that is specific to each platform. Vendors who want to sell into this evaluation regime will have to expose not just headline specs but also noise models, energy costs per operation, and calibration assumptions. That shifts bargaining power from marketing toward reproducible benchmarking.

The governance implication is about verification. The paper shows that architectures optimized under clean conditions fail under realistic non-idealities, and that zero-cost proxies commonly used to speed up architecture search turn negatively correlated with accuracy once noise is included. If future systems use physical neural networks to stimulate, record from, or emulate living neural tissue, the same gap between clean-model performance and real-device behavior becomes a safety issue. A framework that scores every candidate under simulated non-ideality is not a substitute for empirical validation, but it is a necessary precondition for any governance regime that wants to enforce robustness before deployment.

The opportunity is a common evaluation language for heterogeneous computing substrates. The threat is that the method's reliance on opaque LLM heuristics and proprietary foundation models could concentrate authority over which architectures get explored in the hands of whichever labs can afford the best LLM access, even as the hardware backends themselves become more diverse.

The bottom line

Established: UH-NAS co-optimizes neural architectures for accuracy and energy across CPU, GPU, and simulated MZI optical backends using a hardware-agnostic backend abstraction and an LLM-guided evolutionary loop, reaching 97.00 percent MNIST accuracy under worst-case optical non-idealities and exploring substantially more unique architectures than a non-LLM version. Not established: whether the same approach scales to harder tasks, spiking or biological substrates, or fabricated hardware with real energy and noise measurements. What would confirm the optimistic reading is extension to organoid-interface or neuromorphic platforms with published noise models and open backend specifications. What would confirm the cautious reading is the framework being used to green-light physical controllers for living tissue before those empirical validations exist.