IBM Research reports achieving a 52-times design speedup in high-level synthesis estimates for quantum error correction hardware using AI agents, and at a strikingly low cost of just $26 for inference. This advance demonstrates that substantial hardware optimization is possible with a minimal inference budget, alongside the smallest area design.

Researchers combined open frontier models with llm-d’s low-cost distributed inference to create a system capable of actively participating in hardware design. “Model capability, the assignment of agent roles, and the infrastructure serving those models all shape the hardware improvements that are attainable within a real engineering budget,” the team states.

Open Frontier Models Lower Quantum Error Correction Inference Costs

This level of performance was enabled by an agent factory, a system where a coordinator agent directs parallel worker agents to develop and synthesize alternative designs within a defined search budget. The factory’s success hinged on a specific approach to cost management, using open frontier models in conjunction with llm-d, an open-source distributed inference framework. This demonstration moved beyond simply assisting with code writing; the AI agents actively participated in the hardware design process.

The team evaluated their approach using Relay-BP, a quantum error correction decoding algorithm for the [[144,12,12]] bivariate bicycle code, exploring 100 hypotheses per configuration. Costs associated with these explorations covered model inference, the financial implications are significant, not the complete hardware development workflow. A comparable configuration relying entirely on proprietary frontier models incurred a cost 11.5 times higher, reaching $299 under the same search budget.

The efficiency gains stem from several architectural choices within the agent factory. The system replaced sequential reductions with tree structures, exposing opportunities for parallel computation, and connected code changes to their hardware effects through repeated evaluation. Efficient serving further reduced costs by reusing computation associated with shared context, a capability facilitated by llm-d’s cache-aware routing. This routing directs requests to servers holding reusable context, and separates prompt processing from token generation, allowing for independent resource allocation.

“A verifier checks that the proposed improvements are valid, and the designs are ranked by latency and hardware resource use. Those results guide the next round of exploration, allowing the factory to refine its search as it discovers better designs,” explained the researchers. While all three configurations, open, hybrid, and fully proprietary, yielded verified hardware improvements, they produced distinct tradeoffs.

Open and hybrid searches prioritized compact designs, while the proprietary configuration more effectively converted additional resources into parallelism and lower latency. Each configuration was evaluated in a single search run, meaning the results reflect observed searches rather than a definitive ranking of model performance. The ability to conduct more affordable searches opens the door to testing a greater number of ideas, and with improved search strategies, potentially leading to better chips.

This cost-efficiency is particularly relevant as enterprises seek to control rising inference costs and maintain control over sensitive data. The demonstration showed that deploying open-weight models on-premises through llm-d’s distributed inference stack can dramatically lower serving costs while still uncovering valuable hardware optimizations. The team’s findings suggest a viable path toward making AI-driven chip design more accessible and practical for a wider range of organizations.

An agent factory turns hardware optimization into a coordinated cycle of exploration: A coordinator agent directs parallel worker agents to develop and synthesize alternative designs within a defined search budget.
Ruchir Puri, IBM fellow and IBM Research chief scientist

AI Agent Factories Accelerate Relay-BP Decoder Design 52x

This result, detailed in a recent paper, highlights a pathway toward more accessible automated exploration of hardware designs. While a fully proprietary model configuration achieved a higher 204x speedup, the open-model approach delivered substantial optimization at a significantly lower financial cost. The cost differential is striking; the proprietary configuration required $299 in inference costs to reach its peak performance, while the open frontier model configuration completed the same task for $26.

This 11.5-times lower cost stems from the efficient serving of tokens, with the open-model search using approximately 241 million tokens compared to 216 million for the proprietary search. The team achieved a 98% prefix-cache hit rate and a blended serving cost of roughly $0.11 per million tokens, deploying the factory alongside other agent workloads on shared infrastructure.

This infrastructure allowed the team to retain control over the design artifacts processed by the agents, a key benefit of using open-weight models on-premises. Relay-BP decoders, with their repeated message passing, present a useful test case due to their inherent parallelism, although baseline implementations often struggle with sparse graph representations and memory access patterns. The team’s work suggests that the practical value of AI-assisted chip design isn’t solely determined by the best design an agent can produce, but also by the amount of verified exploration an organization can realistically afford.

The team’s findings also demonstrate that open-weight models can be deployed on-premises, allowing for greater control over design artifacts. This is particularly relevant for sensitive applications where data security and intellectual property protection are paramount.

Distributed Inference with llm-d Reduces Design Exploration Budgets

This result hinged on the use of an agent factory, a system where AI agents collaboratively propose, implement, and verify hardware changes, but the cost of running such a system has historically been a barrier to widespread adoption. In contrast, a configuration using open frontier models throughout the process completed the same exploration for just $26, a cost reduction of 11.5 times. This finding suggests that the economic benefits of open-source AI extend beyond reduced licensing fees, impacting the operational expenses of hardware design itself.

The team’s approach used llm-d, an open-source distributed inference framework, to optimize resource allocation and minimize redundant computation. Efficient serving, according to the researchers, reuses computation associated with shared context, further reducing costs and accelerating the design process. The study focused on Relay-BP decoders, a type of error correction code, as a challenging test case for AI-driven hardware exploration.

These decoders require agents to preserve computational integrity while simultaneously optimizing hardware performance, demanding a sophisticated level of reasoning and adaptation. The researchers intentionally constrained all configurations to the same 100-hypothesis budget, ensuring a fair comparison of cost-effectiveness. The open-model configuration, despite achieving a lower peak speedup than the proprietary setup, still delivered a substantial hardware optimization alongside the smallest area design. This cost advantage is not merely about reducing expenses. It’s about expanding the scope of exploration.

Lowering the barrier to entry allows organizations to conduct more design iterations, investigate a wider range of architectural possibilities, and ultimately, potentially discover superior designs. The reciprocal relationship between better hardware and more capable AI is now more accessible, supporting a cycle of continuous improvement in both domains.

Source: https://research.ibm.com/blog/open-agent-factories-and-distributed-inference-for-chip-design

More like this