Welcome to
the Forum

ByJeni Barovian, Co-Founding Advisor, TechArena

The shift from AI training to inference and agentic applications is changing how data moves through infrastructure. Mixture-of-experts models, disaggregated inference, and distributed deployments are placing new demands on connectivity within the rack, across clusters, and between data centers. These developments are increasing the pressure on networking within AI infrastructure.

Network architecture increasingly affects GPU utilization, application performance, and the economics of AI deployments. As workloads become more distributed, operators need to consider not only compute capacity but also how efficiently data moves between accelerators, storage systems, and facilities.

At OCP26, I spoke with Amit Sanyal, who leads product marketing for data center networking at HPE. With Juniper Networks fully integrated into HPE, the company is bringing together switching, routing, software, and infrastructure management to support AI connectivity at multiple levels. Our conversation explored what that means for system builders, neoclouds, and enterprise operators, and how HPE is addressing these challenges through its expanded portfolio.

Inference changes what we're asking of the network

For years, AI networking discussions focused primarily on training. Unlike training, which typically involves coordinated processing across large GPU clusters, inference must also accommodate variable request patterns, latency-sensitive responses, and increasingly distributed workloads. As Amit explained, this introduces its own demands:

• Mixture-of-experts models route tokens between specialized experts, expanding communication across accelerators. When those experts reside on different GPUs, the network must support frequent data exchanges without introducing delays that affect response times.
• Disaggregated inference separates prefill and decode into distinct phases, requiring fast data transfers between them. This architecture allows operators to allocate resources independently to each phase, but also depends on efficient communication.
• Larger models span multiple GPUs, increasing the need for high-bandwidth connections. When model parameters are distributed across accelerators, network performance can affect how efficiently those GPUs work together.
• KV caches may be stored outside GPU memory and retrieved as needed, adding another layer of traffic. This approach can help manage limited GPU memory capacity, but retrieving cached data introduces additional bandwidth and latency considerations.

Agentic AI extends this complexity beyond individual facilities: as applications coordinate multiple models, tools, and services, traffic can move between data centers and across wide-area networks (WANs). These applications may also generate sustained activity as agents execute multiple steps, making network reliability and predictable performance important beyond individual inference requests.

For operators investing heavily in AI accelerators, network performance has a direct impact on utilization. HPE says its newest switches are designed to help GPUs spend more time processing and less time waiting on the network. The challenge is to support these different traffic patterns while maintaining high throughput and low latency.

Open always wins

As AI architectures diversify, open networking standards are gaining importance. Amit pointed to the growing adoption of Ethernet in scale-out networks, where InfiniBand has traditionally played a major role. Now, industry efforts are extending open interconnect options into scale-up environments within the rack to provide system designers more flexibility in how they connect accelerators.

The distinction matters because these environments have different performance requirements:

• Scale-up fabrics connect accelerators within tightly integrated systems, where extremely high bandwidth and low latency are critical.
• Scale-out networks link GPU clusters across racks, which require efficient traffic management to accommodate an increasing number of interconnected devices.

Open standards give infrastructure designers more options for meeting those needs without depending exclusively on proprietary interconnects, while allowing them to evaluate technologies based on specific performance requirements.

"Open networking has always won," Amit said.

HPE participates in several initiatives supporting this transition, including the Ultra Ethernet Consortium (UEC), Ultra Accelerator Link (UALink), and Ethernet for Scale-Up Networking (ESUN).

UEC focuses on enhancing Ethernet for demanding AI and high-performance computing workloads, including transport efficiency and congestion management. UALink defines an open, high-bandwidth interconnect for communication between accelerators within scale-up systems. ESUN explores Ethernet-based approaches to scale-up connectivity, extending the technology into environments traditionally served by specialized fabrics. Together, these efforts reflect an industry-wide push toward greater interoperability across AI infrastructure.

HPE is also incorporating emerging standards into its products. The company describes its QFX5250 switch as Ultra Ethernet Transport ready, an example of how these specifications are influencing commercial networking systems.

Why Juniper changed the equation

When HPE announced its plans to acquire Juniper Networks in 2024, CEO Antonio Neri emphasized the opportunity to deliver a modern AI-driven networking fabric for training and deploying AI. The acquisition expanded HPE's capabilities: Juniper brought an established portfolio of high-performance data center switches, routers, and network management software, extending the company's ability to support connectivity across different components of AI infrastructure.

Together, HPE's networking technologies enable communication across GPU clusters and provide routing between data centers, creating opportunities for customers to source more of their AI infrastructure from a single provider rather than integrating technologies from separate vendors.

"We came from Juniper, and now that we are part of HPE, we have the full stack," Amit said.

HPE can address AI environments at different scales, from individual GPU clusters to large, distributed facilities. Rather than treating connectivity as a separate infrastructure decision, operators can evaluate networking as part of the broader system architecture.

Silicon, systems, software, and the full stack

Amit described HPE's approach in three parts:

1. Silicon and systems. HPE uses merchant and custom silicon to support scale-up, scale-out, and scale-across networking, including the QFX5252 switch tray for Ethernet-based scale-up within AMD Helios racks and a fully liquid-cooled, 100-terabit-class scale-out switch. These designs address different levels of AI connectivity, from high-bandwidth communication within accelerator racks to the network fabrics that connect large GPU clusters. The liquid-cooled system also reflects the growing need to manage power and thermal constraints as switching capacity increases.
2. Software and management. Congestion management helps maintain throughput and reduce packet loss across AI clusters. HPE emphasizes rapid fabric deployment for neocloud operators, while agentic AIOps supports troubleshooting, issue resolution, and early detection of potential system and optics failures. For operators deploying thousands of accelerators, network congestion can reduce effective GPU utilization, while automated configuration and diagnostics help teams bring new infrastructure online and identify problems faster.
3. The full stack. With Juniper now part of HPE, the company can deliver networking alongside compute, storage, and management and orchestration software. This integrated approach supports customers from initial setup through ongoing operations. For system builders and neoclouds, tighter integration across components can simplify configuration. It also gives operators a more coordinated approach to monitoring performance and resolving issues across the AI environment.

Not hypothetical: customers are buying

HPE's expanded networking portfolio is already supporting major AI infrastructure projects. In September 2026, the company announced a $1.2 billion order from Vultr for AMD Helios AI Rack by HPE systems, which incorporate purpose-built networking hardware and software. Amit also highlighted HPE's collaboration with Oracle on multi-gigawatt data center deployments.

HPE expects that momentum to continue. At its Networking Investor Day, the company projected Data Center Networking revenue growth at a compound annual growth rate (CAGR) in the low-to-high 50% range through fiscal 2029.

For neocloud providers such as Vultr, bringing new GPU capacity online involves more than installing accelerators. Organizations must also deploy the networking, power, cooling, and management systems required to operate those resources efficiently. At multi-gigawatt scale, infrastructure decisions can have significant implications for deployment timelines, operational complexity, and long-term costs, particularly when operators must coordinate deployments across multiple facilities simultaneously.

By combining networking with compute, storage, and management capabilities, HPE is positioned to help operators address these challenges as they scale.

Final thoughts

My conversation with Amit highlighted just how central networking has become to the future of AI infrastructure. We spend so much time talking about GPUs, compute capacity, and model performance, but the ability to connect those resources efficiently is equally critical. As inference and agentic applications become prevalent, the relationships between compute, connectivity, and overall system performance are becoming more complex.

With Juniper fully integrated into the company, HPE has a full portfolio of solutions to address scale-up connectivity within the rack, scale-out communication across clusters, and scale-across networking between data centers. That coverage matters as customers build increasingly sophisticated AI environments, often with different technologies and operational needs at each level.

Perhaps the most interesting part of our discussion was how HPE is bringing these capabilities together. The combination of networking, compute, storage, and management software creates opportunities to simplify deployment and ongoing operations. For neoclouds and other operators racing to bring AI capacity online, faster infrastructure configuration and more efficient troubleshooting can have a meaningful impact.

As a more heterogeneous era of AI infrastructure emerges, open, standards-based approaches to networking are essential. Operators need to evaluate how their networking choices will accommodate evolving AI workloads, different accelerator configurations, distributed inference, and future capacity expansion. And solution providers need to be ready to deliver these capabilities with an open, interoperable approach. The goal is not simply to connect more hardware, but to build infrastructure that can adapt as operational requirements change.

Ultimately, networking is more central than ever within broader AI infrastructure planning. Efficient connectivity, configuration, and management and an open approach will be essential to maximizing compute utilization and the value of these investments.

‍

Why institutions that keep data on premises need to own the data, models, hardware and software

Sovereign AI is no longer only a national program. Banks, hospitals, law firms, universities and research labs are asking the same question nations ask: Who owns the data, the model, the hardware and the software? In a 2026 BARC survey of 320 companies, 62% named the use of data and AI in core processes as a leading reason to pursue data sovereignty.¹

Helping customers answer that question is the reason Epic Semi exists. We build for organizations that want to own their intelligence, whether a nation, a company or a single team. For an enterprise, that conversation usually begins with the data.

Why the data stays home

In regulated sectors, keeping data on site is often required by law or contract. Hospitals must protect patient records under privacy laws that apply wherever those records are processed. A law firm under a legal hold has to preserve the relevant records and account for where they are kept and who has accessed them. Data residency laws in many countries require certain records to stay within national borders. In Germany and several other European countries, works councils have co-determination rights over systems that handle employee data, and moving that data to an outside service can add another round of negotiation before a rollout.

Most organizations also see local storage as the safer choice. In Cisco’s 2025 Data Privacy Benchmark Study, which drew on 2,600 privacy and security professionals in 12 countries, 90% of organizations said local storage is inherently safer, even though data localization adds operating costs.²

Partners are asking, too. More vendors want to know where their data will be stored before they share it. A supplier sending design files, a pharmaceutical partner sharing trial results or a client handing over contracts for review may ask for written agreements on where that material is stored and who can access it. An institution that keeps its AI in its own data center, on hardware it controls, can answer them directly.

Keeping data close to where it is processed also means fewer transfers. This reduces chances of accidental exposure, avoids cloud egress fees and cuts round trips for AI agents that query the same records repeatedly.

Built around the customer’s needs

Each institution sets its boundaries differently. A hospital network may want agents that summarize clinical notes without any record leaving the hospital’s own systems. A bank may need separate environments for trading, compliance and customer service, each with its own audit trail.

At Epic Semi, we start with the customer’s requirements and design the stack to meet them. Customers come to us for cooling, interconnect, CPUs, accelerators and storage delivered as a full rack, and increasingly for the orchestration, applications and operations that run on top of it.

An architecture that evolves with the work

No two enterprises run the same agent workflows. A bank, a hospital and a manufacturer each split the work differently between the model and everything around it, including retrieval, database queries, tool calls and security checks that run on general-purpose cores. Research on agentic workloads shows the slowest stage changes from one job to the next.³ No one can specify today the exact mix of general-purpose cores, AI cores and memory every enterprise will need, and that mix will shift as we learn from deployments.

We now use AI in our own chip design work, and it is shortening the time between generations. Faster cycles mean each new chip can rebalance cores and memory around the workloads customers run, instead of locking in one design for years.

Intelligence inside your walls

Epic Semi recently announced the launch of Contrail AIX, our Artificial Intelligence eXecution Platform that puts these ideas into hardware that runs inside an enterprise’s own data center. At its center is Contrail Compute 5050, a RISC-V server CPU and AI superchip that places general-purpose and AI cores on one die. Both share one memory and storage space, which reduces data movement as an agent switches between retrieval, tool calls and model execution.

For institutions focused on protecting their data, security must start in the silicon. Contrail Compute includes trusted secure boot and full-chip security management. Full hardware virtualization lets separate teams, such as a bank’s trading and compliance groups, run isolated environments on shared hardware.

Ubuntu, llama.cpp and open-weight models run on Contrail AIX today. The platform ships under a 5A992C export classification that reaches substantially all markets, so later phases of a deployment fall under the same export rules as the first.

Enterprises have spent years deciding where their data should live. The next decision is who owns the AI systems that use it. If you run AI on premises and want that control, bring us your workflows, and we will help you measure them end to end on Contrail AIX.

Endnotes

1. BARC, “Data Sovereignty 2026: Reality, Relevance, Roadmap,” online survey of 320 companies, conducted February and March 2026, published May 6, 2026, https://barc.com/news/data-sovereignty-2026-survey/.

2. Cisco, “Cisco’s 2025 Data Privacy Benchmark Study: Privacy landscape grows increasingly complex in the age of AI,” April 2, 2025, https://newsroom.cisco.com/c/r/newsroom/en/us/a/y2025/m04/cisco-2025-data-privacy-benchmark-study-privacy-landscape-grows-increasingly-complex-in-the-age-of-ai.html.

3. Ritik Raj, Souvik Kundu, Ishita Vohra, Hong Wang and Tushar Krishna, “Towards Understanding, Analyzing, and Optimizing Agentic AI Execution: A CPU-Centric Perspective,” arXiv:2511.00739, submitted Nov. 1, 2025, revised April 16, 2026, https://arxiv.org/abs/2511.00739.

‍

Of the many industries to build AI for, healthcare might be the least forgiving. Patient well-being, regulation, and compliance all require high degrees of precision, with little room for error.

On a recent podcast, I sat down with Nicklas Frahm, director of platform engineering at Corti, a healthcare AI company founded in 2016, to talk about what it takes to build infrastructure for that environment.

From 911 Calls to a Healthcare AI Platform

Nicklas came to Corti as a mechatronics engineer by training rather than as a traditional software engineer. He specialized in embedded systems, which he said pays off now that he works with GPUs.

Corti started in call recording for emergency call centers, where 911 latency made real-time processing a requirement.

Over time, the Corti team discovered it had a talent for infrastructure, a discovery that shaped what the company became: a platform people who develop healthcare applications can build on. Nicklas, who leads a team of about 10, said they “cover the entire stack,” from the hardware that runs the models to the experience of the ML engineers who bring those models to life for customers.

Corti now spans five verticals: medical-grade speech recognition tuned for medical-term recall and a low word error rate; text generation for documents like a SOAP note, a standard writing method health professionals use to record patient visits; an agentic framework that can autonomously handle checking for drug-to-drug interactions and other tasks; medical coding through its Code Like Humans tool, which uses generative AI for ICD-10 coding; and raw frontier LLM access via a lightning-fast OpenAI-compatible API hosted on European infrastructure via Corti Models.

Compliance by Design

I mentioned that healthcare is often associated with the words “compliance,” “regulations” and “precision” because the stakes of getting it wrong are so high. Nicklas said his team builds to meet BSI C5, SOC 2, ISO 27001, ISO 27017, ISO 27018 and HIPAA requirements.

He pushed back on the idea that compliance is friction. “I see compliance as a way to give you a mandate to build things securely and to make sure that you build it at a high quality,” he said.

Compliance and security are part of the design from the start, he said. If you add them later, it can make model development “very, very hard.”

Building a Sovereign Stack

Nicklas said Corti built its own Kubernetes engine using Cluster API and Talos Linux, chosen for security, reusability and adaptability, so it could swap out the underlying substrate as needed.

The new inference clusters running on that engine went live in winter, Nicklas said, and Corti will soon launch a fully sovereign environment decoupled from hyperscalers.

That work responds to real customer demands, including some customers in Switzerland that require no external access to their systems, and to a broader sovereignty debate Nicklas said starts with raw numbers. By his account, the United States owns about 70% of the world’s GPUs. Owning the technology stack by building it yourself, rather than buying it from someone else, is part of how Corti defines sovereignty.

Concerns about the U.S. CLOUD Act, along with country-specific frameworks like Germany’s C5 and France’s HDS, shape which partners European healthcare customers will trust. Corti still relies on Microsoft Azure but chose Scaleway, a European provider expanding into the U.S., as a strategic partner.

What Stays Open, What Stays Proprietary

Corti has contributed upstream to partners like Scaleway when off-the-shelf tools fell short, including custom tooling for leader election on networks it can’t directly access. It leans on open source too, with Nicklas citing Envoy as an example.

He draws a line, though, between what Corti makes open source and what stays proprietary. The inference stack, models and tooling that make the platform work end to end are off-limits, but the Kubernetes engine is generic enough to share.

“You can’t just be a user, you kind of need to contribute back,” he said.

The TechArena Take

Nicklas’ conversation makes a strong case that healthcare AI’s hardest problems lie in the infrastructure underneath the model. He and Corti have found success by considering compliance frameworks and sovereignty concerns during the design stage rather than bolting them on after the fact.

Corti’s bet is that building both in from the start, from meeting data protection standards like HIPAA to owning its own Kubernetes engine to choosing partners by where they operate, is what lets a healthcare AI company move fast. Open-sourcing the generic infrastructure layer while keeping the healthcare-specific stack proprietary is a sensible way to give back without giving away its core differentiation.

To learn more, visit corti.ai, cortilabs.com, or find Corti’s open-source work at kommodity.io.

‍

A decade ago, Cerebras set out to rethink how computers process AI workloads. Rather than adapt conventional chip architectures, the company designed an entirely new processor and computing system built specifically for AI. That early investment is now paying off as inference becomes an increasingly critical part of the market.

I recently sat down with Cerebras Chief Strategy Officer Andy Hock to discuss the growing demand for its approach amid accelerating deployments and new agreements with OpenAI, AWS, and AMD. As Andy explained, inference speed is becoming a competitive differentiator for businesses building AI-powered applications.

A Better Engine for a New Workload

Cerebras began developing its technology when the scale and direction of the AI market were still uncertain. Its founders recognized that AI-enabled workloads had different computational requirements from graphics, rendering, database management, and other applications traditionally served by general-purpose processors.

“That legacy general purpose compute that was extraordinary to other things was in some sense holding us back as a field from realizing that positive transformational potential,” Andy said.

To address those requirements, Cerebras designed a processor, computing system, and supporting software stack specifically for AI. The result was the Wafer Scale Engine (WSE), a processor built on an entire silicon wafer rather than a conventional individual chip. Andy described it as the world’s largest computer chip, more than 50 times larger than previous designs.

Cerebras now uses the architecture for training and inference, including voice applications, reasoning models, and agentic workflows. According to Andy, its systems can execute certain inference workloads approximately ten times faster than comparable conventional processors.

Why Inference Demand Is Accelerating

The rapid growth of inference reflects a fundamental shift in how organizations use AI, Andy said.

“The reason that we’re seeing a hockey stick of growth in inference is because that’s where AI delivers value back to end users and businesses,” he said.

The launch of ChatGPT in 2022 demonstrated that large language models could perform useful work, driving demand to deploy them in real-world applications. As organizations moved from developing models to using them at scale, inference performance became increasingly critical.

Training remains an important market for Cerebras, but inference places greater emphasis on response times, particularly for applications involving complex reasoning or sequences of agentic actions. The need for speed depends more on the use case than on model size. Cerebras supports both frontier-scale and smaller models, with performance expectations shaped by how users interact with them.

‘Every Millisecond Matters’

Andy drew on his previous experience at Google, where researchers spent years examining how search latency affected user adoption and satisfaction.

“What they found was that every millisecond of latency between when you hit enter or click search and being presented an answer, every millisecond matters,” he said.

The same principle increasingly applies to AI applications. Users accustomed to immediate search results may be less willing to wait for chatbots and agents to generate responses, particularly when those delays accumulate across multistep workflows.

For businesses deploying AI, lower latency can improve responsiveness and enable applications that would otherwise be impractical. The challenge is delivering that performance efficiently as demands on models and workflows increase.

Putting Memory Next to Compute

Andy pointed out that the size of the chip isn’t a gimmick.

“We didn’t make this big chip because it looked cool and we didn’t make it big because it was easy,” he said.

AI workloads, he explained, have three fundamental requirements:

1. Substantial computing capacity
2. Memory located close to the processors
3. High-bandwidth data movement

Conventional architectures place compute and memory on separate chips, connected by relatively narrow links.

“What if we could just put the memory along with the compute and then connect it all at really, really high bandwidth?” he said. “That’s exactly what this is.”

The Wafer Scale Engine uses on-chip SRAM, allowing its cores to access nearby memory with extremely low latency. The architecture delivers several orders of magnitude more memory bandwidth than traditional compute-plus-HBM designs, Andy said.

Building the chip was only the first hurdle. Delivering power, cooling, and data to a wafer-sized processor inside a data center required Cerebras to rethink the surrounding system. For larger models and longer contexts, it distributes workloads across multiple wafers and uses disaggregated inference. Through partnerships with AMD and AWS, complementary hardware can handle prefill (processing the input context), while the Wafer Scale Engine handles decode (generating output tokens).

“If we can produce more fast tokens per wafer scale system by using other hardware, it’s better economics for everybody,” Andy said.

Cerebras also announced a next-generation system at Hot Chips that brings additional memory closer to the wafer, expanding the context it can support. This becomes increasingly important as agentic workloads require longer context windows and greater KV cache capacity.

“These challenges don’t go away,” Andy said. “They just evolve as the market does.”

From National Labs to OpenAI

Cerebras’ customer history reflects the evolution of the AI market: the U.S. Department of Energy, which builds supercomputers for science and energy research, was among its first customers and remains one today. GSK followed, using Cerebras technology for model training and inference to accelerate drug discovery.

The company subsequently formed a strategic partnership with G42 to build a sovereign AI cloud in the UAE and develop open-source Arabic language models.

Most recently, Cerebras announced a deal with OpenAI to deploy 750 megawatts of AI compute, alongside new partnerships with AWS and AMD. “One of our systems today takes about 27 kilowatts and we’re talking about deals... on the order of hundreds of megawatts,” Andy said.

Speed as a Moat

Andy said recent conversations with those in the industry reflected growing demand for improved inference. The questions were practical:  

• How do I get access?  
• Which models do you serve?
• Can you host a dedicated instance of ours?

He pointed to AlphaSense, which uses AI to deliver financial and market insights to CFOs, traders, and investors. For professionals conducting research, the ability to ask questions and receive immediate answers is “not just a nice-to-have in that business, it’s a need-to-have.”

Andy sees the same competitive implications across industries.

“If you and somebody else are building an AI agent platform for finance or for health or for industrial robotics, and you have fast answers, fast control, fast responses that deliver value sooner and your competitors don’t, that’s a moat.”

The TechArena Take

Cerebras has spent a decade approaching AI compute from first principles, developing an architecture that challenges conventional processor design.

As inference demand accelerates and agentic AI raises expectations for response times, Cerebras is well positioned to capitalize on the growing need for faster, more efficient computing. Its expanding partnerships also demonstrate how different computing architectures can work together to address the performance and economic requirements of AI at scale.

To learn more or access the Cerebras fast inference cloud, visit cerebras.ai and watch the full conversation.

‍

One thing has become clear: agentic AI is not going to run in just one type of data center. It will spread across enterprise server rooms, colocation facilities, national labs, and sovereign clouds. Each of those environments requires a different combination of compute, storage, and networking, and all of that heterogeneity has to fit into the same physical infrastructure. At the center of it is the rack.

I asked the HPE team to walk me through it at OCP Summit. Strolling from an open-standard ORv3 rack to a 400-kilowatt Cray supercomputing rack, we called it the mild-to-wild tour. My guide was Jason Zeiler, a data center infrastructure lead. Our conversation raised an interesting question about the future of AI infrastructure: Who gets to define the rack? And can one vendor support both industry-standard configurations and highly specialized systems?

Why the rack now matters more than ever

For decades, the rack was largely a standardized enclosure: you picked a width, bolted in servers, and moved on. But the power, cooling, and networking demands of AI are changing how operators approach rack design.

It’s not just about floor space either. As HPE noted in The Next Platform, the industry is now wrangling with power densities ranging from roughly 30 kilowatts to as much as 1 megawatt. While liquid cooling already extends across entire racks to manage the heat, AI infrastructure also calls for sufficient network bandwidth and reliable power delivery.

These considerations become more complicated as operators combine different hardware generations and workload types within the same facility. A rack architecture needs to accommodate not only its current equipment, but also the operational demands of future upgrades.

Rack dimensions and component placement influence how much hardware operators can deploy within a given footprint, as well as how they maintain and service those systems. The mild-to-wild tour demonstrated just how significantly these requirements vary depending on workload and deployment environment.

Mild: ORv3 and the open-standard path

Our first stop was an Open Rack Version 3 (ORv3) configuration, an OCP-standard rack with 21-inch rail-to-rail spacing. Jason described it as the next generation of scalable infrastructure, a repeatable design that uses Open Rack Unit (OU) increments and provides a consistent platform for HPE’s next-generation servers.

“Mild,” however, doesn’t mean modest. Jason was clear that HPE is planning for greater rack density.

“If customers want to fully load racks, they’re also going to have proper cooling and power,” he said.

ORv3 supports native liquid-cooling capabilities, power shelves, and multiple voltage ranges to accommodate higher-density deployments. This is a deliberate design decision to support scalability, not simply a lower-power alternative.

HPE’s new ProLiant Gen13 family reflects this flexibility, with four servers designed for two different rack standards. According to HPCwire, the DL585a and DL525 use conventional 19-inch racks, while the XD245 and XD285 support 21-inch ORv3 configurations for AI and high-performance computing (HPC) workloads. Availability is staggered: HPE expects the systems to become available in phases beginning later in 2026. This gives operators a choice between established rack infrastructure and newer OCP configurations as they plan future deployments.

HPE is also adapting HPE Juniper Networking QFX Series switches for ORv3-compliant deployments. As rack density increases, networking equipment must be considered as part of the rack’s total thermal and power budget, alongside compute.

But rack density is only part of the equation. Organizations also need the flexibility to adapt their infrastructure as workloads evolve.

When I asked Jason who this rack is really for, he described enterprise deployments ranging from half a rack to a thousand racks. Many customers aren’t building enormous, homogeneous clusters. Instead, they want to deploy different types of technology over time. That could mean mixing CPUs, GPUs, storage, and networking within a standardized rack architecture, then adding or replacing systems as workloads evolve. ORv3 provides a consistent form factor for that kind of incremental expansion.

Wild: the Cray GX5000

At the other end of the spectrum is the HPE Cray GX5000, a purpose-built supercomputing system designed for extreme compute density. And the difference in scale is staggering.

Where ORv3 uses 21-inch rail-to-rail spacing, the GX5000 uses a purpose-built 900-millimeter-wide compute rack, compared with the 21-inch equipment architecture used by ORv3. Standing beside the cabinet made its scale immediately apparent. The larger form factor accommodates substantially greater power density.

According to The Next Platform the GX5000 supports up to 400 kilowatts per rack, while liquid cooling extends to the compute blades, networking, power supplies, and even the bus bar. Rather than adding cooling capabilities to a standardized rack, HPE designed the GX5000 around a fully liquid-cooled architecture.

The increase in density is particularly striking when compared with the previous-generation Cray EX4000. Compared with the previous-generation HPE Cray Supercomputing EX4000, the GX5000 reduces the compute-rack footprint by 25%, moving from a 1,200-millimeter-wide cabinet to a 900-millimeter-wide rack while supporting up to 400 kilowatts per rack.

‍

With its planned $115 billion acquisition of optical components maker Lumentum, Marvell Technology is positioned to significantly expand its role in the AI infrastructure market.

The all-stock transaction, announced November 2, 2026, consolidates Marvell’s expertise in high-speed interconnects, network switching, and custom AI silicon with Lumentum’s lasers, optical transceivers, and specialized semiconductor manufacturing. As AI infrastructure operators confront increasing bandwidth, power, and connectivity requirements, the companies plan to develop more tightly integrated electrical and optical solutions to support more demanding workloads.

Unanimously approved by both boards, the agreement specifies that Lumentum shareholders will receive 4.5 Marvell shares for each share held, representing approximately $1,132 per share and a 23.9% premium. Upon closing, Marvell shareholders will own roughly two-thirds of the combined company, with Lumentum shareholders holding the remaining third.

Bringing Electrical and Optical Connectivity Under One Roof

Marvell’s acquisition reflects the growing significance of optical interconnects as AI infrastructure scales beyond the practical limits of copper-based connectivity. Larger AI clusters require increasing bandwidth between accelerators, switches, and distributed systems. While copper remains critical for short-distance connections, maintaining signal integrity, managing power consumption, and extending reach become bigger challenges as data rates and connection distances expand.

Marvell and Lumentum intend to address these constraints through closer coordination of electrical and optical technologies.

Marvell contributes capabilities in electrical and optical interconnects, network switching, custom XPUs, XPU-attach silicon, and advanced semiconductor design. Its existing portfolio includes PAM and coherent digital signal processors (DSPs), Ethernet switching, PCIe and CXL technologies, and custom AI accelerators. Lumentum adds high-performance lasers, optical transceivers, optical circuit switching (OCS), and compound semiconductor manufacturing, particularly indium phosphide (InP) technology.

The companies plan to develop integrated connectivity solutions across four AI networking domains: scale-across, scale-out, scale-up, and scale-in. These domains span connections between geographically distributed data centers, communication within data centers and accelerator clusters, and emerging optical links closer to individual processors and packages.

The acquisition could also accelerate development of co-packaged optics and other architectures that place optical connectivity closer to compute silicon, reducing electrical signaling distances and associated power requirements.

Expanding Manufacturing and Market Opportunities

Lumentum’s manufacturing capabilities represent an important component of the acquisition. The company manufactures high-speed and high-power lasers using specialized compound semiconductor processes, potentially giving Marvell greater control over optical component development, production capacity, and integration with its semiconductor designs.

The transaction also builds on Lumentum’s expanding role in AI infrastructure, including its relationship with NVIDIA and investments in InP laser manufacturing.

Marvell estimates that the combined company will address a $550 billion total addressable market. It also expects stronger long-term revenue growth, improved non-GAAP gross and operating margins, and more than $1 billion in annual run-rate cost synergies.

The transaction is expected to close in the first half of 2028, subject to shareholder approvals, regulatory clearances, and customary closing conditions. Marvell Chairman and CEO Matt Murphy will retain both positions. Three Lumentum directors, including CEO Michael Hurlston, will join Marvell’s existing seven-member board.

The TechArena Take

Marvell’s acquisition signals a strategic shift toward vertically integrated AI connectivity. The company already has substantial experience in electrical processing and semiconductor technologies underlying high-speed optical links. Lumentum adds the lasers, photonics expertise, and specialized fabrication capabilities required to produce the optical side of those connections.

The strategic value becomes clear as AI infrastructure designers evaluate optical alternatives for increasingly demanding scale-up and scale-in networks. Closer coordination between DSP development, laser technology, packaging, and manufacturing could improve performance, power efficiency, and time to market. If successful, the combined organization could become a major supplier of integrated connectivity technologies spanning advanced chip packages, accelerator clusters, and data center interconnects.

‍

I’ve known Chloe Ma, Chief Business Officer of EpicSemi, since our days working together at Intel. Back then, our conversations centered on one company’s technology. On a recent In the Arena episode, we explored a much bigger question: Does agentic AI require a different approach to compute infrastructure?

EpicSemi believes it does. The company is developing a portfolio of technologies for enterprise agentic AI and sovereign AI, spanning compute, storage, interconnects, cooling and security. According to Chloe, conventional CPU-GPU architectures may not deliver the best economics for complex agentic workflows.

From Answers to Outcomes

Chloe sees agentic AI as the point where generative AI moves beyond answering questions to completing complex enterprise tasks.

She illustrated the difference with a workplace example. A manager asks for last quarter’s European sales figures. While a conventional AI system retrieves the data and provides an answer, an agentic system receives a broader assignment: determine why European sales are declining and recommend corrective action. This requires database queries, analysis, planning and coordination across systems, memory access and security controls. An agent may execute multiple steps before producing a useful result.

“It’s not just providing an answer,” Chloe said. “You have to deliver an acceptable outcome.”

This transition has significant implications for infrastructure design. Rather than optimizing individual inference requests, operators need to consider the performance, efficiency and security of entire workflows.

Putting the Sheeter in the Kitchen

Agentic AI workloads involve multiple types of processing. CPUs manage and coordinate tasks, while AI accelerators perform the intensive calculations required for inference. In conventional systems, data must travel between the CPU’s system memory and the GPU’s high-bandwidth memory (HBM). These transfers add latency and consume energy.

Chloe explained the challenge with a croissant-making analogy.

Making laminated dough requires a specialized machine called a sheeter. Imagine preparing dough in a San Francisco kitchen but keeping the sheeter in San Jose. Every time the dough needs another pass through the machine, someone has to transport it between the two cities.

EpicSemi wants to put the sheeter directly in the kitchen.

The company is developing what Chloe described as “a small, complete and fungible unit” of compute. Its chip combines general-purpose performance cores with AI cores, both based on RISC-V. The AI cores use vector and matrix extensions to deliver approximately 150 TOPS (Tera Operations Per Second) of AI performance. This allows the chip to handle both workflow coordination and AI inference without relying on separate CPU and GPU processors.

Both types of cores share system memory and storage, while the chip also incorporates enhanced security features. By keeping compute and memory resources on the same chip, EpicSemi aims to reduce data movement, latency and energy consumption.

“We believe this could be the unit of compute for agentic AI,” Chloe said.

Chloe envisions these units supporting specialized agents for financial analysis, legal review and IT operations. Because many expert models are considerably smaller than the frontier-scale alternatives, operators could keep them resident in memory and allocate compute resources through software orchestration.

Measuring Completed Tasks, Not Just Tokens

Much of traditional AI infrastructure was designed around the requirements of large-scale model training and high-throughput inference. GPU clusters remain effective for those workloads, but Chloe argues that agentic AI requires different performance metrics.

She drew a parallel with the transition from mainframes to warehouse-scale computing, when operators adopted large clusters of commodity servers to improve cost efficiency and scalability.

For agentic AI, Chloe argues that operators should measure how many tasks they can complete securely within a given power or total cost of ownership (TCO) budget.

“It’s really about how many agentic tasks that you actually securely complete within your power budget or within your TCO budget,” Chloe said.

That shifts the focus from maximizing raw compute performance to improving the efficiency of the entire workflow, including orchestration, memory access and inference.

Sovereignty Goes Beyond Data

As enterprises and nations invest in AI, control over the underlying infrastructure is becoming increasingly important.

Chloe described three dimensions of sovereignty:

1. Data sovereignty concerns control over information.
2. Infrastructure sovereignty encompasses silicon, software, security and supply chains
3. Operational sovereignty requires the expertise to deploy, manage and maintain these systems independently.

EpicSemi uses the open RISC-V instruction set architecture (ISA) to support this approach. Unlike proprietary ISAs, it allows organizations to develop compatible processors without relying on a single architecture vendor.

According to Chloe, this gives nations the freedom to choose among suppliers and potentially develop domestic silicon capabilities.

EpicSemi operates in both Santa Clara and Saudi Arabia, and sovereign AI initiatives in the latter have attracted substantial investment. Across the Gulf Cooperation Council (GCC) region, organizations are investing in infrastructure and Arabic language models to meet regional needs.

Chloe emphasized that sovereignty also depends on practical capabilities. Hardware alone cannot provide operational independence. Organizations also need the software, security capabilities, supply chain relationships and technical expertise to manage their infrastructure.

Software Will Determine Adoption

EpicSemi is working with early adopters, sovereign AI initiatives, and software partners to support deployment of its architecture. Agentic workflow orchestration is particularly important: the software must assign tasks to the appropriate processors, coordinate specialized agents and allocate resources efficiently.

“Without these type of software, our hardware would be useless,” Chloe said.

The company is also pursuing opportunities with enterprises and AI startups in financial services, investment and manufacturing, where organizations increasingly combine conventional applications with AI-driven workflows.

For EpicSemi, the next challenge is to demonstrate measurable performance and efficiency gains across representative agentic workloads while building the software ecosystem required for broader adoption.

The TechArena Take

My biggest takeaway from my conversation with Chloe is that agentic AI could change how we evaluate infrastructure performance.

Tokens per second and raw accelerator throughput remain important, but they don’t capture the full cost of completing a complex, multistep task. Operators also need to account for orchestration, data movement, security, latency and overall resource utilization.

What interests me about EpicSemi’s approach is the focus on these requirements at the architectural level. Its RISC-V design combines general-purpose and specialized compute with shared memory, while its sovereignty strategy gives customers additional choices over how and where they deploy AI infrastructure.

The next step is validation. I’ll be watching EpicSemi’s agentic benchmark results and the development of its software ecosystem. Both will be critical to determining whether smaller, flexible compute units can deliver better performance per watt and per dollar than conventional architectures for enterprise agentic AI.

Listen to the full In the Arena conversation with Chloe Ma to learn more about EpicSemi.

‍

Qualcomm announced Monday that its first-generation High Bandwidth Compute (HBC) silicon has passed validation in the company’s San Diego labs, delivering 18 times the effective memory bandwidth of external Low-Power Double Data Rate 5X (LPDDR5X) and more than five times the bandwidth per watt of High Bandwidth Memory 4 (HBM4) specifications.

The result moves the stacked memory architecture toward commercialization in the Qualcomm Dragonfly AI250 inference accelerator, which is set to begin sampling in the first half of 2027. They also bring an ambition into reality. Qualcomm has argued that memory, not compute, limits AI inference in the agentic era. These results show that HBC is a viable answer to the memory wall, where moving model data between memory and compute consumes much of a system’s time and energy.

Compute Beneath the Memory Stack

As Qualcomm’s SVP & Head of DataCenter Engineering Rashid Attar explained in the announcement and in a recent webinar on TechArena, HBC stacks LPDDR Dynamic Random-Access Memory (DRAM) dies directly on top of an active AI accelerator base die, using wafer-to-wafer direct bonding and through-silicon vias. HBM, by contrast, places DRAM over a base logic die that connects back to a host processor across an ultra-wide memory bus.

Putting compute at the bottom of the stack shortens the distance data travels, which is where much of the energy goes. Because the base die handles the most bandwidth-intensive operations, its link to the main processor needs far less bandwidth.

What Qualcomm Measured

Engineers drove the silicon on industry-standard automated test equipment through production-representative data paths, using randomized weights and activations at production-representative dimensions. The tester stood in for the accelerator. It set a baseline in LPDDR5X mode, then ran an AI workload in HBC mode and compared execution time and energy.

Source: Qualcomm

Qualcomm described the results as estimates, normalizing measured silicon results to target operating conditions and comparing them with published competitor specifications. The company said redundancy and remapping are built in to work around memory defects, and that it has developed and characterized wafer-to-wafer bonding with foundry and assembly partners across multiple technology generations.

The TechArena Take

When Qualcomm introduced HBC, it made a specific argument: in the agentic era, inference performance would be won or lost in memory, and moving compute beneath the memory stack was the inside path to victory. Monday’s results put that idea into working silicon. The 18x bandwidth and 5x-plus efficiency gains now rest on a measured chip rather than an architecture slide, and AI250 sampling remains on track for the first half of 2027.

That proof lands at the right moment. Agentic workloads keep long contexts and KV caches resident, which puts memory capacity and bandwidth under constant pressure. An architecture that improves both gives operators a credible path to lower cost per token. Validated silicon shows Qualcomm’s thesis holding up and gives operators a credible path to faster, cheaper agentic inference. The next proof point is AI250 itself, and whether the gains hold at rack scale on real agentic workloads when samples arrive in 2027.

For more on Dragonfly, HBC and Qualcomm’s data center strategy, visit the Qualcomm Dragonfly Media Platform on TechArena.

Rack densities in AI data centers are climbing faster than anyone predicted. NVIDIA’s NVL72 racks already draw well over 100 kilowatts, and the upcoming NVL576 is expected to push past 600 kilowatts. Cooling is no longer an afterthought, but one of the defining constraints on AI infrastructure deployment, performance, and economics.

At AI Infra Summit, TechArena’s Allyson Klein sat down with Jeff Moore, VP of Strategic Partnerships at Airsys, to discuss these challenges. Airsys has 30 years’ experience, having started in comfort cooling before expanding into mission-critical applications. Its technology has since earned a TechArena award for innovation in AI infrastructure.

Moore’s central message: most of the industry isn’t moving to liquid cooling overnight, and solutions must accommodate existing infrastructure while preparing for higher-density compute.

Liquid Cooling Is a Transition, Not a Switch

Hyperscalers and large-scale AI factories face the most extreme cooling requirements, but many facilities are transitioning more gradually, with GPU platforms that can still be supported by air cooling. Operators want to maximize existing space and power capacity while introducing liquid cooling without extensive capital investment or facility redesign. Supporting that phased transition, rather than forcing a complete overhaul, is where Moore sees the strongest fit for the Airsys portfolio.

Every Watt Has a Thermal Obligation

As densities rise, operators want to direct as much of their available power as possible toward compute. “Every watt that goes to the GPU has a thermal obligation with it,” Jeff explained. Cooling infrastructure must therefore accommodate both current thermal loads and the requirements of future silicon generations.

Compressed timelines add further pressure. Every delay between hardware installation and operational readiness represents lost revenue potential, so cooling has to be delivered and installed on schedule. Airsys supports production from its new half-million-square-foot global headquarters campus in Woodruff, South Carolina, as well as manufacturing facilities in Hungary and China.

Jeff also pointed to growing silicon diversity as inference spreads beyond a single vendor. GPU power requirements are increasing, while emerging 800-volt DC architectures are changing how power is distributed within AI infrastructure. Airsys emphasizes design flexibility and rapid prototyping to accommodate evolving customer specifications, component configurations, and thermal requirements.

3D Printing Changes the Cold Plate Equation

One of Airsys’s key technical differentiators is additive manufacturing. Jeff recalled standing in a booth at OCP two years ago with Airsys founder Yunshui Chen, discussing 3D-printed cold plates when industry adoption was still limited. Today, multiple companies have introduced alternative manufacturing techniques, from ECAM to stack forging.

Airsys uses laser powder bed fusion, an additive manufacturing process established in aerospace that builds metal components layer by layer from digital designs. The company works primarily with aluminum and copper but can incorporate other materials to meet specific performance requirements. Printing enables more complex internal structures, increasing the surface area available for heat transfer and improving thermal performance.

The approach also shortens design and prototyping cycles. As NVIDIA accelerates its GPU development roadmap, cooling suppliers must adapt to new thermal and mechanical requirements on increasingly compressed timelines. Where conventional cold plate manufacturing may require new tooling, additive manufacturing enables Airsys to iterate designs without the same production constraints.

Moving Beyond PUE to Power Compute Effectiveness

Power usage effectiveness (PUE) has long been the industry’s standard metric for data center energy efficiency. But PUE measures total facility energy consumption relative to IT equipment energy use, rather than how productively that energy is converted into compute. Allyson, who worked on PUE during her time at Intel, observed that the industry may have placed too much emphasis on infrastructure efficiency without adequately accounting for computational output.

Airsys has introduced a complementary metric: power compute effectiveness (PCE). Where PUE measures operating energy efficiency, PCE measures allocation or the share of a facility's provisioned power that's available for compute. The more power that can go to compute, the more AI capacity a site can support within the same grid connection.

Brownfield Facilities Aren't Left Behind

AI infrastructure conversations often focus on greenfield builds, but many operators must accommodate higher-density workloads within existing data centers equipped with legacy systems like computer room air conditioning (CRAC) units. Jeff, who recently joined a panel with Equinix and colocation providers, emphasized that these purpose-built facilities need a practical upgrade path as customer requirements evolve.

The answer from Airsys is LiquidPack, a single-phase solution designed to fit within a standard rack footprint. It uses contained impingement spray cooling, with servers installed in cassettes and substantially less fluid than conventional immersion cooling. Because it connects to existing facility water infrastructure, operators can introduce liquid cooling without a facility overhaul.

Jeff said operator response has been very positive. Applications extend to the edge, where Airsys already cools converted cell sites and telecom properties, as well as modular data center deployments.

Think in Cooling Loops, Not Cold Plates

When Airsys first began developing 3D-printed cold plates, the goal was removing two kilowatts of heat. Today, servers integrate multiple GPUs and CPUs within increasingly dense rack-scale architectures, and focusing on the cold plate alone won't cut it.

Jeff argued for evaluating the complete thermal management system, including piping, coolant properties, phase-change behavior in two-phase architectures, manifolds, and the coolant distribution unit (CDU). System-level instrumentation provides the visibility needed to monitor fluid behavior, manage operating conditions, and optimize heat removal across the entire cooling loop.

Two-Phase Cooling, Done Responsibly

Two-phase cooling offers the potential for greater heat transfer efficiency, but refrigerant selection introduces environmental, safety, and operational considerations. Jeff sees progress in the development of PFAS-free alternatives. “I think we’re going to innovate our way forward,” he said, pointing to new refrigerants that have emerged over the past six to 12 months.

Single-phase cooling remains well established, with additive manufacturing extending its capabilities through more effective cold plate designs. However, increasing thermal loads will eventually push the practical limits of coolant flow rates, operating temperatures, and heat removal capacity. Two-phase systems offer another path forward, provided the industry can establish reliable, safe, and scalable approaches to deployment.

The TechArena Take

AI infrastructure is advancing faster than traditional cooling architectures can comfortably accommodate. Yet the transition to liquid cooling won't happen overnight, and operators need solutions that address immediate thermal demands while leaving room for future generations of compute.

Airsys is approaching that challenge from multiple directions, combining additive manufacturing, system-level thermal management, and a portfolio spanning air, single-phase liquid, and emerging two-phase technologies. Its emphasis on power compute effectiveness also reflects a broader shift in how the industry evaluates infrastructure: by how much provisioned power a facility makes available for compute.

Ultimately, the next generation of AI cooling will require greater flexibility across technologies, facility types, and deployment models. Every additional watt directed toward compute creates a corresponding thermal obligation and meeting that demand will be essential to scaling AI infrastructure efficiently.

To hear more about Airsys and their innovative approach to AI cooling, listen to the full episode.

Governments and regulated enterprises have moved sovereign AI from policy to procurement. The builders serving them say those buyers now require proof of control: where the infrastructure sits, who runs it, who can reach the data and how the buyer plans for what it cannot control.

Executive Summary

Sovereign AI has moved from policy to procurement, and the governments and regulated enterprises buying it answer to legislatures, regulators and national security processes. That accountability shapes what they require from the companies that build and run their AI infrastructure. This paper sets out four of those requirements, drawing on authoritative voices building sovereign AI infrastructure in Europe and beyond. Those requirements start with a working definition of control. From there, buyers require infrastructure whose location they can verify against a standard, along with operators and access controls they can audit, down to who holds the encryption keys. They also require a plan for the inputs no single buyer fully controls, from the supply of cutting-edge accelerators to access to frontier models. The paper closes with what a sovereign buyer should ask to see before signing.

Proof, Not Promises

Mistral closed a €3 billion Series D on Sept. 8, 2026, the largest equity round ever completed by a European technology company.  The company plans to scale European compute capacity to 1GW by 2030 and to offer regional data processing to governments and companies that want AI independence from US providers. In late July, the EU opened its call for up to seven AI gigafactories, with €10 billion in public funds and at least €20 billion in expected private investment behind it.

Capacity is only half of what sovereign AI buyers need. The builders serving them say those buyers also have to prove they control it.

“‘Trust us’ isn’t an acceptable answer to a government customer. ‘Here’s how you verify it yourself’ is,” said Nidhi Chappell, Nscale’s president of AI infrastructure. “That shift — from performance-first to verifiable-first — is the biggest change in how I design.”

The shift follows from who the customer answers to. Large hyperscalers design to optimize throughput, cost per token and uptime for a single company’s roadmap, Chappell said.

“Government and regulated entities aren’t just optimizing for performance — they’re accountable to a legislature, a regulator, or a national security process,” she said. “That changes the requirements: auditability, chain of custody for hardware and data, procurement cycles that need more time, and a need to prove things rather than simply deliver them.”

Who Is Buying, and Why Now

Sovereign buyers have moved from talk to action. Eighteen months ago, their posture was rhetorical, said Jonathan Symonds, chief marketing officer at Radiant, the vertically integrated AI infrastructure company Brookfield launched in February.  Now, governments are executing.

“Sovereign nations are also much more serious than they were,” Symonds said. “They are openly putting RFPs into the market.”

The trend is strongest in Europe and extends to South Korea, the United Arab Emirates and Saudi Arabia. Those buyers want speed, a fast path to first production capacity and a clear roadmap to self-sufficiency, he said.

Export controls on AI models also put model providers, not just governments, in the market for compute.

“The first beneficiary of the export controls were other model providers with fewer political entanglements,” Symonds said. “Cohere and Mistral are excellent examples.”

Those model providers now shop for compute in ways they did not six to nine months ago.

Scaleway sells control without the sovereign label. Albane Bruyas, chief commercial officer and chief operating officer at the French cloud provider, avoids the word because its definitions vary so widely.

“What we sell more and what is very interesting for our clients is that we sell a full autonomy,” she said during a recent TechArena podcast. “This means that we have control on all the value chain. This is what makes the pure difference for our clients.”

Gartner forecasts $80 billion in sovereign cloud infrastructure-as-a-service spending this year, a 35.6% increase over 2025, with governments as the main buyers.  Gartner also expects more than 75% of European and Middle Eastern enterprises to geopatriate their workloads by 2030, moving them out of global public clouds and into local options, up from less than 5% in 2025.  Nscale closed a $2 billion Series C at a $14.6 billion valuation in March.

The US CLOUD Act lets American authorities compel US-headquartered providers to hand over data they hold abroad.  Under that law, a provider’s headquarters, not only the location of its data center, determines which government can demand a buyer’s data. In Europe, the EU AI Act requires organizations that deploy high-risk AI systems to keep the logs those systems generate for at least six months, to the extent the logs are under their control. Those obligations take effect Dec. 2, 2027.

Control They Can Define

A buyer has to define control before it can verify it. Chappell defines it this way.

“For us, sovereignty means enabling governments and enterprises to run critical AI workloads under their own legal, operational and security frameworks — with clear control over where infrastructure sits, who operates it, who can access it, and which laws apply,” she said. “That’s not only about where data resides; it’s about control of the compute infrastructure underneath it.”

A sovereign can take more control over some inputs than others. Symonds ranks the inputs a sovereign needs, from data and land to models and compute, by how much control a sovereign can take over each. Data comes first.

“Data is the fuel,” he said. “You wouldn’t bother with sovereignty if you weren’t starting from the data layer.”

After data, Symonds ranks the remaining inputs in this order:

• Powered land
• Running the system itself, or trusting the partner that runs it
• Models
• Build capability
• Networking
• Compute

“You can own and operate the GPUs, but you cannot meaningfully control the upstream supply of cutting-edge accelerators,” Symonds said. “One does not homegrow cutting edge GPU capacity overnight.”

“Control what you can, invest there, reduce risk everywhere else,” he said.

That rule orders a sovereign buyer’s requirements. Buyers verify the inputs they control: where the infrastructure sits, who operates it and who can reach the data. They plan for the inputs they cannot fully control: accelerator supply, access to frontier models and the suppliers they depend upon.

A Location They Can Verify

Location is the first requirement a sovereign buyer verifies. In a standard public cloud region, Chappell said, “‘region’ is often a configuration setting.” Workloads and data can be routed, replicated or failed over across borders by default, and “a customer largely has to take the provider’s word for where things actually run.”

Nscale’s partnership with BT takes a different approach.

“With BT, the sites, the network paths, and the operational access are fixed to UK jurisdiction and governed by BT’s existing security and compliance frameworks, not a config toggle that could quietly change,” Chappell said.

In practice, the physical location of the hardware is known, fixed and auditable. Data does not leave the jurisdiction as part of normal operation.

“The whole environment can be independently verified against a standard, rather than just asserted,” she said.

Nscale matches location to the workload.

“Not every country needs its own hyperscale campus, so the model we’re pursuing is federated,” Chappell said.

Large training campuses sit where power is abundant and low-cost, in the Nordics and Iberia. Those campuses feed smaller in-region or in-country clusters for workloads that have to stay local. BT supplies the in-country half in the UK, with national telco infrastructure and sovereign compute layered on top, serving public sector and enterprise customers.

Location also depends on land and power, the physical inputs Symonds ranks highest.

“You can and should control your physical inputs,” he said.

In a true sovereign sale, he said, the sovereign likely owns the land and the power. Radiant designs, builds and turns up the AI factory, finances it if needed and operates it up to a point.

Power is the requirement buyers most underestimate, Chappell said.

“Most sovereign programs plan the compute procurement first and the power second, and power is what actually gates the timeline,” she said.

Nscale owns its power envelope, which changes how it designs.

“When you own the power envelope, you’re not just backing into a wattage number after the fact,” Chappell said.

Nscale designs density and storage placement around the actual power profile of each site from the start, and it throttles GPU draw dynamically rather than overprovisioning.

Operators and Access They Can Audit

The next set of requirements covers people. Symonds ranks self-operation second, right after powered land. A sovereign has to build the capacity to run its own system, even in phases, he said, or place true trust in an operating partner.

At BT, only vetted personnel operating under UK law can reach the control plane, Chappell said.

“Baseline, not premium,” Chappell said. “In a sovereign or regulated context, those aren’t things a customer upgrades into later — they’re the entry requirement to even be considered for the bid. We don’t access or inspect customer GPU workloads, and that isolation, alongside encryption, verifiable erase and firmware integrity, is built into every deployment, not sold as an add-on tier.”

Access requirements extend to the data itself. In an air-gapped, on-premises build, weights and data persist at rest in the storage system. Symonds said the deciding question there is “who has technical authority over the storage system: who controls access, encryption keys, updates, telemetry and administrative privileges.”

Most modern storage runs as software on standard server and flash hardware, Symonds said. He advises buyers to choose that software from a vendor they trust and architect it so the sovereign operator, not the vendor, retains control of keys, access, software updates and external connectivity.

Vendor domicile still counts. Some countries will not buy storage from companies domiciled in certain other countries, Symonds said, and “that is a legitimate procurement and supply-chain consideration.” Domicile does not replace technical control.

“Vendor domicile and government exposure remain relevant risk factors, but they are not substitutes for those technical controls,” he said.

Auditing access gets harder outside the data center. Chappell calls extending real-time response to the edge, at the same security bar as the core, the hardest thing a sovereign builder has to deliver. A core facility takes some of its trust from physical design: badges, perimeters, cameras. An edge node lacks those by definition, “so you have to rebuild that trust in hardware and software instead of walls, at the same latency budget.”

“The underlying principle is that physical security can’t be your only control once the node leaves the building,” she said.

Proof does not replace trust in the provider. Symonds said Radiant’s customers know the company can deliver because it holds capital, powered land, compute access and software in house.

“They have confidence we are going to be around in 18, 36, 48 months,” he said. “In that sense, trust is another differentiator.”

A Plan for What They Can’t Control

A buyer that verifies every input it controls still depends on inputs it cannot fully control. Symonds expects compute to be the hardest constraint three years out. Hyperscalers, frontier labs and major AI enterprises will continue to absorb a disproportionate share of leading-edge capacity, he said. Most governments will build 50MW to 500MW in that window, and multi-gigawatt scale will elude almost all of them.

Those governments can build data centers and procure models, Symonds said. GPUs and the networking to keep them fed will be harder to secure unless buyers place orders well ahead of time. Public-sector procurement tends to value optionality and staged commitment.

“Getting things built favors the bold,” he said.

Scarce compute raises the value of every GPU a buyer secures.

“Storage decides whether the expensive compute you’ve built actually gets used — that’s a direct line to unit economics,” Chappell said.

AI agents, software that plans and carries out multistep tasks with limited human direction, sharpen the problem. Agentic inference brings unpredictable access patterns as many agents hit shared context or memory stores at the same moment.

“Storage breaks first,” Chappell said.

Under that load, peak demand matters more than the average.

“Architectures tuned purely to averages will look efficient on paper and fail live,” she said.

Models are the second input a buyer cannot fully control. Discussing the US-China contest over model weights, Symonds reduced control to one test.

“This is all about control and, in this context, control is the ability to rescind access,” he said.

Washington and Beijing increasingly regard the weights themselves as strategic technology, Symonds said. In his reading, China wants the ability to pull its free but very good models if the US pulls its expensive but cutting-edge ones. Open weights work the other way.

“Once the weights have been openly released, they are extraordinarily difficult to claw back,” he said. “Open source models offer an opportunity for sovereigns to have real control and real personalization.”

His advice to buyers: “de-risk your model supply chain now.”

Suppliers are the third. Digital buyers too often forget a rule the industry treats as basic, said Albane Bruyas of Scaleway. A company that depends on one strategic supplier has no fallback if that supplier fails, raises prices or receives a data request from a foreign government. In industry, she said, a buyer of a strategic component chooses at least two suppliers, so a fire at one factory still leaves somewhere to buy.

Open hardware makes the second supplier possible, she added. The more open the hardware, the more suppliers an operator can buy from, and more suppliers mean better prices and more capacity. Scaleway buys European when it can, because in her view purchase orders, not grants, build an ecosystem.

Owning the full stack reduces dependency from the other direction. Nscale owns its power, data centers, compute and software, which removes subcontractors from the chain a sovereign buyer has to trace. Radiant builds on the same model.

“Our core advantage was, and continues to be vertical integration,” Symonds said. “Others can match different components, capital, powered land, compute access and expertise, software, but no one yet has demonstrated the ability to unify them all.”

The TechArena Take

Verifiability now stands alongside performance as a requirement in sovereign AI procurement. We expect the sovereign builds of the next three years, most of them in the 50MW to 500MW range Symonds describes, to go to builders who can show a buyer, line by line, which inputs the buyer controls and how a third party can check each one.

• The standard against which the site’s jurisdiction is independently verified.
• The people who can reach the control plane and the law they operate under.
• Who holds the keys, updates and telemetry for the systems where data and weights persist.
• The power plan, before the compute order.
• The plan for accelerator supply, model supply and a second supplier.

We see verification becoming part of the product itself. Builders that document jurisdiction, operator access and data control before the first workload runs give buyers what a legislature or regulator will ask to see. We expect those builders to turn sovereign RFPs into production capacity fastest. For sovereign AI buyers, proof is now part of what they pay for.

‍

Download the Paper

‍

‍

1. Rebecca Szkutak, “Mistral raises €3B as sovereign AI becomes big business,”TechCrunch, Sept. 8, 2026. [Confirm byline.] https://techcrunch.com/2026/09/08/mistral-raises-e3b-as-sovereign-ai-becomes-big-business/

2. European Commission, “EU launches AI Gigafactories call to boost Europe’scomputing capacity and unlock more than €30 billion in investment,” July 30,2026. https://digital-strategy.ec.europa.eu/en/news/eu-launches-ai-gigafactories-call-boost-europes-computing-capacity-and-unlock-more-eu30-billion

3. Deanna Oothoudt, “Brookfield Bets on Sovereign AI With Radiant and Ori Deal,”TechArena, February 2026. https://techarena.ai/content/brookfield-bets-on-sovereign-ai-with-radiant-and-ori-deal

4.Gartner, “Gartner Says Worldwide Sovereign Cloud IaaS Spending Will Total $80Billion in 2026,” Feb. 9, 2026. https://www.gartner.com/en/newsroom/press-releases/2026-02-09-gartner-says-worldwide-sovereign-cloud-iaas-spending-will-total-us-dollars-80-billion-in-2026

5. Gartner, “Gartner Identifies the Top Strategic Technology Trends for 2026,”Oct. 20, 2025. https://www.gartner.com/en/newsroom/press-releases/2025-10-20-gartner-identifies-the-top-strategic-technology-trends-for-2026

6. Nscale, “Nscale Raises $2 Billion in Series C — the Largest in EuropeanHistory,” press release, March 2026. https://www.nscale.com/press-releases/nscale-series-c

7. Jennifer Daskal, “Unpacking the CLOUD Act,” eucrim. https://eucrim.eu/articles/unpacking-cloud-act/

I founded TechArena on a simple belief: the best advancements in technology come from a wider range of voices than our industry usually puts on stage. That’s why this month’s cancelation of the Grace Hopper conference hit so hard. For more than 30 years, Grace Hopper has been where women in computing found mentors, sponsors, and their first real sense that they belonged in this industry. Grace Hopper also leaned into allyship, itself building the widest expanse possible for connection and community.

Anyone who has sat in a room or attended a conference where no one resembles you knows how important this is. Its loss is more than a canceled event. It leaves a hole in the pipeline at the exact moment the leaders of the AI era are being chosen. I’ve spent my career in this industry, from Intel and Micron, to building TechArena, and I know how much of career advancement happens through mentors, allies, champions, and people willing to tell you the unwritten rules. When the rooms where that happens disappear, talented people stall.

That’s why we’re launching TechArena Advisory mentoring now with a sprint of speed-round sessions. This program pairs emerging leaders with our advisors and leaders from our network. Operators who have run businesses, launched platforms, and made the hard calls at the highest levels of tech are dedicating their availability to engage. Our Advisors are collectively grateful and humbled by the collective outpouring of participation from executives who have shaped our industry for decades.

Anyone who is seeking help with connection or earned insight can simply visit our website and sign up for an available slot between November 2nd and 4th. Lakecia Gunter and Raejeanne Skillern, our TechArena Advisors who were planning to teach at Grace Hopper this year, alongside VSORA Chair Sandra Rivera, will also be delivering their course as planned online for all to access. The course is titled, "From Impact to Influence: How Women in Tech Can Prepare for Board Roles." And we're offering any other presenters the opportunity to host sessions on our platform as well.

This program comes from the same commitment to talent across all vectors of inclusion that shapes whose voices we publish, who we put on stage, and how we build our own team. If we want an AI era that works for everyone (and we certainly do), the people building it have to come from everywhere, and that work can’t pause because a conference did.

Storage spent most of the last two decades as the part of the server nobody had to think about very hard. It was evaluated on capacity, reliability and cost per gigabyte, and the interesting engineering happened elsewhere. That is not where the industry is today. As deployments shift from training to inference, the question of where a model keeps its working context has become a performance problem with real money attached to it. I recently sat down with Garrett McKibben, senior director of technical marketing at Graid Technology, to talk about what inference workloads are doing to the memory and storage hierarchy. Garrett and I have been talking about the blending of that hierarchy for a long time, and this conversation was the first one where the economics of it felt concrete.

The One Lane Bridge

Graid Technology has been in market for about five or six years, built around a premise that sounds almost contrarian: remove the RAID controller entirely, let host memory talk directly to the SSDs, and accelerate the I/O path with the GPU. The reason comes down to arithmetic.

“Any single PCIe by 16 Gen 4 was limited to 30 gigs of throughput. Gen 5 is limited to 60 gigs of throughput,” Garrett said. “The analogy we like to use is you've got a four lane highway and to get to your storage, you have to go down to a one lane bridge.”

Buyers have invested heavily in NVMe performance, then sipped it through what he called a tiny coffee straw. That mismatch was the company’s original opening. It has since expanded. The company acquired the VROC assets from Intel roughly a year ago, with VROC by Graid Technology arriving in the coming months. Now it is pointing its efforts toward applying the same I/O thinking from its founding to agentic AI and KV cache.

The Economics of Remembering

As attention shifts from training to inference, the gating factor is not compute. Garrett pointed instead at capacity and the cost per gigabyte across the hierarchy. HBM on a Blackwell part tops out near 180 gigabytes at $50 to $80 per gigabyte. System DRAM improves the economics at $5 to $8 per gigabyte but remains finite. NVMe runs $0.08 to $0.12 per gigabyte with effectively unbounded capacity.

The instinct when a model runs out of room for its KV cache is to add GPU memory. Garrett’s math says that path closes quickly.

“With that limited amount of HBM, in our calculations, you run out of concurrent users after about three or four users,” he said. “As much as maybe NVIDIA would love each of us to go buy a new Blackwell 200 for every three users, it’s not feasible.”

Fetch Versus Recompute

The test for any storage tier in this role is simple to state and harder to satisfy. A GPU that loses its context can always recompute it, and it does so quickly. Storage only earns its place when retrieval beats recalculation.

“Fundamentally, fetch has to be faster than recompute,” Garrett said. “If it takes longer for the GPU to go fetch it…than it takes to compute, then there’s no value.”

Graid Technology published results measuring exactly that. With no KV Cache, recomputation took 29.5 seconds; adding MD RAID raised it to 36.6 seconds. SupremeRAID reduced the KV-cache retrieval path to 9 seconds—3.2x faster than recomputing the context. Garrett was careful not to oversell the finding, noting it holds in that particular scenario rather than every case, but the direction is the point: with an unconstrained link to NVMe and long context windows, fetching wins.

The GPU-side approach costs less than most people assume. The software-only version runs on the GPUs already in the box, consuming roughly 4 percent of capacity, about six streaming multiprocessors, and releasing them the moment the I/O completes. That leaves the CPU free for the applications it is increasingly responsible for orchestrating.

What Rack Scale Changes

The shift from components to rack-level systems makes the storage question harder rather than easier. A request may be initiated on one GPU and serviced elsewhere, which means the context has to be findable across the rack or the pod without adding back the latency the design was meant to remove. Graid Technology is working with open source frameworks including LMCache and Mooncake toward a single namespace that can meet those latency requirements.

Garrett framed why any of this matters with a simple metaphor. He compared a GPU without its KV cache to walking into a room and forgetting why you went in, then having to retrace your steps. With agentic workloads running for hours or days, that retracing is not a minor inefficiency.

“There’s a difference between having them fully utilized recalculating the same thing time and time again versus retrieving it and then moving to the next step,” he said. “It’s no longer about FLOPS. It's about enabling your engineers or your users to get results through their agents.”

The TechArena Take

The blending of the memory and storage hierarchy has been a talking point for the better part of a decade. KV cache overflow now gives it a number attached to it. Fetch versus recompute is a clean test, and it changes the buying conversation: a team with a system that retrieves context instead of recomputing it is not saving on storage, it is recovering GPU cycles it already paid for. That is a materially different business case, and it lands with a different set of decision makers.

The open question is rack scale. Single-server results are encouraging, but inference is becoming a distributed problem, and locating a given KV cache across a pod without reintroducing latency remains unsolved. We’ll be watching which approaches holds up as the unit of deployment keeps getting bigger.

‍

The first generation of AI infrastructure was built primarily for training: moving enormous datasets fast enough to keep GPUs productive through large, compute-intensive runs. Production inference introduces a different set of demands. Time to first token, token throughput, latency, tokens per watt, and customer experience all influence how operators measure success. Agentic AI adds another layer of complexity, with longer, multistep workflows that generate and reuse growing amounts of context.

At Yotta 2026, TechArena CEO and founder Allyson Klein moderated two panels exploring these implications across AI infrastructure. The first, “The Inference Storage Problem: Rethinking Data Architecture for AI at Scale,” examined the demands inference and agentic workloads place on compute, networking, memory, and storage. It brought together Marc Austin, CEO and co-founder of Hedgehog; Sam Clarke, vice president of neocloud and AI business development at VDURA; Angel Camacho, director of product marketing at Western Digital; and Betsy Chernoff, principal AI product marketing manager at WEKA.

The second panel, “Tokens per Watt: Efficiency as the Fastest Path to AI Capacity,” extended the conversation to power, cooling, and facility infrastructure. Chernoff returned, joined by Peter Panfil of Vertiv and Sean Corcoran, Vice President of Sales for PowerOne at Airsys. They explored how increasingly dynamic AI workloads affect power demand, thermal management, and the physical limits of data center capacity.

Together, the discussions revealed a common challenge: delivering more useful AI output requires coordinating resources across the entire infrastructure stack. As inference becomes a larger share of AI workloads, operators must consider how compute, memory, storage, networking, power, and cooling work together to determine performance, efficiency, and capacity. From the conversations, several key insights emerged.

Inference Changes What Infrastructure Needs to Optimize For

Operators increasingly need infrastructure that serves training and inference simultaneously. While training prioritizes completing large compute jobs and keeping GPUs supplied with data, inference introduces new performance requirements. Distributed inference also places greater demands on networking and data movement as pilots scale into production. Angel noted that pilots can comfortably hold large amounts of data in RAM, but that approach becomes prohibitively expensive as applications expand to hundreds or thousands of users.

GPU utilization can also be misleading. An accelerator may remain highly utilized while repeating computations unnecessarily, so operators need to evaluate useful output, response times, and the energy required to generate each token.

“Our job is really to make sure that the GPUs...keep working, keep the beast fed, and also on the inference side, make sure they’re not doing the work twice,” Sam said.

These changing workloads also affect facility infrastructure. Peter described GPU demand shifting rapidly between relatively low and high loads. Electrical systems experience those fluctuations immediately, while thermal systems respond to the resulting heat with a delay. Power and cooling must therefore adapt to increasingly dynamic compute requirements.

“This is not our father’s cloud world anymore,” Peter said. “This is a dynamic world in which the systems are interdependent.”

Inference efficiency, in other words, runs from the application through compute, memory, storage, and networking, all the way to power delivery and heat rejection.

Keeping GPUs Productive Requires Managing Data and Context Intelligently

Agentic AI makes context management an increasingly important infrastructure challenge: agents perform multistep tasks that depend on information generated earlier in the workflow, requiring systems to retain and retrieve growing amounts of context. The key-value (KV) cache preserves this information so models can reuse previous computations.

“Context matters. It’s important that our agents remember what we’ve been doing,” Angel said. “Otherwise, we will have to recompute all that we’ve been doing, spending GPU time.”

High-bandwidth memory (HBM) provides fast access to data but remains expensive and capacity-constrained. Betsy described how storage can extend the memory hierarchy by holding KV-cache data that doesn’t need to remain in HBM, then returning it when required. “We were able to take from high bandwidth memory all of this cache and hold it on storage and then send it back to the GPUs as quick as DRAM or as quick as the system needs,” she said.

This approach allows frequently accessed context to remain close to the GPU while less urgent information moves to lower-cost storage tiers. It also helps reduce unnecessary recomputation, freeing accelerator capacity for new work.

Betsy emphasized that different stages of inference place different demands on compute and memory. Prefill is compute-intensive, while decoding depends more heavily on memory bandwidth. As context windows expand and agentic workflows become more complex, managing data placement across memory and storage becomes increasingly important.

The implications extend to existing GPU fleets. Software-defined infrastructure, GPU mobility, fewer unnecessary data copies, and efficient offloading can help operators extract more useful work from deployed accelerators.

Every unnecessary GPU cycle consumes power and limits available compute capacity, making effective context management critical to inference performance and infrastructure economics.

The Bottleneck Is Moving Beyond the GPU

As inference workloads become more complex, networking plays a greater role in bridging compute with the memory and storage resources it needs. Marc described the network as a bus connecting different levels of the memory hierarchy, from on-chip SRAM to flash storage.

That relationship has implications for network topology too. Training and inference may require different architectures, particularly as storage becomes more directly involved in moving KV-cache data. Disaggregated inference adds another consideration, allowing different stages of a workload to run on resources suited to their computational requirements, while latency-sensitive applications may also drive infrastructure closer to end users.

The constraints also extend beyond data movement. Higher rack densities have increased the importance of thermal management, with direct-to-chip liquid cooling becoming central to supporting more powerful accelerators. Air cooling still has a role, making hybrid approaches important as facilities accommodate a wider range of equipment.

These decisions require closer coordination between mechanical, electrical, and compute teams, since cooling architecture affects facility power consumption and how much capacity remains available for IT equipment.

“It has to be the whole architecture of the data center,” Sean said, describing a design approach that extends “from the chiller to the chip.”

As accelerators become more powerful, infrastructure design increasingly determines how effectively their capabilities can be used.

Optimize the System, Not Just Its Components

Enhancing the performance of an individual component doesn’t necessarily improve the efficiency of the entire AI infrastructure system. Decisions about compute, memory, networking, storage, power, and cooling need to account for how those resources interact before deployment.

“The biggest piece is to think strategically around how you architect across your infrastructure,” Betsy said. “Certainly starting from the GPU, but in addition to that, you have to consider networking [and] storage...well before you start to build.”

The same principle applies to facility infrastructure. “It’s a system. It’s got to be treated like a system,” Peter said. Optimizing individual subsystems can improve their performance without delivering corresponding gains across the facility.

The growing importance of storage illustrates how quickly infrastructure priorities can shift. Betsy noted that roughly a year and a half earlier, storage received little attention in AI infrastructure discussions because it represented a relatively small share of the bill of materials. Today, its role in supporting inference makes it a much more consequential architectural consideration. The challenge is to evaluate design decisions by their effects on overall performance, capacity, and efficiency, rather than the capabilities of any single component.

New Workloads Require New Measures of Efficiency

As inference becomes a larger share of AI workloads, operators need metrics that connect infrastructure performance to useful output.

Angel emphasized the importance of defining success before deciding what to measure. “First of all, metrics. You cannot find success if you don’t measure it,” he said. “You need to first establish what you really want to do. What is the outcome that you are looking for?”

That principle extends to facility infrastructure, where conventional efficiency metrics don’t always capture how much useful AI work a data center produces. Peter advocated measuring tokens per second per watt to connect computational throughput with energy efficiency.

Sean introduced power compute effectiveness (PCE), a metric examining how effectively provisioned building power supports provisioned IT load. At data center scale, even modest improvements in electrical and thermal efficiency can translate into meaningful capacity and cost benefits.

The shift is reaching individual infrastructure components, too. Betsy noted that WEKA’s benchmarks now include tokens per watt per second, reflecting growing interest in evaluating storage through its contribution to overall AI efficiency.

Together, these measures help operators connect workload performance, infrastructure utilization, and facility economics.

The TechArena Take: Start With the Outcome, Then Architect Backward

The central lesson from both panels is that AI infrastructure design must begin with the workload and the outcomes operators need to achieve. Establish the metrics that define success, then architect backward across compute, memory, networking, storage, power, and cooling.

Peter explained the risk of approaching those decisions independently: “If we treat it as the individual subcomponents, we might be able to optimize the performance on one of those subcomponents, but we’re not going to optimize the performance out of the system.”

As inference and agentic AI reshape infrastructure requirements, the ability to coordinate resources across the entire data center will increasingly determine how much useful work operators can deliver from their available capacity. In the inference era, the data center is no longer just the place where AI runs, but part of the AI system itself.