Welcome to
the Forum

Quantum technology has spent years as a domain defined by its promise rather than its deployments. That is beginning to change. The question now is not if quantum belongs on technological roadmaps for real business environments but how quickly it can be made reliable, integrated and usable.

That challenge sits at the center of the work being done at Q-CTRL. Recently, I had the chance to explore that work with the company's vice president of product, Alex Shih, alongside my "Data Insights" cohost, Solidigm's Jeniece Wnorowski. What emerged from the conversation is a clear view that the path for quantum, from laboratory demonstration to enterprise deployment, runs directly through software.

From Deep Science to Deployable Solutions

Q-CTRL's core work is translating quantum physics and quantum control expertise into products that customers can run. Alex described his role as taking deep science and turning it into scalable solutions for data centers, high-performance computing facilities and enterprise businesses. The company also delivers a quantum education platform to help organizations build and scale quantum talent, along with quantum sensing solutions for navigation.

Alex came to the field after a career spanning big technology companies, startups and space technology, most recently at Slack. He was drawn to quantum for its long-term potential. "I've always viewed quantum as a generational step-change technology that would introduce a new compute paradigm," he said.

Controlling Instability With Software

The central technical challenges in quantum systems are hardware instability, driven in large part by noise from both the hardware itself and the surrounding environment, and the difficulty of scaling that hardware. Q-CTRL develops what Alex described as AI-powered error-reduction quantum infrastructure software, which reduces that noise and suppresses the resulting errors so the hardware behaves reliably enough to build on.

That reliability serves two audiences. For hardware teams, it helps accelerate the development of their qubits. For enterprises developing and testing applications on real quantum hardware, it means the output can be trusted, which, as Alex noted, is what allows customers to properly assess how their applications perform.

Built for the Real World, Not the Journal

Alex was emphatic that Q-CTRL's purpose is to put quantum tools into working business environments. "While important to showcase progress in the field, we are not in the business of just publishing papers but rather removing barriers for customers to use our tools," he said. The goal, he explained, is for those tools to be accessible and used by companies around the world.

That focus is reflected in a recent deployment. Q-CTRL went live in Colorado with Elevate Quantum in a time frame that Alex described as a record for the industry, moving from the announcement of a planned system to a working one in only five months, at a fraction of the typical cost. He framed the achievement as a blueprint rather than a one-time result, expecting subsequent deployments to move even faster. "We've seen Murphy's Law at play; we've seen everything that can go wrong, did go wrong. But now we have ways to mitigate that," Alex said.

Broadening the User Base

Q-CTRL has been serving quantum algorithm developers through Fire Opal, which focuses on error reduction for algorithmic execution, and Boulder Opal, delivering autocalibration for hardware vendors. Since then, the company has built layers of abstraction that expand the range of people it can serve, including domain experts such as chemistry analysts and optimization specialists who can submit data in formats familiar to them.

Application developers represent another audience. Companies including ColibriTD in France and Qunova Computing in South Korea have integrated their own applications on top of Q-CTRL's core pipeline and reported improved performance. The company's education platform, Black Opal, meanwhile, targets traditional software developers who are interested in quantum and continues to drive adoption.

Toward Mainstream Integration

Looking ahead, Alex expressed confidence that quantum will integrate into enterprise workflows and production technology stacks. He pointed to Q-CTRL's work with Riken in Japan, home to the Fugaku supercomputer, an IBM Quantum System Two and a Quantinuum system, as an example of exploring hybrid workflows that combine quantum and classical compute.

Based on hardware vendor roadmaps and the company's experience with customer problems, Alex sees commercial quantum computing advantage in just a few years,   following the company's recent demonstration of practical quantum advantage. "We have confidence that in the next two to three years, we will see meaningful quantum advantage in specific workflows, especially around optimization," he said. He cited a case study with Mazda, where quantum methods required five times less training data to reach a vehicle frame design the company had previously optimized with best-in-class tools, translating directly into reduced research and development cost.

The TechArena Take

Quantum computing has long carried the reputation of a technology perpetually a few years away. What stands out in Q-CTRL's approach is its insistence on accessibility and deployment. By positioning software as the connective layer that brings hardware, control electronics and classical infrastructure together, the company is helping shorten a development cycle that has historically demanded teams of specialists and significant capital. For technology decision-makers, the practical takeaway is that quantum commercial readiness may hinge less on any single hardware breakthrough and more on the software and integration work that makes these systems performant at scale.

Learn more about Q-CTRL at q-ctrl.com, or watch the full podcast episode for more information.

As the AI Infra Summit approaches, TechArena has been chatting with the builders of AI infrastructure about how their requirements are changing as deployments scale.

We sat down with Marc Austin, CEO and co-founder of Hedgehog, a company that builds open software-defined networking for AI clusters. Hedgehog’s approach is mattering more now, as AI infrastructure spreads from single racks to multiple sites and a single misconfigured switch can leave GPUs across a cluster waiting on the network.

We talked about how Hedgehog’s software replaces switch-by-switch network engineering with a declarative operating model, why the company holds reference architecture validation from both NVIDIA and the Open Compute Project, and what it takes to keep job completion times down as clusters scale across sites. Here’s what we learned.

Q1: Hedgehog builds an open software-defined fabric that operators manage as one system rather than as a network stitched together switch by switch. What does that let an operator do at scale that the box-by-box model can’t?

A: It turns the network from a monthslong engineering project into a product you deploy.  

Designing an AI network from scratch means making thousands of interdependent decisions (topology, QoS, congestion control, failure domains) and then keeping them consistent across hundreds of switches by hand.  

With Hedgehog, an operator declares the cluster they want, and the fabric handles Day-0 through Day-N automatically: provisioning, lossless Ethernet tuning, congestion-aware routing and failure recovery, all through Kubernetes-native operations.  

That’s what lets a lean cloud team network like a hyperscaler.  

Q2: Hedgehog supports the NVIDIA Cloud Partner reference architecture, validated for Spectrum-X, and the Open Compute Project (OCP) reference architectures for AI training and inference. Why does that dual validation matter to an operator deciding how to build an AI cluster today?

A: At NVIDIA GTC 2026, we announced support for NVIDIA Spectrum-X Ethernet and the NVIDIA Cloud Partner reference architecture. At the 2026 OCP EMEA Summit, our AI training and inference fabric designs became OCP Accepted: 100% open source, hyperscaler reviewed, with complete BOMs available today on the OCP Marketplace.  

An operator can standardize on Spectrum-X, deploy a fully open SONiC-based fabric on OCP hardware, or run both, and keep a single declarative operating model across all of it. No other network vendor gives operators that freedom at the architecture layer.

Q3: In an AI cluster, every idle GPU is usually waiting on the network. What does the fabric have to get right to keep job completion times down, and why is that harder to do with closed, manually operated networks?

A: The metric that matters is job completion time, and the network defends it or destroys it.  

The fabric has to get five things right at once: lossless Ethernet with properly configured RoCEv2 QoS; congestion-aware routing; rail-optimized topology; dual-plane resilience, so a single failure never stalls a training run; and telemetry with automated recovery, so problems are fixed faster than a GPU can starve.  

In a closed, manually operated network, each of those is a specialist tuning exercise repeated per deployment. Any one misconfigured switch quietly taxes every job on the cluster. Hedgehog ships them as validated defaults in the reference architecture and enforces them continuously in software so that the fabric stays out of the GPUs’ way.

Q4: As infrastructure becomes composable, the network becomes the reassembling component. What does that demand from network software?

A: Composability is fundamentally a network abstraction problem, and hyperscalers already showed the answer: open networking plus VPC abstractions. Hedgehog brings that same model to AI clusters.  

Tenants get VPCs that can span multiple accelerator types and pods, with policy-based multitenant isolation, composed and recomposed in software rather than recabled. We’ve proven this live with disaggregated inference: NVIDIA GPUs handling prefill, SambaNova RDUs handling decode and Intel Xeon orchestration, all running across one Hedgehog fabric with no network penalty.  

Because the whole thing is operated through a declarative, Kubernetes-native model, AI infrastructure can be run with a small team.

Q5: As clusters grow, multiply, and spread across sites, where does network software have to go next? What role do open reference architectures play in letting the industry scale without every operator redesigning the network from scratch?

A: Network software has to scale in two directions at once: up and out.  

“Up” means prescriptive scale-out designs, with our OCP reference architectures defining Open Pod Group scalable units from 64 to 1,024 xPUs and a roadmap to Ultra Ethernet as the interconnect evolves.  

“Out” means treating many pods, clusters, and sites as one operable system, with hybrid multicloud routing and multitenant security built in rather than bolted on.  

Open reference architectures are what make that repeatable: Instead of every operator rederiving the design math, validated architectures with complete BOMs are published on the OCP Marketplace for anyone to build from. Everyone gets to network like a hyperscaler, and the industry scales without redesigning the network from scratch each time.

Ahead of AI Infra Summit, TechArena has been talking to companies building the AI data centers of tomorrow. We caught up with Jeniece Wnorowski, Director of Content Strategy & Industry Expert Programs at Solidigm, a company built on decades of memory and storage expertise now focused squarely on keeping GPUs fed as AI workloads scale. As models grow larger and context windows stretch longer, the gap between compute and the data it depends on is becoming one of the sharpest constraints in AI infrastructure.

We talked about why storage requirements have shifted from cheap, reliable capacity to keeping the GPU from ever waiting; where the data path most often breaks down, and what it really takes to size storage for the bursty, synchronized write demands of checkpointing at scale.

Q1: What are AI builders asking storage to do today that they weren't before the AI boom?

A: The shift is from "store it cheaply and reliably" to "keep the GPU from ever waiting." Four demands stand out.

First, raw density per rack unit: AI clusters are power- and space-constrained before they're capacity-constrained, so builders want maximum terabytes per watt and per slot. AI ambitions are grounded by the realities of physical footprint and power constraints, and high-capacity SSDs can dramatically reduce the storage footprint compared with HDD-based solutions. Second, sustained high-throughput reads at massive scale for training data ingestion and checkpoint restores, not just peak IOPS. Third, predictable latency at scale. AI pipelines need storage that can sustain throughput without long-tail latency spikes that leave expensive accelerators idle. And fourth, storage that sits physically and logically closer to the GPU itself.

That last point reflects a fundamental shift in AI infrastructure. Inference, RAG, vector databases, model repositories, and feature stores all require fast access to much larger active datasets than traditional enterprise applications. As models outgrow available HBM, SSDs increasingly act as a memory-expansion tier, storing model weights or KV-cache data that can be swapped in and out of GPU memory on demand. Solidigm partner Metrum AI demonstrated how fast SSDs can extend effective GPU memory capacity by serving as a storage tier for model components not actively resident in GPU RAM

Q2: Where in the data path does feeding the GPU break down most often today?

A: The most common failure points are at the two ends of the training pipeline: data ingestion and checkpoint write/restore.

During training, if storage cannot sustain the random-read throughput needed to keep shuffled batches flowing, GPUs sit idle waiting on data loaders, which is an expensive way to burn compute budget. During checkpointing, entire jobs often pause while model state is written to persistent storage. If those writes are slow, every GPU in the cluster idles simultaneously.

A third bottleneck is emerging between memory and compute: the KV cache and the memory wall. In inference, every token of context generates key/value state that consumes GPU memory. As context lengths, batch sizes, and model sizes grow, KV cache can quickly exhaust available HBM. Operators are then forced to limit context windows, reduce batching efficiency, recompute state, or introduce a lower storage tier. When data is offloaded, storage becomes part of the inference path itself. SSD-backed memory expansion can help hold model weights or KV-cache data that are swapped into GPU memory as needed, extending model capability beyond the limits of HBM alone.

This is one reason the industry is increasingly focused on architectures that move data more directly between SSDs and GPUs, minimizing CPU and software-stack overhead.  

Q3: What do teams most often get wrong about storage for checkpointing at scale?

A: The most common mistake is sizing storage around capacity and average throughput rather than the bursty, synchronized write behavior that checkpointing actually creates. Hundreds or thousands of GPUs may attempt to flush model state at nearly the same moment, creating a concentrated “write” storm that can overwhelm infrastructure designed around average utilization.

Organizations also underestimate the tradeoff between checkpoint frequency and productivity. Checkpoint too rarely, and failures can result in hours of retraining and lost compute. Checkpoint too often, and the storage layer itself becomes the bottleneck.

There's also frequently a mismatch between endurance assumptions and real-world behavior. Teams often purchase expensive high-endurance media for workloads that are primarily large, sequential writes rather than small random overwrites. Understanding the actual checkpoint profile is often more important than simply optimizing for maximum endurance specifications.  

Q4: As drive capacities push past 100 terabytes, what does that density solve for AI builders, and where does it stop helping?

A: What ultra-high-capacity drives solve is largely physical and economic: fewer drives, fewer rack slots, lower power consumption, less cooling overhead dedicated to storage, and simpler deployment when a single storage platform can span multiple capacity points.  

Where density stops helping is when capacity growth outpaces performance growth. Capacity does not scale linearly with IOPS or bandwidth per drive, and ultra-high-capacity SSDs are optimized primarily for density and efficiency rather than maximum performance. Beyond a certain point, adding capacity alone can simply concentrate a bottleneck.

The other consideration is operational resilience. A 122TB drive contains significantly more data than smaller-capacity devices, increasing the importance of data protection strategies, erasure coding, and rebuild planning. However, capacity and resilience should be evaluated together. Modern SSDs can restore data dramatically faster than HDD-based alternatives, meaning organizations must consider both failure impact and recovery time when evaluating the tradeoffs of higher-capacity media.  

Q5: As AI scales, what should operators understand about how storage shapes the cost and quality of the tokens they deliver?

A: The core idea is that GPU utilization and model quality are the outcomes storage ultimately influences.

GPU compute is the scarce and expensive resource in modern AI infrastructure, and storage's job is to ensure that resource remains productive. Every idle GPU second caused by a stalled data loader, slow checkpointing operation, storage-network congestion, or memory bottleneck directly increases training costs and degrades economics.

Storage decisions increasingly affect model quality as well. Faster, denser storage enables organizations to train on larger and more diverse datasets, checkpoint more frequently without sacrificing throughput, and scale model size without making I/O the limiting factor.

On the inference side, the emerging role of SSDs as a memory-expansion tier is especially important. SSD-backed storage can hold model weights or KV-cache data that overflow HBM, allowing operators to support longer context windows and higher user concurrency. That directly affects both the quality of the tokens a model can generate, because it can access more context, and the cost of those tokens, because each GPU can serve more requests efficiently.

Organizations that continue to view storage as a background utility rather than a first-class component of the AI compute stack often discover the consequences in their GPU utilization metrics, inference throughput, and overall, AI economics.

This summer, TechArena has been asking the companies that build AI infrastructure how their requirements are changing as deployments scale.

For our latest conversation in this series, we sat down with Ryan Brown, Director of Data Center Product Management at Phononic, a company that puts cooling directly on the component, right where heat is generated. The approach matters more now, as rack densities climb and the list of thermally sensitive parts stretches well beyond the GPU.

We talked about what targeted, solid-state cooling makes possible inside a dense AI rack; how pairing it with liquid cooling changes both performance and economics; and why coolers that read voltage, current, and temperature at each hotspot turn thermal management into an operational tool. Here's what we learned.

Q1: Phononic puts solid-state cooling right on the hotspot, cooling a specific component on demand rather than the whole room or loop. What does that kind of targeted cooling make possible in a dense AI rack?

A: Traditional cooling systems remove heat from the rack or server level. Phononic’s solid state approach addresses heat exactly where it is generated—at the hotspot. By actively and dynamically controlling temperature at the component level, operators can maintain tighter thermal margins on the devices that matter most, even as rack densities continue to climb.

That targeted cooling enables higher compute utilization, more predictable performance, and greater design flexibility. Rather than overcooling an entire rack to protect a handful of critical components, operators can deliver cooling precisely where it is needed, helping unlock higher-density AI infrastructure while improving overall system efficiency.

Q2: Liquid cooling is the default now, and you describe solid-state as working with it, targeting hotspots so operators can run warmer loops and over-provision less. How does that pairing change the performance and the economics?

A: We do not see solid-state cooling as a replacement for liquid cooling—we see it as a complementary technology. Liquid cooling excels at removing large amounts of heat from the system, while solid-state cooling delivers millisecond-scale, precision temperature control at the most thermally sensitive locations.

That combination allows operators to run warmer coolant temperatures, reduce thermal variability across critical components, and minimize the need for excessive cooling headroom. The result is a better balance between performance and efficiency: improved utilization of expensive AI hardware, lower cooling-system complexity, and potentially reduced energy and infrastructure costs.

Q3: Everyone watches the GPU, but you cool hotspots across HBM stacks, voltage regulators, and the lasers inside high-speed optics, where you already ship tens of millions of coolers. As density climbs, which of these is becoming the real limit on performance?

A: The answer is increasingly “all of the above.” GPUs remain the focal point because they consume so much power, but AI systems are becoming constrained by a growing number of thermally sensitive components.

High-bandwidth memory is critical because memory performance directly affects AI throughput. Voltage regulators influence system stability and power delivery. Meanwhile, optical interconnects are becoming increasingly important as clusters scale, and laser temperature stability directly impacts optical performance and reliability.

As AI infrastructure becomes more tightly integrated, performance is increasingly determined by the weakest thermal link in the system. That is why component-level thermal management across compute, power, memory, and optics is becoming essential.

Q4: Your cooling also reads voltage, current, and temperature at each hotspot and feeds that back through standard APIs. What does that thermal intelligence let an operator do that cooling alone never could?

A: Cooling is only part of the story. When thermal systems become intelligent sensing platforms, operators gain real-time visibility into the health and operating conditions of critical components.

That data can support predictive maintenance, optimize workload placement, identify emerging failure risks, and enable dynamic thermal control strategies. Instead of reacting after performance degrades, operators can make proactive decisions based on actual device conditions.

In AI infrastructure, where uptime and utilization are paramount, thermal intelligence turns cooling from a supporting utility into an operational tool that helps improve reliability, efficiency, and asset management.

Q5: The industry has seen racks climb from 20 kilowatts to 100 and now toward a megawatt. Where does cooling have to go next, and what are you building for it?

A: The next generation of AI infrastructure will require a layered approach to thermal management. As rack power continues to rise, broad system-level cooling alone will not be enough. Operators will need precision thermal control at the component level alongside advanced liquid-cooling architectures.

Phononic is focused on enabling that future with scalable solid state cooling solutions that address the most challenging hotspots in compute, optics, memory, and power electronics. We believe the industry is moving toward cooling systems that are not just more powerful, but also more intelligent, localized, and responsive.

As AI deployments scale, the winners will be the architectures that can maximize compute performance while minimizing energy consumption and infrastructure overhead—and precision thermal management will be a key part of that equation.

Throughout the summer, TechArena has been sitting down with the companies behind the AI buildout to hear how requirements are shifting up and down the stack.

For this installment, we caught up with Paul Quigley, Chief Strategic Relations Officer at Airsys, a company with more than 30 years in mission-critical cooling, built on precision air systems and now expanding into rack-level liquid cooling as AI densities climb. With data center operators running air and liquid side by side, power is becoming the constraint behind nearly every infrastructure decision.

We talked about why air cooling keeps carrying part of the thermal load even in the densest AI environments; what operators run into when they retrofit liquid cooling into facilities that were never designed for this much heat; and how a new metric, Power Compute Effectiveness, measures the share of a facility's provisioned power that is structurally available for compute. Here's what we learned.

Q1: Airsys has spent more than 30 years in mission-critical cooling and its lineup now spans both precision air systems and rack-level liquid cooling with LiquidRack. What are data center operators asking you for today that they weren’t before the AI boom?  

A: The conversation has changed considerably over the past few years. Before the AI boom, customers were primarily concerned with whether their cooling systems could reliably support existing infrastructure. Today, they must plan much further ahead. They are asking how cooling architecture can reduce the amount of provisioned electrical capacity assigned to supporting infrastructure and make more capacity structurally available for AI compute.

Power has become a primary constraint, so every infrastructure decision comes back to how much provisioned capacity can ultimately be allocated to compute. Customers are also looking for flexibility. They know server technology is evolving quickly, and they want cooling infrastructure that can adapt rather than become a bottleneck. Cooling is no longer simply supporting the infrastructure; it has become an integral part of the overall AI strategy.

Q2: Much of the liquid cooling conversation assumes a new build. When an operator raises density in a data center that already exists, what are the specific pain points they run into?

A: Most existing data centers were not designed for today’s AI workloads. As rack density increases, the facility must remove substantially more heat than it was originally designed to handle.

The challenge is that many liquid-cooling solutions assume the facility can be redesigned around them. That can require building-wide piping, large centralized cooling distribution units, and extensive infrastructure changes that add cost, complexity, and disruption. Many operators do not want to take that approach—and some facilities physically cannot accommodate it.

Operators need a practical way to introduce liquid cooling where it is needed, expand it over time, and avoid turning every retrofit into a major construction project. That is why we have focused so much attention on simplifying liquid-cooling retrofits.

Q3: As racks get denser, does air cooling stop making sense?

A: It is not that simple. Air cooling continues to play an important role, and many operators will manage hybrid air-and-liquid environments for years to come.

What AI has changed is the amount of heat each rack must handle. As thermal demands increase, physical limitations begin to determine the appropriate cooling approach. Liquid is more effective at transferring heat away from high-power components, which is why it is becoming the preferred option for the highest-density AI workloads.

Liquid cooling will carry an increasing share of the thermal load and become a centerpiece of many cooling strategies. Air cooling, however, will remain necessary for portions of the IT load and other heat-producing components. Both must therefore be considered as part of a comprehensive cooling strategy.

Q4: Airsys has put forward Power Compute Effectiveness as a new measure. What is it, and what led you to introduce it?

A: Power Compute Effectiveness (PCE) emerged from a fundamental question: within a data center’s provisioned electrical capacity, what proportion is structurally available for IT compute? As AI workloads drive higher power densities and available power becomes increasingly constrained, understanding this allocation is critical for capacity planning and infrastructure design.

PCE is the ratio of provisioned IT compute electrical capacity—after accounting for cooling and heat rejection, electrical conversion and distribution losses, and auxiliary facility loads—to total provisioned facility electrical capacity within a declared facility boundary and redundancy basis. PCE does not replace Power Usage Effectiveness (PUE): PUE measures operational energy efficiency using consumed energy, whereas PCE evaluates how provisioned electrical capacity is structurally allocated at the design and planning level. Together, the metrics provide complementary perspectives on operational efficiency and the capacity available to support compute.

Q5: As AI pushes rack density higher, what should operators understand about how cooling shapes the cost of the compute they deliver?

A: Every AI workload requires electrical power and produces heat that must be removed. The cooling architecture—and the provisioned electrical capacity required to support it—directly affects how much of the facility’s total power envelope remains structurally available for compute.

The first question should not always be, “How do I get more power?” It should also be, “Am I making the best use of the power already provisioned?” Cooling can have a significant impact on that answer. A cooling strategy that requires less provisioned electrical capacity for heat removal can make additional capacity available for AI compute, potentially increasing deployable compute within the existing power envelope without waiting years for new utility infrastructure.

TechArena has spent the summer connecting with companies building AI infrastructure to learn more about the challenges and evolving requirements they face.

For our latest conversation in this series, we had the opportunity to chat with Andy Pernsteiner, Field CTO at VAST Data, a company built on the idea that the data platform matters as much as the compute it feeds. That idea carries new weight now that agentic AI is turning retrieval into a real-time workload.

We talked about where bottlenecks hide along the training and inference pipeline, from WAN ingest and bursty checkpoint writes to the KV cache reads that come with every generated token; what agentic retrieval requires from the storage underneath it; and what changes when storage, database, and AI compute run as one platform instead of a chain of stitched-together systems. Here's what we learned.

Q1. VAST is built on the idea that the data platform, not just the GPU, sets the ceiling on AI performance. Why does data decide how much useful work a GPU actually does?

A: For model builders, GPUs are the most expensive resource in their pipelines. For nearly everyone else, GPUs are the scarcest resource they have, outside of skilled staff. That's why feeding them the right data matters so much.

“Right data” means a few things at once:

• Prepped and clean. Data that hasn't been properly formatted, deduplicated, and cleansed causes the GPU to waste cycles processing input it can't use, or producing output that has to be thrown away and redone. Either way, that's expensive compute spent twice.
• Governed and trustworthy. Data that isn't correctly tagged, secured, and governed to an organization's (or the law's) requirements can produce work that has to be discarded outright, regardless of whether the model output itself was correct. Same waste, different root cause.
• Available on demand. GPUs are only earning their keep when they're actively computing a model or an inference result. Every cycle spent waiting on I/O is a cycle that's gone for good, you can't bank idle GPU time and use it later.

Q2. Training and inference move data through a long pipeline, from ingest and preparation to checkpointing and retrieval. Where along that pipeline do the real bottlenecks show up?

A: There are the obvious bottlenecks, compute, network, and storage performance, and some less obvious ones.

Training

• ‍Ingest: Usually, a bulk load (data acquisition) or continuous scraping (web, social, sensor feeds, etc.). The bottleneck here is typically the network (usually WAN/internet) combined with latency (distance from source to destination) and how much concurrency/parallelism you can bring to bear. Most training data starts life outside the four walls of the data center, so it has to be brought in first. That “tax” doesn't need to be repaid on every subsequent training run, except for new datasets being added. ‍
• Preparation: Typically bottlenecked by compute and/or a lack of parallelism. Frameworks like Dask and Spark are good at parallelizing this work, but it takes real expertise to shard efficiently across large datasets and compute pools. CPU becomes the constraint when prep servers are undersized relative to the GPU fleet they're feeding. Increasingly, GPUs themselves are being used for parts of this step, as the major distributed frameworks add support for offloading preprocessing onto whatever compute is available. ‍
• Checkpointing: Most training runs now use asynchronous checkpointing, which smooths out peaks and reduces burstiness, but there still needs to be enough available bandwidth that a run doesn't stall waiting on a checkpoint write. It's important to separate async checkpoint writes from the synchronous writes that are also part of the training loop, since a flood of bursty checkpoint traffic can starve out sync flows that, by nature, block the job from proceeding.

Inference

• ‍Model load: Not always relevant in a busy environment where popular models are already resident, but time-to-first-token (TTFT) is directly affected by how long it takes to load a large model into GPU memory. Frontier models are large enough now that this requires real network and storage bandwidth over highly optimized, RDMA-capable fabrics. GPU high-bandwidth memory will always outrun the network feeding it, so the job is to remove as much friction as possible from that I/O path. ‍
• Prefill / prompt processing: Largely GPU/compute-bound, but this phase also writes KV cache aggressively as it's computed. In environments using external KV cache (i.e., on external storage rather than GPU memory alone), that requires high bandwidth to the storage layer and a storage subsystem that can keep up, not just with the writes, but with the reads needed to verify the current state of the cache. ‍
• Decode (token generation): Once prefill is done, decode produces output tokens one at a time, and every step reads back the KV cache accumulated so far. Unlike prefill, this phase is memory-bandwidth-bound, not compute-bound: the GPU spends more time waiting on data than doing new math. It gets harder as context windows grow, since KV cache size grows right along with them. It also matters in disaggregated serving architectures, where prefill and decode run on separate GPU pools and the KV cache computed during prefill has to move to the decode pool fast enough that decode isn't left waiting. Time-per-output-token ends up being a function of how quickly that cache can be read and moved, not just raw GPU FLOPs.

Q3. Retrieval, vector search, and RAG have become heavy, real-time workloads as AI moves toward agents. What does that ask of the data layer underneath?

A: Agentic workloads turn retrieval into something closer to a real-time transactional workload than a batch job, and that changes what's required of the data layer:

• KV cache. As context windows grow and agents chain multiple reasoning steps together, KV cache read/write volume grows with them. The data layer needs to serve that cache with low, consistent latency, both writing it as it's generated and reading it back to check state, without becoming the thing an agent is waiting on mid-reasoning.
• Vector retrieval. An agent making a decision often triggers a vector search inline, in the critical path, not as a background task. That means the data layer has to support low-latency similarity search at scale, across potentially billions of vectors.
• Associated datasets. Vector search rarely stands alone, the agent usually needs the structured or unstructured source data the vector points back to (documents, tables, metadata) to actually ground its answer. Ideally, the source data (such as a document or a text chunk) resides close to or on the same system as the embeddings, not only to reduce the complexity of the system, but also to maintain tight lineage between source and derivative, especially in environments where data can change or be deleted frequently.

Q4. VAST brings storage, database, and AI compute into one platform rather than separate systems. What does that unification make possible, and where are its limits?

A:

What it makes possible

• Eliminates the data movement that fragments most AI stacks today. For example, the ETL pipelines that copy data between a storage system, a vector database, a feature store, and a compute cluster just so each system can do its one job. Data is prepped, stored, and retrieved in place.
• Collapses the latency in the retrieval loop described above: vector search, KV cache, and associated datasets can be served from the same platform instead of stitched together across systems with their own network hops.
• Simplifies governance and security to a single policy plane. Access controls, encryption, and lineage are enforced once, at the platform level, instead of being re-implemented (and potentially re-broken) in every downstream system that touches a copy of the data.
• Lets compute run where the data already lives. Embedding generation, preprocessing, or lightweight inference can happen against data in place rather than requiring another export/import cycle.
• Reduces the operational burden: fewer hardware and software systems to purchase, provision, patch, monitor, scale, and reconcile with each other.

Where the limits are

Unification is a platform capability, not a substitute for organizational discipline. VAST provides the platform and policy frameworks, but domain experts and policy owners still need to be directly involved in the design of end-to-end systems to ensure efficient and consistent pipelines.

Organizations with an already-fragmented stack need a practical migration path, not a rip-and-replace. Unification pays off fastest on new workloads and gets adopted incrementally on existing ones.

It doesn't eliminate bottlenecks that live outside the platform's control, such as WAN ingest from external sources or network fabric choices upstream of the cluster.

Q5. In the end, how does the data platform shape the cost and quality of the tokens an operator can deliver?

A: Cost and quality both trace back to the same root cause: whether GPU cycles are spent on useful work.

Eliminating bottlenecks directly lowers cost per token, because idle or stalled GPU time doesn't produce any tokens at all, it's pure overhead on top of very expensive hardware. A platform that keeps GPUs fed converts a higher percentage of the compute you're paying for into actual output.

Governance and lineage shape quality and trust. Knowing what data trained or informed a given model, and being able to prove it, is what lets an operator stand behind the tokens it delivers, particularly in regulated industries or anywhere provenance matters. It also reduces the risk of hallucinated or non-compliant output making it into production.

Cataloging and labeling ensure the right data gets used in the first place, which avoids wasting compute on irrelevant, duplicate, or low-quality data, the single most common cause of both wasted GPU cycles and degraded model output.

Put simply: the data platform doesn't just support the AI infrastructure, it sets the ceiling on how efficiently that infrastructure can turn dollars into good tokens.

A single Arm Neoverse Compute Subsystem (CSS) deployment helped a partner ship a system-on-chip (SoC) in 13 months and shave roughly 80 engineering years off the build. That return is exactly why conversations at this year’s OCP EMEA Summit in Barcelona kept circling back to one question: is the era of monolithic silicon finally giving way to a true chiplet economy, and if so, what holds the new stack together?

I sat down with Dong Wei, Arm Fellow and Lead Standards Architect, and Colin Brix, VP Marketing at AMI, for an episode of The Control Plane podcast sponsored by AMI, where we looked at the rise of the chiplet economy.  

Challenges to Chiplet Adoption

Dong was careful not to declare the monolithic chip dead. The ability to design a single die is not going anywhere, he explained, but AI is straining its limits because CPU, IO, and memory increasingly want different process nodes. That mismatch, combined with cost and performance pressure, is pushing integrators toward chiplet designs. Hyperscalers including AWS, Microsoft, and Google are already there, stitching together verified IP and custom accelerators into bespoke silicon. Arm introduced the AGI CPU last month using the same approach.

Moving from a handful of vertically integrated SoC vendors to an open, multi-vendor chiplet ecosystem requires foundational standards work. Dong compared the road ahead to the long evolution of Peripheral Component Interconnect Express (PCIe), which took decades to deliver true interoperability. Chiplets will be tougher, he said, because they are integrated inside the package using costly manufacturing processes, with no easy plugfest equivalent.

Firmware as the Glue Layer

If standards define the rails, firmware drives the train. “I think I see firmware eventually will become a glue layer to tie these things together,” Dong said. He laid out the layers that need standardization beyond physical and electrical interfaces, including telemetry, discovery, configuration, and control. Arm’s contribution of the Foundational Chiplet System Architecture (FCSA) to OCP is one piece of that picture, and Dong pointed to UCIe and other industry efforts as complementary work.

Colin picked up the thread from the firmware partner’s perspective. AMI has spent 40 years being the quiet orchestrator of the ecosystem, but that is changing. With chiplet designs, security, power, and thermal behavior all have to be designed in from the start rather than bolted on later, which means AMI now listens all the way downstream to the customer of the customer and projects that voice back upstream. “It’s not just the silicon enablement partner of the past,” he noted.  

The shift to multi-vendor packages also creates a reliability problem that the monolithic era did not have to solve. A standardized communication path lets firmware redistribute workloads and isolate misbehaving regions while the system keeps running, he said.

Standards That Free Innovation

The conversation tackled the perennial worry that standardization smothers proprietary edge. The right place to draw the line is between interfaces and implementations. “The standards should be focused on the interfaces rather than the implementations.” This way, vendors can keep what is beneath the interfaces proprietary information, while the framework gives the system a common communication path. He compared this to the standardization of coffee pods: if every espresso capsule were a different shape, you would need a different machine for each one. However, once the capsules are standardized, the machines can compete on other parameters. Colin agreed, adding that in his view, the absence of standards actually slows innovation downstream because vertical applications never get the building blocks they need.

The TechArena Take

Although a lot of progress has been made in the development of chiplets, foundational work needs to happen for the broader market to follow. Both Colin and Dong agreed that open interface standards will decide whether chiplets become a true ecosystem or a playground for a few integrators. Firmware will be the layer that makes reliability and sustainability achievable at the package level. As silicon gets more modular, open standards plus firmware are what keep the whole system working smoothly, and Arm and AMI are positioning themselves as the two ends of that pipeline.

Every new AI data center hits the same wall before it runs out of ambition: power. Operators can order more accelerators, but the grid connection, the cooling budget and the rack density are fixed. That ceiling is quietly rewriting what gets built, and it has turned efficiency from a line on the TCO sheet into the thing that decides how much useful AI work a site can deliver.  

As part of our summer series with the companies building the AI stack ahead of the AI Infra Summit this September, I sat down with Eddie Ramirez, VP of marketing for Arm's Infrastructure Business. Arm has spent years making the case for performance per watt in the data center, and that case lands differently now that power is the hard limit rather than a background concern.  

We talked about why the CPU is becoming the control plane for an entire rack, what open chiplet standards like UCIe and CHI have to hold together as AI silicon turns into an assembly of parts, and how operators can keep pulling useful work out of a data center over a lifespan that will outlast several generations of models. Here's what I learned.

Q1: Arm has built its data center story around performance per watt. As AI data centers run into hard power limits, why does efficiency decide what an operator can actually build?

A: AI infrastructure is increasingly constrained by fixed power, cooling and rack density rather than demand for compute. As AI deployments scale, operators are no longer optimizing individual servers. They're optimizing entire racks, and ultimately entire data centers. Efficiency therefore isn't simply about lowering TCO, but it is about how much useful AI work can be delivered within a fixed power envelope. As AI workloads become more continuous and agentic, maximizing performance at the rack level, not just within a single server, will increasingly define competitive AI infrastructure.

Q2: As AI infrastructure evolves from running individual models to supporting continuous, agentic workloads, the CPU is taking on new responsibilities – from AI execution to orchestrating entire racks of accelerators. How do you see the CPU's role evolving, and where does it make the most sense for AI work to run?

A: AI systems are evolving from executing individual models to coordinating fleets of specialized models that are continuously reasoning, retrieving information, calling tools, managing memory and moving data across large clusters of compute pools. Those continuous system-level tasks naturally belong on the CPU. As AI infrastructure scales, the CPU becomes the control plane for the entire rack, coordinating data movement, scheduling work, feeding accelerators efficiently, and ensuring entire system resources are fully utilized. The result isn't a competition between CPUs and GPUs, but a better-balanced system where each processor is optimized for the work it does best – and work is assigned to the best suited processor.

Q3: Arm’s chiplet strategy leans on open standards like UCIe and CHI to combine pieces from different vendors in a single package. As AI silicon turns into an assembly of chiplets, what needs to hold together for that integration to deliver?

A: Open standards like UCIe and AMBA CHI are essential because they allow compute, AI accelerators, memory, and networking to evolve independently while still working together as a cohesive system. That gives silicon providers the flexibility to innovate faster without sacrificing software portability or ecosystem compatibility. That's also the thinking behind Arm's Foundation Chiplet System Architecture (FCSA), which we contributed to the Open Compute Project to help establish an open, interoperable foundation for next generation chiplet-based AI infrastructure.

Q4: With Compute Subsystems, Arm hands partners pre-validated cores so they can reach production faster and put their own effort into the accelerator that sets them apart. What does that change about who can build competitive AI silicon, and how they build it?

A: Building AI silicon at the leading edge is extraordinarily complex – beyond the digital design, advanced node processes and packaging layer on additional challenges. Everything we can do to simplify our partner’s journey, accelerate the path to silicon and de-risk production holds a great deal of value. With Arm Compute Subsystems, partners receive a production-ready compute foundation that's already validated for performance, software compatibility, and system integration. That allows engineering teams to focus their investment on the capabilities that differentiate them, whether that's AI acceleration, networking, memory architecture, or custom system innovation. The result is faster time to market, lower development risk, and broader innovation across the AI ecosystem.

Q5: As operators look to maximize useful AI work over the lifetime of a data center, what characteristics will matter most in the compute layer?

A: AI models will evolve far faster than the infrastructure that supports them. Data centers are long-term investments that are both capital and labor intensive, so the compute layer must continue delivering value as workloads, models, and software stacks change over time. That means balancing performance, energy efficiency, memory bandwidth, software portability, and ecosystem support at rack scale, not simply optimizing for a single benchmark or today's leading model. The most successful AI platforms will be those that consistently deliver more useful AI work within fixed power and space constraints as AI continues to evolve.

The plot of the 1969 Western calls for James Garner to produce exasperated and amused facial expressions as his character attempts to emigrate to Australia but is frustrated by a lack of funds. The character he portrays, Jason McCollough, is stuck in a nineteenth century frontier town put on the map due to an accidental discovery of gold. Because the cost of living in town inflates hourly and the pay is good, McCollough, a brilliant gunslinger, reluctantly accepts the job as the local sheriff.  

This also happens to be a solid description of most IT and sysadmins I’ve worked with over the years. Quick on the draw when it comes to technical challenges, sysadmins are a jack of all trades who can resolve issues with both ingenuity and a refreshing lack of perfectionism when stabilizing a situation. Also: many sysadmins are not in the job they envisioned when they graduated, must work with the equivalent of the town ‘character’, and have gleeful management who are thrilled someone else deals with the local mess of bad horses, bad men and bad decisions.

One challenge for a sysadmin is that like the sheriff, when things are going smoothly, neither of them is front of mind for day-to-day operations. You only notice them when unwelcome changes occur, such as a core software platform provider gets bought out by an aggressive organization operating like a PE who then squeezes the clientele for more juice. As a sysadmin you have no good choices: costs might suddenly triple, but you built your disaster recovery plan on this technology. You’re “encouraged” to move to new products if you know what's good for you – can you really afford to walk away from the new owner who decided to weaponize their newly held asset against you, their customer?  

Even the low-grade annoyances can flare up out of the blue. Accepting a service providers’ half-hearted developer experiences, inscrutable services catalogs, or byzantine documentation because they offer stability in reliability, identity and service boundaries seems like reasonable tradeoff. Until the service provider starts surprising you with issues in authorization checks and access controls across your VMs, then responds to legitimate customer sysadmin concerns with a statement: “No one has really found that exploit in a real world situation.”

Notice that even mature, decades-old technologies from suppliers who had been stable, if not accommodating, business partners can be a new wrinkle for sysadmins. A new singularity in the form of AI springs new leaks in what had been a relatively steady-state environment. Today, sysadmins have a trifecta of fleet-wide technology changes that are necessary to get the most from AI, bad actors who use AI to launch attacks at a scale human teams can’t match, and a core technology that no one has yet mastered the art of setting or meeting budget projections for. A recent KPMG report states that nearly 1/3 of corporate leaders are finding it difficult to understand the operating costs, much less control them when it comes to AI implemented at enterprise scale. It’s not just financial operations either: there are new potential legal exposures related to copyright ownership, new supplier contract terms and/or unintended regulatory missteps.

For the AI-plagued sysadmin, here are a few shortlist areas for your triage efforts.  

First, protect the areas where the entire company could be put at risk:

• For product development, consider a private AI implementation or open weights model, small language models or industry-specific models you fine tune. This can help ensure your use of AI in product development isn’t a two-way channel that trains your supplier’s AI on how you develop your products as the basis for future competitors. This is especially important if your product happens to be software and is currently protected through copyright.  
• Ensure that any new tools or technology suppliers can meet your insurability requirements through contract negotiations or vendor selection processes
• Employ AI for threat detection including fraud detection while being very deliberate and prescriptive about where AI is allowed to take action to remediate.

Next, use “multi-cloud” principles to ensure company operations can run independent of specific models to avoid vendor API lock-in and build in finOps modeling to anticipate budget “tops and tails."

No CFO has ever been excited to hear sysadmins tell them “I have no idea what budget I need, or whether I can hit the budget within 5% of my estimates”. Like cloud services as they matured, AI tooling prices have experienced wild swings in the last three years, and model retirement has already caused headaches for what were once stable enterprise workflows.  

Finally, look for real-time operational governance capabilities in your AI solutions.

The regulatory landscape for AI is broad, varies greatly between states and nations, and can carry significant penalties for missteps. In the old west, the town charter and the town council were necessary but not sufficient to tackle the opportunities and challenges that came from a sudden influx of gold prospectors. Similarly, AI governance committees are necessary, but not sufficient in a playing field that changes at the speed of compute.

In the end, the role of the sysadmin hasn’t changed nearly as much as the environment around them. Like the reluctant sheriff, they step in not because the job is easy or glamorous, but because someone has to keep order when complexity and chaos collide. As AI reshapes the frontier yet again, the tools may evolve and the stakes may rise, but the mission remains the same: protect the town, steady the systems, and make sure everyone gets through the day without noticing just how close things came to going off the rails. So, this Sysadmin Day, remember—the quiet stability you rely on is no accident; it’s the work of someone who’s always on watch, even when you don’t see them.

Leading up to the AI Infra Summit in September, TechArena has spent the summer talking with the companies building AI infrastructure, from hyperscalers to storage, memory and networking, through to the power, cooling, and silicon that hold these systems together.

In our latest Five Fast Facts Q&A, we chatted with Tony Pialis, EVP and GM of Data Center at Qualcomm. He explains what changes when a company that grew up counting every milliwatt in mobile brings that discipline to inference. We discussed the memory wall, why data movement now costs more than compute, and what it takes to sell a rack instead of a chip. Here's what we learned.

Q1: Qualcomm comes to the data center from mobile, where every milliwatt counts. What does designing for energy efficiency first let you do in an inference rack that a chip built for the data center might miss?

A: Qualcomm’s perspective is different because we grew up solving compute problems under severe power constraints. In mobile, every milliwatt matters because battery life, thermals, and form factor are critical. That mindset translates directly to our AI inference solutions, since the real challenge in modern data centers is no longer peak compute, but delivering the most AI work within a constrained power and investment envelope.  

Designing for efficiency first enables us to optimize the entire rack around cost-per-token and power efficiency rather than chasing benchmark peaks. Qualcomm Dragonfly combines specialized CPUs, AI accelerators, memory innovation through High Bandwidth Compute (HBC), and advanced connectivity in a disaggregated architecture designed specifically for inference. Our multi-generation roadmap is focused on maximizing performance per watt, improving token economics, and lowering total cost of ownership at scale.

Q2: What do you see as the real limit on AI inference today, and what did designing around it lead you to build differently?

A: The biggest constraint in AI inference isn’t raw compute anymore—it’s moving data efficiently. Model sizes are growing far faster than memory bandwidth and capacity, creating what the industry increasingly describes as the memory wall. Qualcomm believes that simply adding more compute does not solve the problem if memory becomes the bottleneck.

That realization led us to develop HBC, a near-memory computing architecture that brings compute and memory much closer together. Instead of shuttling massive amounts of data back and forth, HBC performs more processing near memory, reducing movement, lowering power consumption, improving effective memory bandwidth, and lowering overall system costs. HBC is poised to deliver significantly higher bandwidth-per-watt and capacity-per-watt compared with traditional approaches. The outcome is infrastructure designed specifically for modern AI inference workloads where efficiency, scalability, and predictable economics matter as much as performance.

Q3:. Qualcomm built its name in silicon, and Dragonfly now sells the whole rack as one liquid-cooled product. What did moving from selling chips to delivering a full rack force you to learn, and what still trips operators up in that shift?

A: I would characterize Dragonfly as a portfolio of rack-scale platforms rather than a single liquid-cooled product. Depending on the deployment, Dragonfly systems can support air or direct-liquid cooling. What is important is that customers increasingly evaluate AI infrastructure at the system level—not as isolated chips.

AI infrastructure is increasingly a systems problem rather than a chip problem. Customers no longer evaluate silicon in isolation—they care about rack-level performance, power consumption, networking, software orchestration, cooling, and overall cost of ownership. Qualcomm Dragonfly reflects that reality by bringing CPUs, AI accelerators, memory architecture, connectivity, software, and custom silicon together into a unified data center platform.

Optimizing one component is not enough. The infrastructure must work as an integrated system. One persistent challenge for operators is balancing compute, memory, networking, and power at scale. Bottlenecks often emerge not from lack of processing power but from inefficient data movement, infrastructure complexity, or underutilized resources. Our approach is to simplify this by delivering a rack-scale platform designed around inference efficiency, open software, disaggregated compute that help customers scale economically as agentic AI dramatically increases token demand.

Q4: Qualcomm has pulled its CPU, accelerators, connectivity DSPs, and management software together partly through acquisition and partnership. What has to be true for those pieces to run as one machine rather than a bundle, and where are the seams still real?

A: To operate as a single platform, every layer must be designed around common system objectives: performance, efficiency, openness, and scale. Qualcomm’s strategy combines CPUs, AI accelerators, connectivity technologies, custom silicon, orchestration software, and developer tools into a unified architecture optimized for AI inference. The software layer is especially important because it coordinates workloads across heterogeneous compute resources and abstracts complexity from developers and operators.

At the same time, we believe openness matters. AI infrastructure is becoming increasingly heterogeneous, with CPUs, GPUs, XPUs, and specialized accelerators coexisting. The real challenge is enabling those components to work together efficiently. The seams that remain are often industry-wide—not unique to Qualcomm—including interoperability across different hardware ecosystems, software stacks, and deployment environments. Our goal is to minimize those seams through open standards and unified software rather than proprietary lock-in.

Q5: Inference increasingly gets bought on cost per token, not peak specs. From where Qualcomm sits, what most decides whether an operator can hold a predictable cost per token, and how much of that is set at the memory and rack level?

A: We believe the industry is undergoing a fundamental shift from measuring AI infrastructure by peak FLOPS to measuring it by tokens per watt and ultimately cost per token. As agentic AI drives massive growth in inference requests, economics become the defining factor. Operators need infrastructure that can deliver consistent throughput while managing power, cooling, and hardware utilization efficiently.

Memory architecture plays a central role because data movement increasingly consumes more energy than computation itself. That’s why Qualcomm invested in High Bandwidth Compute, which is designed to reduce energy consumed moving data, improve effective bandwidth, and lower total cost of ownership. But cost per token is ultimately determined at the rack level as well. Compute, memory, software orchestration, networking, and cooling must be optimized together. Qualcomm Dragonfly was built around that systems view, using a disaggregated rack-scale architecture to maximize utilization and efficiency. In our view, the winners in AI inference will be those who can consistently deliver the best performance-per-watt, performance-per-dollar, and long-term economics—not simply the highest headline specifications.

The world tends to measure artificial intelligence (AI) progress in two units: the number of graphics processing units (GPUs) deployed and the megawatts they consume. Yet the people who actually build these systems know that the GPU count is only one part of a much larger engineering effort. Power delivery, cooling, networking, and storage all have to come together before a single accelerator does useful work, and the discipline of fitting those pieces together is becoming one of the most consequential roles in the data center.

On a recent TechArena Data Insights episode, Solidigm’s Jeneice Wnorowski and I explored that hidden layer of the AI buildout with Hitesh Kumar, a GPU cluster architect at Nebius, an AI cloud company that has been drawing attention across the neocloud landscape. Our conversation moved from custom hardware and proprietary software to power heavy AI workloads.  

The System Beyond the GPU

Hitesh described his role as starting once a site has been selected and power secured, and ending when responsibility passes to the logistics and deployment teams. That gap, he explained, is where his team plans clusters, maps them onto floor plans, and decides what connects to what. The work is deliberately broad, and much of it has little to do with the accelerators themselves.

“My job as a GPU cluster architect focuses on a lot of things that aren’t GPUs,” he said. Power and cooling are a major part of the discussion, and so are networking, storage, and the management plane that orchestrates everything. In fact, it’s only after that supporting infrastructure is planned that the headline GPU count comes back into the picture.

Storage and Interconnects Move to the Foreground

Storage has always mattered for training AI models, as data must reach the GPUs fast enough to avoid stalls. Hitesh noted that inference, or running those models in production, is now creating fresh demand beyond that baseline. With a large language model (LLM) chat bot for example, the full state of a long conversation increasingly needs to move off the GPU, and sometimes off the server entirely, with storage acting as an intermediate tier between what a GPU holds close and what it will need soon.

Interconnects are evolving in a similar trajectory. Hitesh traced a path from today’s pluggable optics that enable a connection to one to two cables toward denser designs with tens of ports per unit, and eventually toward co-packaged optics that place optical engines directly in the server. Each step raises new challenges in cooling, cabling, and supply chain readiness that operators must manage alongside the mature technology they rely on today.

The Hidden Costs of Scale

Adding GPUs sounds simple, but Hitesh pointed to two realities that catch teams off guard. The first is component failure. Citing Meta’s published study of a 16,000 GPU cluster, he noted that Meta’s team experienced a failure every three hours on average. At that rate, software must become genuinely fault tolerant, and operations, spares, and logistics all have to scale to match.

The second unanticipated challenge is power behavior. Hitesh highlighted the scale of the power that clusters now draw, and he described how a checkpoint pause can drop a rack tens of kilowatts in moments. Swings that large can affect the grid, pushing operators to consider capacitors, batteries, or software mitigations that rarely make the headlines.

Building Smart and Looking Ahead

For organizations early in their AI journey, Hitesh offered valuable guidance. Going from zero to one, he said, “you really don’t want to be thinking about buying your own infrastructure,” given the high startup costs and overhead. Cloud resources make sense first, followed by colocation, and eventually a dedicated site once demand justifies it.

Looking forward, he expects steady efficiency gains, with “every part of your stack” delivering more performance per dollar. The change he finds most interesting is in network topology. He anticipates clusters organized as “lots of small islands of very tightly connected GPUs,” linked by sparser scale-out and scale-across fabrics. He sees the same fractal pattern emerging across NVIDIA rack-scale designs, Google’s tensor processing units (TPUs), and Huawei’s accelerators alike.

The TechArena Take

Hitesh’s perspective provides a useful counterpoint to conversations that reduce AI infrastructure to a single metric. The teams that succeed will treat power, cooling, storage, and networking as first-class design decisions rather than afterthoughts, and will scale their operational maturity in step with their hardware. As models grow and inference proliferates, the competitive advantage will belong to operators who understand the entire system. For decision makers planning their own buildouts, that full-system discipline is no longer optional.

If you want to learn more about Nebius visit https://nebius.com/

Leading up to AI Infra Summit this September, we're continuing our conversations with players across the data center stack to learn what AI infrastructure demands and what's being built to meet it. Those talks run from hyperscalers through storage, memory, and networking, out to power, cooling, and rack-scale architecture.

For this installment, I was delighted to catch up with Steve Thorne, chief commercial officer of CelLink. The company spent more than a decade building flat, flexible power circuits for electric vehicles, and it has now brought that form factor to the data center. Its PowerPlane swaps out thick bundles of discrete wires for a single laminated circuit thin enough to route thousands of amps in less space than a cable harness used to fill. Motherboards dock straight to it. And because that same flat layer can align with liquid-cooling manifolds and pull heat off the backside of the board, where vertical power regulators run hottest, Thorne makes the case that power and cooling no longer have to fight for the same cramped space. Here's what I learned.

Q1: CelLink came away from OCP 2025 asserting that the unit of design has moved from the server box to the rack and beyond, with whole racks and pods sold as finished products. What does designing at rack scale change about how power has to be delivered, and what still trips operators up in that shift?

A: Designing at rack scale forces power delivery to be considered as a first-class constraint alongside compute and cooling, rather than an afterthought bolted on at the end. Flat, flexible power delivery infrastructure must now be co-designed across the full rack and pod, rather than being treated as independent subsystems. The old approach of designing subsystems in isolation has caused integration oversights and schedule delays. By coupling flat power delivery with cooling infrastructure from the start, operators can improve current-carrying capacity, reduce energy loss, and avoid the last-minute surprises that still trip up many rack-scale deployments.

Q2: Your PowerPlane delivers thousands of amps through a flat circuit under a millimeter thick, and motherboards dock straight to it instead of being wired one cable at a time. What does taking power delivery flat and into the rack free up, and why does that matter for an AI server?

A: The flat geometry of the CelLink flex harness eliminates the volumetric overhead of round wire bundles, freeing space inside the compute tray for larger compute modules and liquid cooling infrastructure. Silicon vendors also benefit: by routing power interconnects to the underside of the chip, more I/O and fiber optic connections can be placed around the GPU or AI accelerator periphery. Moving the Power Distribution Network (PDN) out of the PCB also enables more thermally-efficient delivery—less energy lost to heat means more power available for compute.

Beyond space, CelLink flex harnesses enable automated system assembly. Their flat, thin profile is far more compatible with robotic pick-and-place equipment than conventional cable bundles, reducing manual labor and improving assembly consistency at scale.

Q3: CelLink contends that efficiency now must span the whole power path, from the grid down to the last centimeters inside the server. Where is the most power and space lost today, and how do higher-voltage delivery and vertical power to the back of the board change that picture?

A: Significant power is lost inside the GPU tray along the conversion chain from 800V down to 50V, 12V or 6V, and finally sub-1V at the silicon. At each step down in voltage, current rises sharply, and I²R losses compound accordingly. CelLink flex harnesses address this by carrying high voltage deeper into the GPU tray, keeping current lower across a longer path and reducing resistive losses before the final point-of-load conversion.

Q4: You describe power and cooling as things that should not fight for space. How do you co-design a flat power layer to work with liquid cooling in the same rack, and what does that integration unlock that assembling them separately cannot?

A: In conventional racks, power arrives via large-gauge, heavy, inflexible wire bundles, while liquid cooling relies on bulky pipes and hoses. Both compete for the same constrained volumetric space inside the tray. CelLink flex harnesses resolve this conflict by flattening power delivery to under 1mm in z-height, spreading it across tight spaces and allowing cooling infrastructure to occupy spaces where it actually needs to be.

The flat geometry also creates a natural thermal advantage: large, flat surfaces are far more effective for heat extraction than liquid cooling plates shaped around round wires or thick copper busbars. Co-designing the power and cooling layers together unlocks efficiency gains that assembling them separately simply cannot achieve.

Q5: CelLink came into the data center from high-volume EV manufacturing, where flexible circuits replaced bundles of hand-built wiring. What does that heritage teach about taking manual assembly out of rack build-outs, and where does rack-integrated power and cooling have to go next as density climbs?

A: CelLink's high-volume EV manufacturing heritage directly informs its data center approach. Flat, lightweight flex harnesses are well-suited to robotic pick-and-place assembly, and aluminum conductor designs proven in EVs and drones for weight and cost reduction translate naturally to rack environments where efficiency and thermal performance are equally critical.

The same principles that simplified EV wiring apply to rack build-outs, where loose cables and manual terminations introduce variability and assembly errors. CelLink flex harnesses can be placed directly adjacent to GPU components without cable management overhead. As rack density continues to climb, the next step is deeper integration, where the power delivery harness is co-designed and co-assembled with liquid cooling cold plates as a single unified subsystem.

Ahead of the AI Infra Summit this September, TechArena is spending the summer with companies up and down the AI stack to learn the latest on AI infrastructure requirements. Our conversations run from hyperscalers to storage, memory and networking, out to power, cooling, and the connectivity now holding these systems together.

I had the pleasure of chatting with Vishal Chandrasekar, head of product management at Ayar Labs, a co-packaged optics pioneer, to talk about what happens when AI scale-up outgrows the single rack. We covered why copper runs out of room, what an optical fabric has to get right, and how moving light into the package changes the economics of an AI data center. Here's what I learned.

Q1. AI scale-up is pushing systems beyond the boundaries of a single rack. What are chip and system designers asking for today that they did not need before the rapid growth of AI?

A: The fundamental requirement has changed. Chip and system designers are no longer optimizing only for the performance of an individual GPU, ASIC or switch. They are trying to connect thousands of accelerators so they can operate as a single unified system, with the bandwidth and latency needed to support increasingly large AI models and inference workloads.

That creates new demands around bandwidth density, power efficiency, reach and architectural flexibility. System designers need connectivity that can extend across multiple racks without the power penalties and signal-integrity challenges associated with driving high-speed electrical signals over longer distances. They also need a solution that fits into the standard fab, packaging flows and system-in-package architectures they already use.

This is where co-packaged optics becomes essential. Ayar Labs’ TeraPHY optical engine brings optical connectivity directly into the compute or switch package, while the SuperNova external light source provides the reliability and serviceability required for data center deployment. Together, they give designers a practical path to scale beyond the rack without rebuilding their design and manufacturing flows around a proprietary architecture.

Q2. Copper has supported generations of computing infrastructure. Where does it begin to constrain AI scale-up, and why can’t the industry simply continue improving electrical connectivity?

A: Copper remains effective over short distances, but its limitations become much more pronounced as bandwidth and scale-up domain sizes increase. Moving extremely high-bandwidth electrical signals requires more power as the distance grows, while signal integrity becomes increasingly difficult to maintain. That effectively traps high-performance AI scale-up architectures within individual racks.

This creates a difficult tradeoff. System designers can keep accelerators physically close together, concentrating enormous amounts of power and heat in increasingly dense racks, or distribute compute across multiple racks and accept a connectivity penalty. Neither is sustainable as AI systems grow.

CPO removes that tradeoff by using light to extend high-bandwidth, low-latency connectivity across tens of meters. Ayar Labs’ solution delivers up to 10 times higher bandwidth, 10 times lower latency and three to five times greater power efficiency than conventional copper and pluggable alternatives. That allows compute resources to be distributed across multiple racks while continuing to function as one scale-up domain.

Q3. Ayar Labs talks about enabling thousands of GPUs across racks to operate as a single unified system. What has to be true for an optical fabric to deliver that experience?

A: The optical fabric cannot become a new bottleneck. It has to deliver extremely high bandwidth and consistently low latency across the entire scale-up domain so accelerators can exchange data as though they were part of one tightly integrated system. It also has to consume very low power so as to not alter the economics of AI deployments which are often power constrained.

It also has to provide enough bandwidth density within the package. The TeraPHY optical engine uses microring modulator technology to deliver two terabits per second (Tbps) of shoreline bandwidth density per millimeter. That makes it possible to bring significant connectivity directly into an XPU or switch package without consuming the power and package area associated with traditional electrical interfaces.

Just as importantly, the architecture has to remain flexible. AI scale-up infrastructure is being deployed using UALink, PCIe, NVLink, ESUN and other standards and specifications. Ayar Labs has designed a protocol-agnostic architecture that can support evolving requirements without locking customers into one protocol, light-source provider or vendor-specific infrastructure or manufacturing flow. That combination of performance and compatibility is what turns optics from a component-level improvement into the foundation for a scalable AI fabric.

Q4. CPO has been discussed for years, but demonstrations are very different from high-volume deployment. What separates a promising optical technology from a manufacturing-ready solution?

A: The real test is whether the technology can be manufactured, integrated and serviced using the processes the semiconductor and data center industries already rely on. A successful demonstration matters, but customers also need confidence in reliability, supply, packaging, fiber attachment, thermal performance and production yield.

Ayar Labs has developed and shipped multiple generations of the TeraPHY optical engine, with thousands of units in customers’ hands. TeraPHY is built on proven silicon photonics and standard CMOS manufacturing, using a chiplet architecture and UCIe-based electrical interface designed to fit within existing ASIC and system-in-package designs.

Manufacturing readiness also extends beyond the optical engine itself. Ayar Labs has established partnerships across foundries, advanced packaging providers, OSATs, ASIC design services and ODMs. The company has developed reference designs and manufacturing flows at the package, board and rack levels, that support high-volume assembly and field serviceability.

CPO reaches volume when it becomes a natural extension of existing semiconductor design and manufacturing processes rather than a specialized architecture customers have to build around.

Q5. As scale-up domains grow to thousands of GPUs, how does moving from copper to CPO change the economics of the compute an AI data center can deliver?

A: The economics of AI infrastructure are increasingly determined by how efficiently a system can keep expensive compute resources working productively. It is not enough to add more GPUs if the network cannot feed them data quickly enough or if the power and cooling required to connect them consume an unsustainable share of the data center’s capacity.

Copper’s limited reach forces operators to concentrate compute within very dense racks or sacrifice bandwidth as they distribute it. CPO allows compute to be spread across multiple 100-kilowatt racks while preserving the high-bandwidth, low-latency connections needed for AI scale-up. That reduces thermal concentration and gives data center architects more flexibility in how they deploy power, cooling and compute capacity.

At the same time, Ayar Labs’ scale-up CPO solution provides five to 10 times greater bandwidth and three to five times greater power efficiency than conventional alternatives. As token volumes grow, those improvements can translate into more useful compute from the same power envelope and at a lower cost per token.

CPO is therefore not simply a faster way to move data. It changes the unit economics of AI infrastructure by enabling more accelerators to work together efficiently.

Leading up to the AI Infra Summit in September, we're continuing our conversations with AI infrastructure companies across the stack, from hyperscalers to storage, memory and network, to power and cooling and beyond.

I had the pleasure of sitting down with Simon Jesenko, chief executive officer/ chief financial officer of Iceotope, a precision liquid cooling player. Here's what I learned:

Q1: Iceotope’s precision liquid cooling seals the server and pulls heat off every component, not just the processor, with a dielectric coolant. What does cooling the whole board let an operator do that a cold plate on the chip alone can’t?

A: Cooling a server with direct-to-chip cooling may still require air cooling for the components that are not cooled by the cold plates. Cooling the whole server with dielectric fluid allows the operator to cool all heat generating components within the server: GPUs, CPUs, memory, networking, and PSUs.

Sealing the server into a protective enclosure and eliminating air cooling removes fan noise and allows the server to be positioned anywhere within the rack and the room. Cooling with dielectric fluid removes any fear that the operator may have about the proximity of water / PG25 inside the server. Servicing does not require the connection, disconnection, or modification of any plumbing containing water / PG25.

Find more information.

Q2: Precision liquid cooling isn’t only for the hottest GPUs. You cool storage, networking, and mainstream servers too. What kind of workloads and environments does that open precision liquid cooling up to?

A: This allows high-performance computing to move out of an air-cooled server room or data center, and to where it is actually needed. Customers can bring their compute clusters on-premises and deploy in harsh environments in a ruggedized chassis. Since Iceotope seals the server in a protective enclosure without fans, hardware is not exposed to external contaminants, so it can be deployed in harsh environments. Dielectric cooling fluid operates at temperatures up to 35° C, so it can effectively cool systems in extreme / outdoor environments

Q3: As rack densities pass what air and single-method approaches can carry, how far does chassis-level liquid cooling scale, and what changes along the way?

A: Hotter components, or more concentrated hot spots in the server can be cooled with fluid, depending on its temperature and flow rate. Hotter components require colder coolant and higher flowrates. The effectiveness of the coolant at (re)moving the heat is dependent on its specific heat capacity. Water has a higher heat capacity than dielectric fluid but it cannot come into direct contact with electronic components.

At scale, the factor against which all liquid cooling will be measured is the temperature of the coolant into the rack’s manifold (whether this is water/PG25 or dielectric. Currently, the target for this temperature is 45ºC. Any step (heat exchange) between two fluids or fluid to air is an inefficiency in the system (as some heat is lost in the process). As a result, the most efficient system is one which has fewer (or no) heat exchangers.

Q4: Your system captures almost all of the heat into liquid, so it can be reused rather than dumped. What does capturing heat that cleanly open up for an operator?

A: Captured heat can be reused in other deployments, for example a precision liquid cooled data center in a hotel basement could use the excess heat to heat the hotel pool. The heat capture and reuse ability turns datacenters into a community asset rather than a drain on resources.

Q5: What is the biggest thing still standing between precision liquid cooling and mainstream adoption, and how are you closing that gap?

A: Market perception is that precision liquid cooling is the same as tank immersion cooling, and requires too much infrastructure to deploy. In smaller form factors, the rising rack density in datacenters doesn’t have the same urgency for edge deployments. More information.

ASAPP SVP Priya Sawant joins the Data Insights podcast to discuss platform engineering, AI agent observability, and how engineering teams maintain trust while building with probabilistic technology.

‍

The AI Infra Summit is just around the corner, and TechArena is proud to return as a media sponsor. This summer, we've been trading notes with the companies building the AI stack, from silicon to systems, to hear where AI infrastructure requirements are really heading.

I had the pleasure of sitting down with Lynn Comp, Head of Global Sales and GTM for Intel's AI Center of Excellence. We covered what enterprises actually need from their infrastructure as they move from experimenting to measuring, why the real case for AI gets made on total cost and value per useful output rather than peak performance, and who answers for it when an AI system gets something wrong. Here's what I learned.

Q1: You’ve argued the hard problem in AI isn’t the model, it’s getting the same answer twice: determinism and time-based predictability. What does infrastructure have to deliver for an enterprise to trust agentic AI in production, and where does it fall short today?

A: Infrastructure needs to have very tight design constraints for a given AI based agent, avoiding the ability of an agent to exercise functionality beyond its intended purpose. Many of the exploits that have been uncovered come from an agent supporting prompts that go well beyond the intended purpose of the agent’s function. Simple, straightforward functional design and thinking of agent design like building an appliance helps keep the architecture clean, which then has the byproduct of being more deterministic and predictable.  

Q2: You spend your days with enterprises operationalizing AI. What do they most need from their infrastructure to make it pay off, and what has shifted this past year as they move from experimenting to measuring?

A: The baseline need is to have a data architecture that can support AI operations – both using business data as a baseline input as well as during AI-based workflows. The AI is generic until it applies your business information and context, but it is difficult to see ROI from AI that spends the majority of tokens on the data input processes rather than getting insights from the underlying data. Beyond that, the next most important question relates to enterprise governance since AI operations can change the company posture related to copyright, underwriting and regulatory compliance questions.

As companies shift from experimenting to measuring, I’m seeing harder questions being asked about private and hybrid AI as the true costs of frontier models hosted in hyperscaler data centers become more obvious. No CFO is happy when they are told “I don’t know how much budget I will need for AI Ops at scale, nor can I guarantee I can hit that budget” by their sysadmins and IT architects.

Q3: Much of the AI conversation fixates on the biggest, most expensive hardware. Where does the real case get made on total cost and value per useful output, rather than peak performance?

A: One of the hardest benefits to articulate and demonstrate on a balance sheet is better operational efficiency, since it’s not a hard cost you can show gains or reductions in a way similar to when purchasing hard goods or services. Because of that, buying new expensive hardware that has overhead just from having to deploy new cooling strategies is very difficult to pencil out against “went from 7 days to 5 minutes in developing dashboards that report on overall fleet health”. The current approach on explaining the value in that example is to claim fewer employees are required, which is counterbalanced by increasing token costs from suppliers.

I hear from multiple enterprises that they’ve applied the multi-cloud skills developed over the last 10 years to AI. They are learning to mix and match privately optimized AI and small language models that have been fine-tuned with judicious application of frontier models, hyperscaler services and neoclouds. For the health-meter dashboard example, prototyping is done on a frontier model and then the workflow is moved to a local SLM to improve operational economics.

Q4: The industry is moving from buying components to running one integrated machine. From where you sit with enterprises, is that the real shift, and what does it change about what they buy and how they operate?

A: The combinatorial explosion with agentic AI between discrete agents, frameworks, disparate API ecosystems and overlapping capabilities is resulting in security gaps, version conflict and unpredictable TCO. At the same time, market changes on the virtualization front are causing enterprise decision makers to be very cautious about anything that resembles vendor lock-in since even stable, trusted suppliers can suddenly be acquired by a larger company with very different customer management principles. I have already seen enterprises creating their own interfaces and allowing model selection under their internal APIs so model-independence is built in from the start, and I suspect similar architecture principles will be carried forward where there are important control points in an enterprises’ tech stack.

Q5: You’ve pointed out that most enterprises still can’t answer a basic question: when an AI system produces a harmful or biased result, who is accountable? What has to live in the infrastructure itself, the telemetry, the provenance, the controls, before anyone can answer that honestly, and how much are enterprises still taking on faith today?

A: The telemetry and the provenance for data as well as decision traceability have to be operational code running within the AI systems and the overall orchestration of the AI operations. Any decision that would be difficult to defend in a current regulatory framework needs to have a human in the loop, and at the same time, a human has to have the right insights support from the AI Operations so the human isn’t overwhelmed with overwhelming amounts of data in the process.  

It’s hard to believe that we are now midway through 2026.  Once again, I am looking back at my crystal ball to see how the three major trends I predicted would meaningfully affect the automotive industry have actually played out.

To be clear, these weren’t thoughts pulled out of thin air; they reflected observations of events that had already transpired and that I expected would see significant traction. If you’ve been tracking the industry lately, you’ve probably noticed that the turmoil I described has only intensified. This isn’t a cyclical downturn; it’s a fundamental rewiring of how cars are conceived, built, and sold. And the scorecard, I’m pleased to report, is largely validating what I foresaw.

1. AI-Driven Product Development: Hit the Bullseye (With a Catch)

On AI-driven product development, the prediction has hit the bullseye. As of early 2026, 91% of global OEMs have moved generative AI beyond proof-of-concept into production use, with design cycle reductions of 65–80% are now being reported across the industry. Physics-informed neural networks have reduced the need for physical prototyping by 70%. The 50% reduction in development costs I anticipated is now being exceeded by early adopters. What once demanded months of tedious traceability mapping for ASIL (Automotive Safety Integrity Level) compliance is now orchestrated by agentic AI systems that provide 24/7 compliance monitoring. The competitive moat has indeed shifted from engineering expertise to the sophistication of AI training data and computational infrastructure - exactly as I predicted. However, an extreme reliance on AI has proven to have backfired on Ford as they have recently re-hired over 300 “grey beard” engineers to fix Ford’s quality systems. Overreliance on automated AI tools had negatively impacted their quality assurance pipelines, leading to an unsustainable spike in vehicle recalls.

2. The Software-Defined Vehicle (SDV) Divide Widens

The software-defined vehicle divide has widened even faster than I expected, and the incumbent carnage has been brutal. Ford’s cancellation of its “Lightning” SDV platform was merely the opening act; in 2026, they have since scrapped the all-electric F-150 project entirely to redirect capital toward expanding its hybrid lineup, acknowledging that its EV division is projected to lose up to $5.5 billion in a single year. GM delayed its electric truck production expansion to at least mid-2026 and reintroduced plug-in hybrids it had previously discontinued. Meanwhile, tech-native companies like Tesla and Rivian, together with Chinese OEMs like BYD and NIO, continue to push new functions weekly via over-the-air (OTA) updates while legacy OEMs remain shackled to three-to-five-year hardware refresh cycles. The market bifurcation into haves and have-nots is happening at a heightened pace, and the estimated three-to-four-year delay in SDV deployment for traditional OEMs is creating a compounding disadvantage that grows more insurmountable by the quarter.

3. Incentive Withdrawals, Hybrid Bridges, and the European Twist

On the incentive withdrawal and EV momentum front, the narrative has played out almost exactly as scripted - but with a twist I didn’t fully anticipate.

The withdrawal of EV tax credits in the US in September 2025 triggered the predicted pullback: battery electric vehicle (BEV) sales fell 23% year-on-year in Q1 2026, and plug-in hybrid (PHEV) sales collapsed 53%. Ford and GM’s tactical retreat to hybrids is precisely the “rational bridge technology” response I described. However, what I underestimated was the speed of the policy rebound. France and Germany renewed or reintroduced purchase incentives in early 2026, and European BEV sales surged 36% year-on-year in Q1 as a result. The hybrid resurgence is real - hybrid sales in Europe’s top five markets broke the one-million barrier for the first time in any quarter, reaching a record 42% market share - but the fundamental EV cost-crossover momentum remains unstoppable. This is particularly true in China, where BYD and Geely continue delivering 300-mile range vehicles below $20,000.

The Strategic Takeaway

The strategic implications I laid out are proving more urgent than ever. The great bifurcation is no longer theoretical; it is unfolding in real-time across earnings calls, factory announcements, and market share tables.

Companies that treated AI-driven design as a productivity tool rather than the new basis of competition are now scrambling to catch up. However, as Ford found out, there is such a thing as too much of a good thing. OEMs that attempted to orchestrate SDV platforms across fragmented supplier ecosystems are watching their architectures collapse under their own weight. And those that tethered their EV strategy to Western policy cycles are discovering that policy is a fickle foundation.

The companies thriving are the ones that recognized automotive manufacturing has become a data and software business that happens to produce vehicles. The laggards are still arguing about whether the transformation is real.