Welcome to
the Forum

The cost of serving a model increasingly comes down to how fast accelerators can move data between one another as AI inference workloads shift toward multistep reasoning and mixture-of-experts models.  

Faster chips are not the answer, said Aanchal Sharma, senior director of product management at Astera Labs, because they buy more compute, not less waiting. Put the fastest chip in the world behind a slow fabric, and you have built what she calls “an expensive space heater.”

We sat down with Aanchal as part of our series about how companies that build AI infrastructure are seeing requirements change as deployments scale. Astera Labs builds fabric switches and connectivity products that let AI accelerators from different vendors work together inside a rack.  

She talked about what it takes to prove multi-vendor interoperability before deployment, why Astera Labs builds Scorpio and Taurus as open rather than proprietary interconnects, and where the fabric has to scale next as clusters grow toward hundreds of thousands of accelerators. Here’s what we learned.

Q1: Rack-scale AI infrastructure now depends on many vendors' silicon working together inside one system: accelerators, memory and fabric all from different suppliers. What does that interoperability require in practice, and where does it tend to break down?

A: At rack scale, physics doesn't care whose logo is on the silicon. A GPU from one vendor, a CPU from another, memory from a third: They all have to hold a stable low-latency link under real production traffic every hour of every day for years. That's the bar.  

Most failures show up exactly where you'd expect: signal integrity breaking down across longer or noisier channels, firmware that was never tested together choking on link training, and faults that only appear once hundreds of accelerators fire the same collective operation at the same instant.  

We stress-test these exact combinations in our Cloud-Scale Interop Lab before a customer ever racks a single unit so customers can deploy with confidence and focus their engineering resources on AI innovation rather than infrastructure integration challenges.

Q2: Fabric switches and retimers now have to work across GPUs, CPUs and memory sourced from different vendors within the same rack. What does proving that kind of multi-vendor interoperability involve before a customer can put it into production?

A: It involves validating the rack as a complete system, not just each component in isolation.  

We re-create the customer's topology with the specific CPUs, GPUs, memory devices, fabric switches and retimers, then test across operating systems, software stacks, workloads and protocols. Working with ecosystem partners, we use co-emulation, test automation and continuous regression testing to identify compatibility issues and confirm seamless integration before deployment.

The result is a validated configuration that reduces integration risk and accelerates time to deployment. That's the entire point of an interop lab: catch every integration risk on our bench months before a customer ever has to.

Q3: Astera Labs builds Scorpio and Taurus as open multi-vendor scale-up fabric rather than a single vendor's proprietary interconnect. What does an open fabric let operators do that a closed single-vendor interconnect can't?

A: An open fabric means an operator never has to bet the entire rack on one company's roadmap. Mix the best GPU with the best CPU with the best memory; swap a supplier when lead times blow out; carry hardware forward across generations instead of tearing it out.  

Taurus supports this model across Ethernet, UALink and ESUN, while Scorpio provides an open software-defined fabric architecture that supports diverse accelerators, system topologies, and both open and platform-specific protocols.  

Compared with a closed interconnect, this keeps architecture and supplier choices open as performance, availability and platform requirements change.

Q4: As inference moves into production, a growing share of what it costs to serve a model comes from moving data between accelerators, not from the accelerators themselves. Why can't faster chips alone fix that?

A: Faster chips buy more compute. They don't buy less waiting.  

As inference workloads move toward multistep reasoning and mixture-of-experts architectures, more of the total job becomes accelerators talking to each other rather than accelerators doing math. Put the fastest chip in the world behind a slow, high-latency fabric and you've built an expensive space heater, with GPUs sitting idle waiting on data instead of generating tokens.  

That's exactly why Scorpio builds acceleration for collective operations, like Hypercast, directly into the fabric; why Taurus keeps those links clean and low power across Ethernet, UALink and ESUN as clusters scale; and why COSMOS gives operators the visibility to find and kill communication bottlenecks in real time. Tokens per watt and tokens per dollar get decided in the fabric long before anyone reads a chip's spec sheet.

Q5: As clusters grow toward hundreds of thousands of accelerators spread across racks and rows, where does the connecting fabric have to go next? How do open standards and products built on them fit that path?

A: The fabric has to scale in three directions at once: scale up inside the rack, scale out across racks and rows, and increasingly scale across between clusters and sites.  

Scale-up keeps pushing toward higher radix and lower latency inside the rack. Scale-out has to connect racks and rows without the distance tax of added latency and cost eating the gains. Scale-across is the newest of the three, holding performance together across distances that scale-up and scale-out were never built to cover.  

That means more optical connectivity, higher-density switching, and fabric that carries intelligence, not just bits, accelerating collective operations instead of passively relaying them. Open standards like CXL, Ethernet, PCIe, UALink and ESUN give customers one way to build that path without betting the company, but that's only part of the story. Some of the largest deployments we support run on NVLink Fusion, and others need a fully custom link built around one customer's architecture. Products like Taurus and Scorpio are built to make all three paths work.

As new AI cloud providers appear, many compete on a single number: how many GPUs they can rent by the hour. Dan Brown, who leads hardware engineering at DigitalOcean, sees the real contest elsewhere. In a recent TechArena Data Insights episode, Solidigm’s Jeniece Wnorowski and I talked with Dan and Solidigm account executive Ty MacAdam about building and delivering holistic data center infrastructure solutions for AI workloads. Dan's argument was consistent throughout: raw GPU capacity means little if data cannot reach it quickly, and that is where DigitalOcean is placing its bet.

The Data Velocity Problem

Dan frames the core AI infrastructure challenge around three factors: where and how data sets are captured, how quickly that data can travel from its source into an AI cluster, and how fast systems can exchange tokens between systems in the AI cluster to reduce the data down into a meaningful set that can be used for inference. When any of those steps drags, the entire pipeline slows.

“The AI solution designs are limited by what we call data velocity,” Dan said. “How quickly can you get your data in, transform it, and get it back out for next useful step in the pipeline.”

Much of the friction, he noted, comes from customers trying to run modern AI against aging storage. Many arrive with legacy arrays that have been in service for a decade or more, then discover that infrastructure cannot keep pace. Moving that data into DigitalOcean’s high-performance block and object storage, or directly onto the AI machines, is where the gains show up. Solidigm supplies the high-speed NVMes, including PCIe Gen5 drives, that sit close to the GPUs and keep the read and write cycle tight.

More Than a GPU Landlord

Dan is direct about how DigitalOcean differs from the wave of providers rushing into AI hosting. Many, he argues, do one thing only.

“They’re taking GPUs and servers, shoving them into boxes and then selling them by the hour to customers,” Dan said. “We call it a GPU landlord.” With those providers, he explained, customers must bring their own data scientists, hardware engineers, and expertise to make the hardware useful.

DigitalOcean’s answer is a full suite of compute and storage products that surround the GPU and share the same backbone network and data center, tightly coupled to improve velocity. Rather than selling raw capacity by the hour, Dan said, the company is building a unified cloud where every resource around the AI cluster exists to make that cluster faster and more cost effective to run.

Preference-Aware Routing

A clear expression of that approach is DigitalOcean Inference Router, which reads the intent behind each request and matches it to the model that best fits the developer’s task and priorities, whether they are optimizing for quality, cost, or latency, without any application-side routing logic. The approach is grounded in years of preference-aware routing research from the team behind the open-source Arch-Router work, now part of DigitalOcean. Dan described the result as token as a service, and he said the productivity effect has been striking.

Dan said the productivity gain has been dramatic: teams can now prototype a working product for a few hundred dollars in minutes, rather than standing up entire teams of administrators and data scientists to build the back end first.  

Density and the Nanosecond

Asked what organizations still overlook, Dan pointed to density and reached for a piece of computing history. Admiral Grace Hopper famously wore an 11.8-inch-long coil of copper wire. She used this to illustrate that for every such length of copper in a system, a nanosecond of delay is introduced in that system. Across a large AI cluster, Dan noted, miles of copper and fiber add up to real delay.

DigitalOcean’s response is to build the densest possible clusters, adopt liquid cooling, and keep local Solidigm storage on each server so data moves host to host in microseconds rather than milliseconds. That translates directly into shorter compute time, faster time to first token, and lower customer costs.

Ty reinforced the point from the storage side. “Storage used to be a passive layer where everybody just thought of it as where the data would sit. That doesn’t hold up anymore,” he said. “Storage isn’t where the data sits. It’s more about how fast intelligence moves.” He added that Solidigm’s high density quad-level call (QLC) NAND puts more data closer to compute at lower power, which reshapes the total cost of ownership math at deployment scale.

The TechArena Take

DigitalOcean’s message to technology decision makers is a useful reminder to a market fixated on GPU counts. Buying accelerators and fast networking is only part of the equation. If data cannot reach those GPUs quickly, or if teams overspend on oversized models for narrow jobs, the investment underperforms. Refreshing legacy storage, right-sizing a model to a task, treating proximity between data and compute as a design decision rather than an afterthought are practical places to start to get the most out of an AI infrastructure investment. As agentic workloads raise the pressure further, the providers building for velocity, not just capacity, look best positioned for what comes next.

For more information, listen to the full podcast and visit DigitalOcean.com.

The same chip architecture that Axelera AI built for the power, cost and latency constraints of edge vision AI can now also handle multiuser generative AI workloads at the edge, using the same Metis and Europa chip family and toolchain.

We sat down with Manuel Botija of Axelera AI, which builds AI chips for edge devices. Manuel said that expanding capability matters as more AI workloads move beyond the data center and edge devices are asked to run increasingly complex models. Europa, he said, is built to run vision-language models and agentic AI directly on edge hardware.

We also talked about what digital in-memory computing changes about chip performance; how customers worldwide are using Axelera AI’s technology; and what it takes to offer a predictable cost per token as inference workloads grow.  

Here’s what we learned.

Q1: Much of the progress in AI to date has come from scaling the data center, but only a handful of players can build at that scale. Where did Axelera AI see a different path?

A: Most of the industry has tried to scale AI from the data center out to the edge, and that direction has never really worked. Data center architectures are built for abundant power, abundant budget and abundant time. None of that is available at the edge. So the chips and systems built for one environment were getting forced into a location they were never designed for, which meant serious tradeoffs had to be made.

We believed there was a different way and started by holding those “limits” as real limits. We built an architecture that solves for small power envelopes, limited budgets and time from the beginning, because at the edge, those constraints are real.  

Our first architecture, the Metis AIPU, had to hit real performance within a tight power envelope and a real price point, and that discipline is exactly what let us scale to the Europa AIPU without compromising the efficiency we perfected. We have even taken that edge architecture with both Metis and Europa and built server-class products supporting customers globally with full-length, full-height PCIe cards built around multiple AIPUs.

Q2: Axelera AI made its name at the edge with the Metis platform. What are customers using the products you’ve brought to market to solve?

A: Customers are putting Metis to work solving real problems in places most people never think to look.  

A large global convenience store chain uses it for real-time shelf monitoring, catching stockouts as they happen. An agritech company in New Zealand built it into automated apple sorters, and a healthcare technology company in Europe uses it inside an automated pill sorting machine, both cases where accuracy and speed directly affect people’s safety.  

The utilization keeps expanding. A drone company in Croatia deploys Metis in a drone pack built for search and rescue missions. In India, a company built a cargo monitoring system for the country’s busiest port, bringing computer vision to logistics at scale.

These are just a handful of the deployments in production, with new uses for vision and localized models emerging around the world. What stands out is how different these use cases are from one another, and how each one found its way to edge AI because it was the only architecture that could meet their constraints on power, cost and latency at once. It’s a reminder of how much innovation is happening at the edge, in every corner of the world.

Q3: At the heart of your chips is digital in-memory computing on a RISC-V core. What does moving the math into memory change about performance and efficiency?

A: Traditional processors spend most of their energy and time moving data back and forth between memory and compute, especially for the matrix-vector multiplication that makes up roughly 80% of AI inference workloads.  

Digital in-memory computing performs that math directly inside the memory cell, so the data barely has to move at all. That single change removes the biggest bottleneck in conventional architectures and turns into real gains in both speed and power. The RISC-V controlled dataflow is what makes this programmable and precise rather than a fixed-function shortcut. It gives us four independently programmable cores that can run parallel model execution with deterministic, digital accuracy, unlike analog in-memory approaches that trade precision for efficiency.  

That combination is why the Metis architecture delivers up to 15 TOPS per watt and three times better performance per watt than GPU-based solutions, all while keeping accuracy indistinguishable from a full floating-point model. The result is an architecture that scales cleanly. The same principles that made the Metis AIPU efficient at 214 TOPS carry through to the Europa AIPU at 629 TOPS, which means moving the math into memory isn’t just a one-time efficiency trick. It’s the foundation of our roadmap.

Q4: Axelera AI is scaling one architecture from Metis at the edge up to Europa for multiuser generative AI. What does that move open up for your customers?

A: The Europa AIPU is built on the same digital in-memory computing and RISC-V dataflow foundation as Metis, but with meaningful additions: twice the AI cores, native video decode, and 16 vector cores dedicated to pre- and post-processing, all of which make Europa perfectly suited for larger and more complex models and analytics. Those additions are what make native transformer support and multiuser generative AI workloads possible at the edge.

What stays constant for customers is the development experience. The same Voyager toolchain carries across both chips, so a team that built a pipeline on Metis isn’t starting over on Europa. That continuity is what turns an architectural leap into a growth path instead of a rebuild.  

Metis remains an efficient powerhouse for computer vision use cases at the edge, including embedded systems where power and space are tightly constrained. Europa extends that foundation into a highly flexible AI accelerator built for a broader range of inferencing needs, from vision-language models and agentic AI to dozens of concurrent high-resolution video streams running natively at the edge. For customers, that means the same tooling and philosophy they already trust, now available across an even wider set of use cases.

Q5: Two years out, what will separate an inference platform that lets an operator promise a predictable cost per token from one that can’t?

A: Digital in-memory computing removes the data movement bottleneck that makes costs scale unpredictably as workloads grow, so the economics hold steady whether an operator is running a handful of streams or scaling to dozens of concurrent workloads.

The other separator is architectural flexibility. Operators locked into separate chips for CNNs and separate chips for transformer-based models face a cost structure that multiplies every time they add a new model type. A unified architecture that handles both on the same silicon keeps cost per token stable as workloads evolve from vision to generative AI, which is the discipline that will define the platforms operators can build a business on two years from now. Europa delivers that flexibility to customers.

Lastly, when models are run on one’s own infrastructure, the cost per token is much more manageable than running models in the clouds through providers that change cost structures at their discretion. With the Axelera Voyager toolchain, customers can use open-source models tuned to their weights and deploy them in-house. This includes Voyager Wingman, an agentic platform that can build pipelines based on natural language prompts.

We sat down with David Driggers, CEO and founder of Cirrascale Cloud Services, a neocloud that runs dedicated hardware for AI workloads across NVIDIA, AMD, Qualcomm and Tenstorrent accelerators. Cirrascale’s approach is relevant now as enterprises move from proof-of-concept AI deployments toward production and need infrastructure that matches each workload to the chip built for it.

We talked about how Cirrascale matches workloads to the right accelerator as models grow, how its billing model creates cost predictability, and what separates neoclouds built for low-latency inference from those built for training alone. Here's what we learned.

Q1: Why run every accelerator on dedicated hardware?

A: Cirrascale gets classified as a neocloud, but we were building this hardware before the category had a name. We designed the industry’s first 8-GPU server back in 2012 working directly with NVIDIA, and we were infrastructure partners to OpenAI when they were still an 8-10 person team. That hardware DNA is why we can run NVIDIA, AMD, Qualcomm and Tenstorrent side by side, alongside our work with Google on Google Distributed Cloud, and actually know what each one is good for. No single chip serves a 1B-parameter model and a 400B-parameter model economically, so our customers get the accelerator that fits their workload instead of bending their workload to fit whatever one vendor is selling.

Q2: How does billing create predictability?

A: We don’t charge for data ingress or egress, which removes one of the biggest hidden cost swings customers deal with elsewhere. We also give customers the choice between token-based pricing for intermittent workloads and GPU-hour billing for anything running 24/7, so the bill follows how they use the platform. And because we track token consumption at a granular level, customers get real insight into where they’re burning input tokens inefficiently. That data helps them optimize the model, not just the invoice. Inference is forever, as we like to put it, and those costs compound. That’s why we built a cost calculator: Customers can model their own workload before they commit to anything.

Q3: What’s standing between customers and more capacity, and what’s changed?

A: Most enterprises are still in the POC phase with models like Gemini, GPT and Claude, trying to prove utilization before they can get more approved budget. Nobody’s handing out incremental AI spend without proof it moves the needle. That’s part of why we run Private Gemini on Google Distributed Cloud. It gets a frontier model into production inside the customer’s own security boundary instead of leaving it stalled in evaluation. On the hardware side, we’re pushing into multimodal, larger-context inference. What’s changed in the last year is that hybrid deployment, on-prem for steady workloads plus neocloud for real-time workloads across regions, is becoming the norm. I expect that to be standard by the end of this year and accelerate hard through 2027.

Q4: What have you learned matching workload to chip?

A: We built the load balancing in the Cirrascale Inference Platform ourselves, specifically because a model’s ideal hardware changes as it grows. An 8B model that becomes a 70B model isn’t cost-optimal on its original chip anymore, and that kind of migration is really hard to do on-prem. The single biggest cost lever is capacity utilization. The platform puts each workload on the most efficient capacity that still clears its latency target, so customers aren’t paying premium rates for headroom they don’t need.  

Q5: What separates the neoclouds that thrive from the ones that fade?

A: Training and inference are completely different animals. In training, you can run a thousand GPUs and barely notice a network hiccup. In inference, that same hiccup is a failed request in front of a customer. The neoclouds that survive are the ones built for real-time workloads and low latency from the ground up, not the ones bolting inference onto training infrastructure. Carrier-hotel-grade connectivity is a big part of that, which is why we partnered with Telehouse to deploy the Cirrascale Inference Platform inside their data centers. It also means being able to serve regulated industries, where FedRAMP High and CMMC 2.0 are the price of entry rather than a roadmap item. We also give customers real ownership flexibility. Some customers own their GPUs outright while we run the networking and storage around them. And we’re expanding our data center footprint this year to put that capacity and failover closer to where the inference actually happens.

This summer, TechArena has been talking to the companies building the AI data centers of tomorrow.  

We caught up with Steven Carlini, chief advocate for AI and data centers at Schneider Electric, which designs power, cooling and controls for AI facilities as one integrated system. As rack densities climb and utility power becomes the tightest constraint on new AI capacity, how quickly that infrastructure can be designed and deployed is becoming one of the sharpest bottlenecks in the industry.

We talked about the changing role of power in AI projects, what’s driving how fast new AI capacity can come online today, and where the gap is widest between what AI buildouts need and what today’s grid and supply chain can deliver.

Q1: AI infrastructure is shifting from separately bought parts to one integrated system. How does Schneider Electric help operators design power, cooling and control as a whole?

A: AI infrastructure performs best when power, cooling, controls and software are designed as one integrated technology system instead of assembled piece by piece. Schneider Electric acts as an energy technology partner by bringing together electrical distribution, liquid cooling, racks, digital management and services into a unified architecture in the form of reference designs and prefabricated modules. These help customers reduce complexity, uncertainty and risk, and they help accelerate deployment. By designing these systems together from the outset, operators can improve efficiency, simplify operations and reduce time to power for new AI capacity.

Q2: As the rack becomes the unit of design, how is power’s role on an AI project changing?

A: Power has become a strategic enabler of AI deployment. As rack densities continue to climb, we are bumping up against the laws of physics and running out of physical space for the power inputs. Delivering reliable power to each rack is as challenging as delivering liquid cooling and as important as delivering power to the facility. Every decision around electrical distribution, cooling and controls affects how quickly infrastructure can be commissioned. Designing around the rack helps operators maximize available power, support higher densities and bring AI capacity online faster.

Q3: What determines how quickly new AI capacity can come online right now?

A: Today, speed is driven by time to power. Access to utility power or constructing onsite or adjacent prime power (energy parks) dedicated to the data center, electrical infrastructure, cooling and coordinated supply chains all influence how quickly AI clusters can be deployed. Organizations that take an integrated approach to infrastructure planning and use validated, end-to-end solutions can reduce project complexity, shorten commissioning timelines, and accelerate time to revenue.

Q4: Where is the widest gap between what AI buildouts need and what today’s grid and supply chain can deliver?

A: Demand for AI infrastructure is growing much faster than traditional power infrastructure can expand. Operators need more capacity, higher density cooling and shorter deployment timelines than existing facilities were designed to support. Closing that gap requires maximizing every available megawatt through efficient infrastructure, modernizing the grid, streamlining processes for expansion and access, building new energy parks where needed, and deploying integrated and intelligent power distribution solutions that reduce time to power.

Q5: What is the next hard problem in power that the industry hasn’t solved yet?

A: The next challenge is not only securing more power; it is reducing the time it takes to bring that power into service and making every available kilowatt work harder to generate tokens and intelligence. As AI workloads continue to scale, operators need infrastructure that can adapt dynamically across power, cooling and operations. The path forward is not simply adding more capacity but creating smarter infrastructure that helps organizations accelerate time to power while meeting the growing demands of AI. That requires moving from disconnected systems to intelligent, integrated infrastructure. With EcoStruxure, Schneider Electric connects energy, power and facility systems to help operators gain real-time visibility, improve efficiency and optimize performance.

Want to learn more about Schneider Electric's AI data center deployments? Check out this article.

Per aspera ad astra: through hardship, to the stars. It’s a fitting concept for this industry, where the work that gets us somewhere extraordinary can look ordinary. Cooling systems. Fabrics. Data layers. Control planes. These critical systems often don’t get the attention they deserve given they determine how far AI can actually go. When we launched Ad Astra: AI Infrastructure Competition as an official media partner of the 2026 AI Infra Summit, we wanted to aim a light at that work.

We asked for 200 words. Make the case for why your technology belongs among this year’s most significant advances in AI infrastructure. Our judging panel composed of TechArena Advisors evaluated every entry against three criteria:

• alignment to one of the summit’s focus areas

• real-world impact backed by measurable outcomes

• clarity that a senior technical or business decision-maker could absorb without decoding jargon

Today I’m pleased to announce the results.

Winner: Airsys

Airsys took the top spot with LiquidRack, a liquid cooling system built around a straightforward observation: AI is driving rack densities that existing cooling infrastructure was never designed to handle.

Rather than routing cooling through centralized facility infrastructure, LiquidRack integrates it directly into the rack. Airsys makes the case that this changes the deployment math for operators, who can add AI capacity incrementally, expand over time, and extract more compute from the power they already have rather than overhauling an entire facility to get there.

The company backs the approach with specifics. Airsys reports that its patented spray cooling technology transfers heat up to three times more efficiently than conventional approaches, freeing more power for compute. It cites a PUE below 1.02, no water consumption, and roughly 80% less dielectric fluid than immersion cooling. The company also notes that LiquidRack has been validated in customer deployments across multiple industries and geographies, which moves the conversation past the lab.

Our TechArena Advisor Laura St. John gave the following reasoning for choosing Airsys as the winner:

“Airsys’s LiquidRack paired a clear innovation with the strongest real-world evidence in the pool, backed by validated deployments across multiple industries, not just projections. In a field tackling one of AI infrastructure's biggest bottlenecks, cooling density, LiquidRack stood out as ready to deploy today.”

Honorable Mentions

Three additional companies earned recognition from our Advisors.

VAST Data argued that AI has transformed computing while much of the underlying data infrastructure still reflects an earlier era of enterprise IT, with separate storage systems, databases, event streams, and vector stores endlessly copying the same information. The VAST AI OS collapses those layers into a single shared platform. The company says it now powers more than 3 million GPUs worldwide, including AI factories where one system serves over a billion CUDA cores.

Cornelis Networks made a data movement case that emphasizes the network’s critical role in the AI era. The company points to production AI clusters running at 30 to 50 percent model FLOPs utilization, with the gap lost to congestion and synchronization stalls. Its CN6000 SuperNIC carries Omni-Path, RoCEv2, and Ultra Ethernet on a single 800 Gbps adapter. Cornelis modeling indicates AllReduce collectives complete roughly 24 percent faster than standard Ethernet, which the company translates to about 13% less training time for a 250-billion-parameter model on a simulated 10,000 GPU cluster.

Rafay addressed the operational sprawl of running an AI data center, unifying Kubernetes, virtual machines, SLURM, bare metal, GPUs, and AI services under one control plane, with Token Factory extending that model into inference by turning GPU capacity into metered, standardized services. In production environments, Rafay reports improving GPU cluster efficiency by 25% to 40% and supporting three to five times more tenants without adding headcount.

The TechArena Take

The binding constraint on AI infrastructure has moved off the chip and into everything surrounding it, and the companies competing hardest right now are the ones working that perimeter. Airsys addressed thermal and power headroom, VAST Data addressed the data layer, Cornelis Networks addressed movement between GPUs, and Rafay addressed the operational model that holds the whole thing together. Congratulations to Airsys, VAST Data, Cornelis Networks, and Rafay. We will see you in Santa Clara this September.

Read more about our winners here.

Free Tickets and Discounts from TechArena

If you haven’t yet registered for the AI Infra Summit, now is the time to act! As part of our partnership, we’re pleased to offer benefits to our TechArena followers who want to join the conversation in Santa Clara this fall:

• Free expo tickets (valued at $447) are available to qualified individuals working within the AI infrastructure field. Apply for free expo tickets.

• A registration discount is available for full access and VIP tickets. Quote TECHARENA15 on the registration page to save 15%.

TechArena has won two Gold Stevie Awards as part of the 23rd Annual International Business Awards, organizers of the prestigious competition announced today.  

The “In the Arena” podcast, hosted by TechArena founder and CEO Allyson Klein, earned Gold in the technology category, and the TechArena Forum, the company’s online platform devoted to AI technologies, infrastructure and the people who build them, earned Gold for achievement in web writing and content.

“This is such a proud day for TechArena and every team member who has contributed, from landing a podcast guest, to developing a question set, to recording and editing our content, to drafting an article that speaks to our North Star of educating on AI infrastructure’s advancement," Klein said.  

Back-to-Back Recognition for Podcast

The Gold marks the second straight year "In the Arena” has earned a Stevie. TechArena’s flagship podcast brings listeners inside the conversations shaping AI, data centers, silicon, edge computing and sustainability, drawing guests from the companies that are building that future.  

“To have the judges recognize that for a second year, and with Gold, means the world to this team," Klein said.

The judges took notice of both the show's pace and its pull.  

"Keeping up a clip of 260+ episodes since 2024 is hard enough, but landing a Nobel laureate and Fermilab experts shows how much industry pull this show has," one judge wrote. "Allyson skips the surface-level AI hype and gets right into the actual, deep infrastructure problems enterprise tech is facing."

A second judge wrote, "TechArena has worked hard to become a trusted, independent platform that has earned recognition as a vital voice in the global technology community." The judge also pointed out the high-caliber guest list, which has featured leaders from Synopsys, Arm, Intel, Cerebras, Walmart and CoreWeave, calling the show "one of enterprise tech's most prolific voices."  

A TechArena Forum First

The Forum's Gold marks the first Stevie recognition for TechArena's written editorial platform, which in a two-year span has published 734 pieces across 11 topic verticals including AI, quantum computing, silicon, edge, sustainability and cybersecurity. The Forum pairs original reporting and analysis from TechArena's editorial team with contributed perspectives from technologists and executives across the industry.

The judges singled out the substance behind that output. "Remarkably substantial editorial output ... with genuine expert contributors from major companies," one wrote. "Real editorial rigor rather than vendor-driven filler content."

Another judge praised the Forum for delivering "high-quality, insightful content with impressive technical depth while making complex topics accessible to enterprise audiences," adding that its editorial model and diversity of expert contributors "create a valuable knowledge platform."

"Building the Forum has been a labor of love from Day 1. It was something Allyson and I discussed at the dawn of TechArena," said Rachel Horton, head of content and editorial.  

What Comes Next

The double win caps a banner stretch for TechArena, and the team is not slowing down. New episodes of “In the Arena” publish weekly, and the Forum continues to expand its roster of expert voices across the technologies reshaping the enterprise.

The International Business Awards, the world's premier business awards program, honor achievement by organizations of every size and type across more than 60 nations. Last year's competition drew more than 3,800 nominations from organizations in 78 nations and markets. Winners were determined by the average scores of executives worldwide serving on the program's judging committees, and Gold marks the highest score tier a nomination can achieve.

Quantum technology has spent years as a domain defined by its promise rather than its deployments. That is beginning to change. The question now is not if quantum belongs on technological roadmaps for real business environments but how quickly it can be made reliable, integrated and usable.

That challenge sits at the center of the work being done at Q-CTRL. Recently, I had the chance to explore that work with the company's vice president of product, Alex Shih, alongside my "Data Insights" cohost, Solidigm's Jeniece Wnorowski. What emerged from the conversation is a clear view that the path for quantum, from laboratory demonstration to enterprise deployment, runs directly through software.

From Deep Science to Deployable Solutions

Q-CTRL's core work is translating quantum physics and quantum control expertise into products that customers can run. Alex described his role as taking deep science and turning it into scalable solutions for data centers, high-performance computing facilities and enterprise businesses. The company also delivers a quantum education platform to help organizations build and scale quantum talent, along with quantum sensing solutions for navigation.

Alex came to the field after a career spanning big technology companies, startups and space technology, most recently at Slack. He was drawn to quantum for its long-term potential. "I've always viewed quantum as a generational step-change technology that would introduce a new compute paradigm," he said.

Controlling Instability With Software

The central technical challenges in quantum systems are hardware instability, driven in large part by noise from both the hardware itself and the surrounding environment, and the difficulty of scaling that hardware. Q-CTRL develops what Alex described as AI-powered error-reduction quantum infrastructure software, which reduces that noise and suppresses the resulting errors so the hardware behaves reliably enough to build on.

That reliability serves two audiences. For hardware teams, it helps accelerate the development of their qubits. For enterprises developing and testing applications on real quantum hardware, it means the output can be trusted, which, as Alex noted, is what allows customers to properly assess how their applications perform.

Built for the Real World, Not the Journal

Alex was emphatic that Q-CTRL's purpose is to put quantum tools into working business environments. "While important to showcase progress in the field, we are not in the business of just publishing papers but rather removing barriers for customers to use our tools," he said. The goal, he explained, is for those tools to be accessible and used by companies around the world.

That focus is reflected in a recent deployment. Q-CTRL went live in Colorado with Elevate Quantum in a time frame that Alex described as a record for the industry, moving from the announcement of a planned system to a working one in only five months, at a fraction of the typical cost. He framed the achievement as a blueprint rather than a one-time result, expecting subsequent deployments to move even faster. "We've seen Murphy's Law at play; we've seen everything that can go wrong, did go wrong. But now we have ways to mitigate that," Alex said.

Broadening the User Base

Q-CTRL has been serving quantum algorithm developers through Fire Opal, which focuses on error reduction for algorithmic execution, and Boulder Opal, delivering autocalibration for hardware vendors. Since then, the company has built layers of abstraction that expand the range of people it can serve, including domain experts such as chemistry analysts and optimization specialists who can submit data in formats familiar to them.

Application developers represent another audience. Companies including ColibriTD in France and Qunova Computing in South Korea have integrated their own applications on top of Q-CTRL's core pipeline and reported improved performance. The company's education platform, Black Opal, meanwhile, targets traditional software developers who are interested in quantum and continues to drive adoption.

Toward Mainstream Integration

Looking ahead, Alex expressed confidence that quantum will integrate into enterprise workflows and production technology stacks. He pointed to Q-CTRL's work with Riken in Japan, home to the Fugaku supercomputer, an IBM Quantum System Two and a Quantinuum system, as an example of exploring hybrid workflows that combine quantum and classical compute.

Based on hardware vendor roadmaps and the company's experience with customer problems, Alex sees commercial quantum computing advantage in just a few years,   following the company's recent demonstration of practical quantum advantage. "We have confidence that in the next two to three years, we will see meaningful quantum advantage in specific workflows, especially around optimization," he said. He cited a case study with Mazda, where quantum methods required five times less training data to reach a vehicle frame design the company had previously optimized with best-in-class tools, translating directly into reduced research and development cost.

The TechArena Take

Quantum computing has long carried the reputation of a technology perpetually a few years away. What stands out in Q-CTRL's approach is its insistence on accessibility and deployment. By positioning software as the connective layer that brings hardware, control electronics and classical infrastructure together, the company is helping shorten a development cycle that has historically demanded teams of specialists and significant capital. For technology decision-makers, the practical takeaway is that quantum commercial readiness may hinge less on any single hardware breakthrough and more on the software and integration work that makes these systems performant at scale.

Learn more about Q-CTRL at q-ctrl.com, or watch the full podcast episode for more information.

As the AI Infra Summit approaches, TechArena has been chatting with the builders of AI infrastructure about how their requirements are changing as deployments scale.

We sat down with Marc Austin, CEO and co-founder of Hedgehog, a company that builds open software-defined networking for AI clusters. Hedgehog’s approach is mattering more now, as AI infrastructure spreads from single racks to multiple sites and a single misconfigured switch can leave GPUs across a cluster waiting on the network.

We talked about how Hedgehog’s software replaces switch-by-switch network engineering with a declarative operating model, why the company holds reference architecture validation from both NVIDIA and the Open Compute Project, and what it takes to keep job completion times down as clusters scale across sites. Here’s what we learned.

Q1: Hedgehog builds an open software-defined fabric that operators manage as one system rather than as a network stitched together switch by switch. What does that let an operator do at scale that the box-by-box model can’t?

A: It turns the network from a monthslong engineering project into a product you deploy.  

Designing an AI network from scratch means making thousands of interdependent decisions (topology, QoS, congestion control, failure domains) and then keeping them consistent across hundreds of switches by hand.  

With Hedgehog, an operator declares the cluster they want, and the fabric handles Day-0 through Day-N automatically: provisioning, lossless Ethernet tuning, congestion-aware routing and failure recovery, all through Kubernetes-native operations.  

That’s what lets a lean cloud team network like a hyperscaler.  

Q2: Hedgehog supports the NVIDIA Cloud Partner reference architecture, validated for Spectrum-X, and the Open Compute Project (OCP) reference architectures for AI training and inference. Why does that dual validation matter to an operator deciding how to build an AI cluster today?

A: At NVIDIA GTC 2026, we announced support for NVIDIA Spectrum-X Ethernet and the NVIDIA Cloud Partner reference architecture. At the 2026 OCP EMEA Summit, our AI training and inference fabric designs became OCP Accepted: 100% open source, hyperscaler reviewed, with complete BOMs available today on the OCP Marketplace.  

An operator can standardize on Spectrum-X, deploy a fully open SONiC-based fabric on OCP hardware, or run both, and keep a single declarative operating model across all of it. No other network vendor gives operators that freedom at the architecture layer.

Q3: In an AI cluster, every idle GPU is usually waiting on the network. What does the fabric have to get right to keep job completion times down, and why is that harder to do with closed, manually operated networks?

A: The metric that matters is job completion time, and the network defends it or destroys it.  

The fabric has to get five things right at once: lossless Ethernet with properly configured RoCEv2 QoS; congestion-aware routing; rail-optimized topology; dual-plane resilience, so a single failure never stalls a training run; and telemetry with automated recovery, so problems are fixed faster than a GPU can starve.  

In a closed, manually operated network, each of those is a specialist tuning exercise repeated per deployment. Any one misconfigured switch quietly taxes every job on the cluster. Hedgehog ships them as validated defaults in the reference architecture and enforces them continuously in software so that the fabric stays out of the GPUs’ way.

Q4: As infrastructure becomes composable, the network becomes the reassembling component. What does that demand from network software?

A: Composability is fundamentally a network abstraction problem, and hyperscalers already showed the answer: open networking plus VPC abstractions. Hedgehog brings that same model to AI clusters.  

Tenants get VPCs that can span multiple accelerator types and pods, with policy-based multitenant isolation, composed and recomposed in software rather than recabled. We’ve proven this live with disaggregated inference: NVIDIA GPUs handling prefill, SambaNova RDUs handling decode and Intel Xeon orchestration, all running across one Hedgehog fabric with no network penalty.  

Because the whole thing is operated through a declarative, Kubernetes-native model, AI infrastructure can be run with a small team.

Q5: As clusters grow, multiply, and spread across sites, where does network software have to go next? What role do open reference architectures play in letting the industry scale without every operator redesigning the network from scratch?

A: Network software has to scale in two directions at once: up and out.  

“Up” means prescriptive scale-out designs, with our OCP reference architectures defining Open Pod Group scalable units from 64 to 1,024 xPUs and a roadmap to Ultra Ethernet as the interconnect evolves.  

“Out” means treating many pods, clusters, and sites as one operable system, with hybrid multicloud routing and multitenant security built in rather than bolted on.  

Open reference architectures are what make that repeatable: Instead of every operator rederiving the design math, validated architectures with complete BOMs are published on the OCP Marketplace for anyone to build from. Everyone gets to network like a hyperscaler, and the industry scales without redesigning the network from scratch each time.

Ahead of AI Infra Summit, TechArena has been talking to companies building the AI data centers of tomorrow. We caught up with Jeniece Wnorowski, Director of Content Strategy & Industry Expert Programs at Solidigm, a company built on decades of memory and storage expertise now focused squarely on keeping GPUs fed as AI workloads scale. As models grow larger and context windows stretch longer, the gap between compute and the data it depends on is becoming one of the sharpest constraints in AI infrastructure.

We talked about why storage requirements have shifted from cheap, reliable capacity to keeping the GPU from ever waiting; where the data path most often breaks down, and what it really takes to size storage for the bursty, synchronized write demands of checkpointing at scale.

Q1: What are AI builders asking storage to do today that they weren't before the AI boom?

A: The shift is from "store it cheaply and reliably" to "keep the GPU from ever waiting." Four demands stand out.

First, raw density per rack unit: AI clusters are power- and space-constrained before they're capacity-constrained, so builders want maximum terabytes per watt and per slot. AI ambitions are grounded by the realities of physical footprint and power constraints, and high-capacity SSDs can dramatically reduce the storage footprint compared with HDD-based solutions. Second, sustained high-throughput reads at massive scale for training data ingestion and checkpoint restores, not just peak IOPS. Third, predictable latency at scale. AI pipelines need storage that can sustain throughput without long-tail latency spikes that leave expensive accelerators idle. And fourth, storage that sits physically and logically closer to the GPU itself.

That last point reflects a fundamental shift in AI infrastructure. Inference, RAG, vector databases, model repositories, and feature stores all require fast access to much larger active datasets than traditional enterprise applications. As models outgrow available HBM, SSDs increasingly act as a memory-expansion tier, storing model weights or KV-cache data that can be swapped in and out of GPU memory on demand. Solidigm partner Metrum AI demonstrated how fast SSDs can extend effective GPU memory capacity by serving as a storage tier for model components not actively resident in GPU RAM

Q2: Where in the data path does feeding the GPU break down most often today?

A: The most common failure points are at the two ends of the training pipeline: data ingestion and checkpoint write/restore.

During training, if storage cannot sustain the random-read throughput needed to keep shuffled batches flowing, GPUs sit idle waiting on data loaders, which is an expensive way to burn compute budget. During checkpointing, entire jobs often pause while model state is written to persistent storage. If those writes are slow, every GPU in the cluster idles simultaneously.

A third bottleneck is emerging between memory and compute: the KV cache and the memory wall. In inference, every token of context generates key/value state that consumes GPU memory. As context lengths, batch sizes, and model sizes grow, KV cache can quickly exhaust available HBM. Operators are then forced to limit context windows, reduce batching efficiency, recompute state, or introduce a lower storage tier. When data is offloaded, storage becomes part of the inference path itself. SSD-backed memory expansion can help hold model weights or KV-cache data that are swapped into GPU memory as needed, extending model capability beyond the limits of HBM alone.

This is one reason the industry is increasingly focused on architectures that move data more directly between SSDs and GPUs, minimizing CPU and software-stack overhead.  

Q3: What do teams most often get wrong about storage for checkpointing at scale?

A: The most common mistake is sizing storage around capacity and average throughput rather than the bursty, synchronized write behavior that checkpointing actually creates. Hundreds or thousands of GPUs may attempt to flush model state at nearly the same moment, creating a concentrated “write” storm that can overwhelm infrastructure designed around average utilization.

Organizations also underestimate the tradeoff between checkpoint frequency and productivity. Checkpoint too rarely, and failures can result in hours of retraining and lost compute. Checkpoint too often, and the storage layer itself becomes the bottleneck.

There's also frequently a mismatch between endurance assumptions and real-world behavior. Teams often purchase expensive high-endurance media for workloads that are primarily large, sequential writes rather than small random overwrites. Understanding the actual checkpoint profile is often more important than simply optimizing for maximum endurance specifications.  

Q4: As drive capacities push past 100 terabytes, what does that density solve for AI builders, and where does it stop helping?

A: What ultra-high-capacity drives solve is largely physical and economic: fewer drives, fewer rack slots, lower power consumption, less cooling overhead dedicated to storage, and simpler deployment when a single storage platform can span multiple capacity points.  

Where density stops helping is when capacity growth outpaces performance growth. Capacity does not scale linearly with IOPS or bandwidth per drive, and ultra-high-capacity SSDs are optimized primarily for density and efficiency rather than maximum performance. Beyond a certain point, adding capacity alone can simply concentrate a bottleneck.

The other consideration is operational resilience. A 122TB drive contains significantly more data than smaller-capacity devices, increasing the importance of data protection strategies, erasure coding, and rebuild planning. However, capacity and resilience should be evaluated together. Modern SSDs can restore data dramatically faster than HDD-based alternatives, meaning organizations must consider both failure impact and recovery time when evaluating the tradeoffs of higher-capacity media.  

Q5: As AI scales, what should operators understand about how storage shapes the cost and quality of the tokens they deliver?

A: The core idea is that GPU utilization and model quality are the outcomes storage ultimately influences.

GPU compute is the scarce and expensive resource in modern AI infrastructure, and storage's job is to ensure that resource remains productive. Every idle GPU second caused by a stalled data loader, slow checkpointing operation, storage-network congestion, or memory bottleneck directly increases training costs and degrades economics.

Storage decisions increasingly affect model quality as well. Faster, denser storage enables organizations to train on larger and more diverse datasets, checkpoint more frequently without sacrificing throughput, and scale model size without making I/O the limiting factor.

On the inference side, the emerging role of SSDs as a memory-expansion tier is especially important. SSD-backed storage can hold model weights or KV-cache data that overflow HBM, allowing operators to support longer context windows and higher user concurrency. That directly affects both the quality of the tokens a model can generate, because it can access more context, and the cost of those tokens, because each GPU can serve more requests efficiently.

Organizations that continue to view storage as a background utility rather than a first-class component of the AI compute stack often discover the consequences in their GPU utilization metrics, inference throughput, and overall, AI economics.

This summer, TechArena has been asking the companies that build AI infrastructure how their requirements are changing as deployments scale.

For our latest conversation in this series, we sat down with Ryan Brown, Director of Data Center Product Management at Phononic, a company that puts cooling directly on the component, right where heat is generated. The approach matters more now, as rack densities climb and the list of thermally sensitive parts stretches well beyond the GPU.

We talked about what targeted, solid-state cooling makes possible inside a dense AI rack; how pairing it with liquid cooling changes both performance and economics; and why coolers that read voltage, current, and temperature at each hotspot turn thermal management into an operational tool. Here's what we learned.

Q1: Phononic puts solid-state cooling right on the hotspot, cooling a specific component on demand rather than the whole room or loop. What does that kind of targeted cooling make possible in a dense AI rack?

A: Traditional cooling systems remove heat from the rack or server level. Phononic’s solid state approach addresses heat exactly where it is generated—at the hotspot. By actively and dynamically controlling temperature at the component level, operators can maintain tighter thermal margins on the devices that matter most, even as rack densities continue to climb.

That targeted cooling enables higher compute utilization, more predictable performance, and greater design flexibility. Rather than overcooling an entire rack to protect a handful of critical components, operators can deliver cooling precisely where it is needed, helping unlock higher-density AI infrastructure while improving overall system efficiency.

Q2: Liquid cooling is the default now, and you describe solid-state as working with it, targeting hotspots so operators can run warmer loops and over-provision less. How does that pairing change the performance and the economics?

A: We do not see solid-state cooling as a replacement for liquid cooling—we see it as a complementary technology. Liquid cooling excels at removing large amounts of heat from the system, while solid-state cooling delivers millisecond-scale, precision temperature control at the most thermally sensitive locations.

That combination allows operators to run warmer coolant temperatures, reduce thermal variability across critical components, and minimize the need for excessive cooling headroom. The result is a better balance between performance and efficiency: improved utilization of expensive AI hardware, lower cooling-system complexity, and potentially reduced energy and infrastructure costs.

Q3: Everyone watches the GPU, but you cool hotspots across HBM stacks, voltage regulators, and the lasers inside high-speed optics, where you already ship tens of millions of coolers. As density climbs, which of these is becoming the real limit on performance?

A: The answer is increasingly “all of the above.” GPUs remain the focal point because they consume so much power, but AI systems are becoming constrained by a growing number of thermally sensitive components.

High-bandwidth memory is critical because memory performance directly affects AI throughput. Voltage regulators influence system stability and power delivery. Meanwhile, optical interconnects are becoming increasingly important as clusters scale, and laser temperature stability directly impacts optical performance and reliability.

As AI infrastructure becomes more tightly integrated, performance is increasingly determined by the weakest thermal link in the system. That is why component-level thermal management across compute, power, memory, and optics is becoming essential.

Q4: Your cooling also reads voltage, current, and temperature at each hotspot and feeds that back through standard APIs. What does that thermal intelligence let an operator do that cooling alone never could?

A: Cooling is only part of the story. When thermal systems become intelligent sensing platforms, operators gain real-time visibility into the health and operating conditions of critical components.

That data can support predictive maintenance, optimize workload placement, identify emerging failure risks, and enable dynamic thermal control strategies. Instead of reacting after performance degrades, operators can make proactive decisions based on actual device conditions.

In AI infrastructure, where uptime and utilization are paramount, thermal intelligence turns cooling from a supporting utility into an operational tool that helps improve reliability, efficiency, and asset management.

Q5: The industry has seen racks climb from 20 kilowatts to 100 and now toward a megawatt. Where does cooling have to go next, and what are you building for it?

A: The next generation of AI infrastructure will require a layered approach to thermal management. As rack power continues to rise, broad system-level cooling alone will not be enough. Operators will need precision thermal control at the component level alongside advanced liquid-cooling architectures.

Phononic is focused on enabling that future with scalable solid state cooling solutions that address the most challenging hotspots in compute, optics, memory, and power electronics. We believe the industry is moving toward cooling systems that are not just more powerful, but also more intelligent, localized, and responsive.

As AI deployments scale, the winners will be the architectures that can maximize compute performance while minimizing energy consumption and infrastructure overhead—and precision thermal management will be a key part of that equation.

Throughout the summer, TechArena has been sitting down with the companies behind the AI buildout to hear how requirements are shifting up and down the stack.

For this installment, we caught up with Paul Quigley, Chief Strategic Relations Officer at Airsys, a company with more than 30 years in mission-critical cooling, built on precision air systems and now expanding into rack-level liquid cooling as AI densities climb. With data center operators running air and liquid side by side, power is becoming the constraint behind nearly every infrastructure decision.

We talked about why air cooling keeps carrying part of the thermal load even in the densest AI environments; what operators run into when they retrofit liquid cooling into facilities that were never designed for this much heat; and how a new metric, Power Compute Effectiveness, measures the share of a facility's provisioned power that is structurally available for compute. Here's what we learned.

Q1: Airsys has spent more than 30 years in mission-critical cooling and its lineup now spans both precision air systems and rack-level liquid cooling with LiquidRack. What are data center operators asking you for today that they weren’t before the AI boom?  

A: The conversation has changed considerably over the past few years. Before the AI boom, customers were primarily concerned with whether their cooling systems could reliably support existing infrastructure. Today, they must plan much further ahead. They are asking how cooling architecture can reduce the amount of provisioned electrical capacity assigned to supporting infrastructure and make more capacity structurally available for AI compute.

Power has become a primary constraint, so every infrastructure decision comes back to how much provisioned capacity can ultimately be allocated to compute. Customers are also looking for flexibility. They know server technology is evolving quickly, and they want cooling infrastructure that can adapt rather than become a bottleneck. Cooling is no longer simply supporting the infrastructure; it has become an integral part of the overall AI strategy.

Q2: Much of the liquid cooling conversation assumes a new build. When an operator raises density in a data center that already exists, what are the specific pain points they run into?

A: Most existing data centers were not designed for today’s AI workloads. As rack density increases, the facility must remove substantially more heat than it was originally designed to handle.

The challenge is that many liquid-cooling solutions assume the facility can be redesigned around them. That can require building-wide piping, large centralized cooling distribution units, and extensive infrastructure changes that add cost, complexity, and disruption. Many operators do not want to take that approach—and some facilities physically cannot accommodate it.

Operators need a practical way to introduce liquid cooling where it is needed, expand it over time, and avoid turning every retrofit into a major construction project. That is why we have focused so much attention on simplifying liquid-cooling retrofits.

Q3: As racks get denser, does air cooling stop making sense?

A: It is not that simple. Air cooling continues to play an important role, and many operators will manage hybrid air-and-liquid environments for years to come.

What AI has changed is the amount of heat each rack must handle. As thermal demands increase, physical limitations begin to determine the appropriate cooling approach. Liquid is more effective at transferring heat away from high-power components, which is why it is becoming the preferred option for the highest-density AI workloads.

Liquid cooling will carry an increasing share of the thermal load and become a centerpiece of many cooling strategies. Air cooling, however, will remain necessary for portions of the IT load and other heat-producing components. Both must therefore be considered as part of a comprehensive cooling strategy.

Q4: Airsys has put forward Power Compute Effectiveness as a new measure. What is it, and what led you to introduce it?

A: Power Compute Effectiveness (PCE) emerged from a fundamental question: within a data center’s provisioned electrical capacity, what proportion is structurally available for IT compute? As AI workloads drive higher power densities and available power becomes increasingly constrained, understanding this allocation is critical for capacity planning and infrastructure design.

PCE is the ratio of provisioned IT compute electrical capacity—after accounting for cooling and heat rejection, electrical conversion and distribution losses, and auxiliary facility loads—to total provisioned facility electrical capacity within a declared facility boundary and redundancy basis. PCE does not replace Power Usage Effectiveness (PUE): PUE measures operational energy efficiency using consumed energy, whereas PCE evaluates how provisioned electrical capacity is structurally allocated at the design and planning level. Together, the metrics provide complementary perspectives on operational efficiency and the capacity available to support compute.

Q5: As AI pushes rack density higher, what should operators understand about how cooling shapes the cost of the compute they deliver?

A: Every AI workload requires electrical power and produces heat that must be removed. The cooling architecture—and the provisioned electrical capacity required to support it—directly affects how much of the facility’s total power envelope remains structurally available for compute.

The first question should not always be, “How do I get more power?” It should also be, “Am I making the best use of the power already provisioned?” Cooling can have a significant impact on that answer. A cooling strategy that requires less provisioned electrical capacity for heat removal can make additional capacity available for AI compute, potentially increasing deployable compute within the existing power envelope without waiting years for new utility infrastructure.

TechArena has spent the summer connecting with companies building AI infrastructure to learn more about the challenges and evolving requirements they face.

For our latest conversation in this series, we had the opportunity to chat with Andy Pernsteiner, Field CTO at VAST Data, a company built on the idea that the data platform matters as much as the compute it feeds. That idea carries new weight now that agentic AI is turning retrieval into a real-time workload.

We talked about where bottlenecks hide along the training and inference pipeline, from WAN ingest and bursty checkpoint writes to the KV cache reads that come with every generated token; what agentic retrieval requires from the storage underneath it; and what changes when storage, database, and AI compute run as one platform instead of a chain of stitched-together systems. Here's what we learned.

Q1. VAST is built on the idea that the data platform, not just the GPU, sets the ceiling on AI performance. Why does data decide how much useful work a GPU actually does?

A: For model builders, GPUs are the most expensive resource in their pipelines. For nearly everyone else, GPUs are the scarcest resource they have, outside of skilled staff. That's why feeding them the right data matters so much.

“Right data” means a few things at once:

• Prepped and clean. Data that hasn't been properly formatted, deduplicated, and cleansed causes the GPU to waste cycles processing input it can't use, or producing output that has to be thrown away and redone. Either way, that's expensive compute spent twice.
• Governed and trustworthy. Data that isn't correctly tagged, secured, and governed to an organization's (or the law's) requirements can produce work that has to be discarded outright, regardless of whether the model output itself was correct. Same waste, different root cause.
• Available on demand. GPUs are only earning their keep when they're actively computing a model or an inference result. Every cycle spent waiting on I/O is a cycle that's gone for good, you can't bank idle GPU time and use it later.

Q2. Training and inference move data through a long pipeline, from ingest and preparation to checkpointing and retrieval. Where along that pipeline do the real bottlenecks show up?

A: There are the obvious bottlenecks, compute, network, and storage performance, and some less obvious ones.

Training

• ‍Ingest: Usually, a bulk load (data acquisition) or continuous scraping (web, social, sensor feeds, etc.). The bottleneck here is typically the network (usually WAN/internet) combined with latency (distance from source to destination) and how much concurrency/parallelism you can bring to bear. Most training data starts life outside the four walls of the data center, so it has to be brought in first. That “tax” doesn't need to be repaid on every subsequent training run, except for new datasets being added. ‍
• Preparation: Typically bottlenecked by compute and/or a lack of parallelism. Frameworks like Dask and Spark are good at parallelizing this work, but it takes real expertise to shard efficiently across large datasets and compute pools. CPU becomes the constraint when prep servers are undersized relative to the GPU fleet they're feeding. Increasingly, GPUs themselves are being used for parts of this step, as the major distributed frameworks add support for offloading preprocessing onto whatever compute is available. ‍
• Checkpointing: Most training runs now use asynchronous checkpointing, which smooths out peaks and reduces burstiness, but there still needs to be enough available bandwidth that a run doesn't stall waiting on a checkpoint write. It's important to separate async checkpoint writes from the synchronous writes that are also part of the training loop, since a flood of bursty checkpoint traffic can starve out sync flows that, by nature, block the job from proceeding.

Inference

• ‍Model load: Not always relevant in a busy environment where popular models are already resident, but time-to-first-token (TTFT) is directly affected by how long it takes to load a large model into GPU memory. Frontier models are large enough now that this requires real network and storage bandwidth over highly optimized, RDMA-capable fabrics. GPU high-bandwidth memory will always outrun the network feeding it, so the job is to remove as much friction as possible from that I/O path. ‍
• Prefill / prompt processing: Largely GPU/compute-bound, but this phase also writes KV cache aggressively as it's computed. In environments using external KV cache (i.e., on external storage rather than GPU memory alone), that requires high bandwidth to the storage layer and a storage subsystem that can keep up, not just with the writes, but with the reads needed to verify the current state of the cache. ‍
• Decode (token generation): Once prefill is done, decode produces output tokens one at a time, and every step reads back the KV cache accumulated so far. Unlike prefill, this phase is memory-bandwidth-bound, not compute-bound: the GPU spends more time waiting on data than doing new math. It gets harder as context windows grow, since KV cache size grows right along with them. It also matters in disaggregated serving architectures, where prefill and decode run on separate GPU pools and the KV cache computed during prefill has to move to the decode pool fast enough that decode isn't left waiting. Time-per-output-token ends up being a function of how quickly that cache can be read and moved, not just raw GPU FLOPs.

Q3. Retrieval, vector search, and RAG have become heavy, real-time workloads as AI moves toward agents. What does that ask of the data layer underneath?

A: Agentic workloads turn retrieval into something closer to a real-time transactional workload than a batch job, and that changes what's required of the data layer:

• KV cache. As context windows grow and agents chain multiple reasoning steps together, KV cache read/write volume grows with them. The data layer needs to serve that cache with low, consistent latency, both writing it as it's generated and reading it back to check state, without becoming the thing an agent is waiting on mid-reasoning.
• Vector retrieval. An agent making a decision often triggers a vector search inline, in the critical path, not as a background task. That means the data layer has to support low-latency similarity search at scale, across potentially billions of vectors.
• Associated datasets. Vector search rarely stands alone, the agent usually needs the structured or unstructured source data the vector points back to (documents, tables, metadata) to actually ground its answer. Ideally, the source data (such as a document or a text chunk) resides close to or on the same system as the embeddings, not only to reduce the complexity of the system, but also to maintain tight lineage between source and derivative, especially in environments where data can change or be deleted frequently.

Q4. VAST brings storage, database, and AI compute into one platform rather than separate systems. What does that unification make possible, and where are its limits?

A:

What it makes possible

• Eliminates the data movement that fragments most AI stacks today. For example, the ETL pipelines that copy data between a storage system, a vector database, a feature store, and a compute cluster just so each system can do its one job. Data is prepped, stored, and retrieved in place.
• Collapses the latency in the retrieval loop described above: vector search, KV cache, and associated datasets can be served from the same platform instead of stitched together across systems with their own network hops.
• Simplifies governance and security to a single policy plane. Access controls, encryption, and lineage are enforced once, at the platform level, instead of being re-implemented (and potentially re-broken) in every downstream system that touches a copy of the data.
• Lets compute run where the data already lives. Embedding generation, preprocessing, or lightweight inference can happen against data in place rather than requiring another export/import cycle.
• Reduces the operational burden: fewer hardware and software systems to purchase, provision, patch, monitor, scale, and reconcile with each other.

Where the limits are

Unification is a platform capability, not a substitute for organizational discipline. VAST provides the platform and policy frameworks, but domain experts and policy owners still need to be directly involved in the design of end-to-end systems to ensure efficient and consistent pipelines.

Organizations with an already-fragmented stack need a practical migration path, not a rip-and-replace. Unification pays off fastest on new workloads and gets adopted incrementally on existing ones.

It doesn't eliminate bottlenecks that live outside the platform's control, such as WAN ingest from external sources or network fabric choices upstream of the cluster.

Q5. In the end, how does the data platform shape the cost and quality of the tokens an operator can deliver?

A: Cost and quality both trace back to the same root cause: whether GPU cycles are spent on useful work.

Eliminating bottlenecks directly lowers cost per token, because idle or stalled GPU time doesn't produce any tokens at all, it's pure overhead on top of very expensive hardware. A platform that keeps GPUs fed converts a higher percentage of the compute you're paying for into actual output.

Governance and lineage shape quality and trust. Knowing what data trained or informed a given model, and being able to prove it, is what lets an operator stand behind the tokens it delivers, particularly in regulated industries or anywhere provenance matters. It also reduces the risk of hallucinated or non-compliant output making it into production.

Cataloging and labeling ensure the right data gets used in the first place, which avoids wasting compute on irrelevant, duplicate, or low-quality data, the single most common cause of both wasted GPU cycles and degraded model output.

Put simply: the data platform doesn't just support the AI infrastructure, it sets the ceiling on how efficiently that infrastructure can turn dollars into good tokens.

A single Arm Neoverse Compute Subsystem (CSS) deployment helped a partner ship a system-on-chip (SoC) in 13 months and shave roughly 80 engineering years off the build. That return is exactly why conversations at this year’s OCP EMEA Summit in Barcelona kept circling back to one question: is the era of monolithic silicon finally giving way to a true chiplet economy, and if so, what holds the new stack together?

I sat down with Dong Wei, Arm Fellow and Lead Standards Architect, and Colin Brix, VP Marketing at AMI, for an episode of The Control Plane podcast sponsored by AMI, where we looked at the rise of the chiplet economy.  

Challenges to Chiplet Adoption

Dong was careful not to declare the monolithic chip dead. The ability to design a single die is not going anywhere, he explained, but AI is straining its limits because CPU, IO, and memory increasingly want different process nodes. That mismatch, combined with cost and performance pressure, is pushing integrators toward chiplet designs. Hyperscalers including AWS, Microsoft, and Google are already there, stitching together verified IP and custom accelerators into bespoke silicon. Arm introduced the AGI CPU last month using the same approach.

Moving from a handful of vertically integrated SoC vendors to an open, multi-vendor chiplet ecosystem requires foundational standards work. Dong compared the road ahead to the long evolution of Peripheral Component Interconnect Express (PCIe), which took decades to deliver true interoperability. Chiplets will be tougher, he said, because they are integrated inside the package using costly manufacturing processes, with no easy plugfest equivalent.

Firmware as the Glue Layer

If standards define the rails, firmware drives the train. “I think I see firmware eventually will become a glue layer to tie these things together,” Dong said. He laid out the layers that need standardization beyond physical and electrical interfaces, including telemetry, discovery, configuration, and control. Arm’s contribution of the Foundational Chiplet System Architecture (FCSA) to OCP is one piece of that picture, and Dong pointed to UCIe and other industry efforts as complementary work.

Colin picked up the thread from the firmware partner’s perspective. AMI has spent 40 years being the quiet orchestrator of the ecosystem, but that is changing. With chiplet designs, security, power, and thermal behavior all have to be designed in from the start rather than bolted on later, which means AMI now listens all the way downstream to the customer of the customer and projects that voice back upstream. “It’s not just the silicon enablement partner of the past,” he noted.  

The shift to multi-vendor packages also creates a reliability problem that the monolithic era did not have to solve. A standardized communication path lets firmware redistribute workloads and isolate misbehaving regions while the system keeps running, he said.

Standards That Free Innovation

The conversation tackled the perennial worry that standardization smothers proprietary edge. The right place to draw the line is between interfaces and implementations. “The standards should be focused on the interfaces rather than the implementations.” This way, vendors can keep what is beneath the interfaces proprietary information, while the framework gives the system a common communication path. He compared this to the standardization of coffee pods: if every espresso capsule were a different shape, you would need a different machine for each one. However, once the capsules are standardized, the machines can compete on other parameters. Colin agreed, adding that in his view, the absence of standards actually slows innovation downstream because vertical applications never get the building blocks they need.

The TechArena Take

Although a lot of progress has been made in the development of chiplets, foundational work needs to happen for the broader market to follow. Both Colin and Dong agreed that open interface standards will decide whether chiplets become a true ecosystem or a playground for a few integrators. Firmware will be the layer that makes reliability and sustainability achievable at the package level. As silicon gets more modular, open standards plus firmware are what keep the whole system working smoothly, and Arm and AMI are positioning themselves as the two ends of that pipeline.

Every new AI data center hits the same wall before it runs out of ambition: power. Operators can order more accelerators, but the grid connection, the cooling budget and the rack density are fixed. That ceiling is quietly rewriting what gets built, and it has turned efficiency from a line on the TCO sheet into the thing that decides how much useful AI work a site can deliver.  

As part of our summer series with the companies building the AI stack ahead of the AI Infra Summit this September, I sat down with Eddie Ramirez, VP of marketing for Arm's Infrastructure Business. Arm has spent years making the case for performance per watt in the data center, and that case lands differently now that power is the hard limit rather than a background concern.  

We talked about why the CPU is becoming the control plane for an entire rack, what open chiplet standards like UCIe and CHI have to hold together as AI silicon turns into an assembly of parts, and how operators can keep pulling useful work out of a data center over a lifespan that will outlast several generations of models. Here's what I learned.

Q1: Arm has built its data center story around performance per watt. As AI data centers run into hard power limits, why does efficiency decide what an operator can actually build?

A: AI infrastructure is increasingly constrained by fixed power, cooling and rack density rather than demand for compute. As AI deployments scale, operators are no longer optimizing individual servers. They're optimizing entire racks, and ultimately entire data centers. Efficiency therefore isn't simply about lowering TCO, but it is about how much useful AI work can be delivered within a fixed power envelope. As AI workloads become more continuous and agentic, maximizing performance at the rack level, not just within a single server, will increasingly define competitive AI infrastructure.

Q2: As AI infrastructure evolves from running individual models to supporting continuous, agentic workloads, the CPU is taking on new responsibilities – from AI execution to orchestrating entire racks of accelerators. How do you see the CPU's role evolving, and where does it make the most sense for AI work to run?

A: AI systems are evolving from executing individual models to coordinating fleets of specialized models that are continuously reasoning, retrieving information, calling tools, managing memory and moving data across large clusters of compute pools. Those continuous system-level tasks naturally belong on the CPU. As AI infrastructure scales, the CPU becomes the control plane for the entire rack, coordinating data movement, scheduling work, feeding accelerators efficiently, and ensuring entire system resources are fully utilized. The result isn't a competition between CPUs and GPUs, but a better-balanced system where each processor is optimized for the work it does best – and work is assigned to the best suited processor.

Q3: Arm’s chiplet strategy leans on open standards like UCIe and CHI to combine pieces from different vendors in a single package. As AI silicon turns into an assembly of chiplets, what needs to hold together for that integration to deliver?

A: Open standards like UCIe and AMBA CHI are essential because they allow compute, AI accelerators, memory, and networking to evolve independently while still working together as a cohesive system. That gives silicon providers the flexibility to innovate faster without sacrificing software portability or ecosystem compatibility. That's also the thinking behind Arm's Foundation Chiplet System Architecture (FCSA), which we contributed to the Open Compute Project to help establish an open, interoperable foundation for next generation chiplet-based AI infrastructure.

Q4: With Compute Subsystems, Arm hands partners pre-validated cores so they can reach production faster and put their own effort into the accelerator that sets them apart. What does that change about who can build competitive AI silicon, and how they build it?

A: Building AI silicon at the leading edge is extraordinarily complex – beyond the digital design, advanced node processes and packaging layer on additional challenges. Everything we can do to simplify our partner’s journey, accelerate the path to silicon and de-risk production holds a great deal of value. With Arm Compute Subsystems, partners receive a production-ready compute foundation that's already validated for performance, software compatibility, and system integration. That allows engineering teams to focus their investment on the capabilities that differentiate them, whether that's AI acceleration, networking, memory architecture, or custom system innovation. The result is faster time to market, lower development risk, and broader innovation across the AI ecosystem.

Q5: As operators look to maximize useful AI work over the lifetime of a data center, what characteristics will matter most in the compute layer?

A: AI models will evolve far faster than the infrastructure that supports them. Data centers are long-term investments that are both capital and labor intensive, so the compute layer must continue delivering value as workloads, models, and software stacks change over time. That means balancing performance, energy efficiency, memory bandwidth, software portability, and ecosystem support at rack scale, not simply optimizing for a single benchmark or today's leading model. The most successful AI platforms will be those that consistently deliver more useful AI work within fixed power and space constraints as AI continues to evolve.

The plot of the 1969 Western calls for James Garner to produce exasperated and amused facial expressions as his character attempts to emigrate to Australia but is frustrated by a lack of funds. The character he portrays, Jason McCollough, is stuck in a nineteenth century frontier town put on the map due to an accidental discovery of gold. Because the cost of living in town inflates hourly and the pay is good, McCollough, a brilliant gunslinger, reluctantly accepts the job as the local sheriff.  

This also happens to be a solid description of most IT and sysadmins I’ve worked with over the years. Quick on the draw when it comes to technical challenges, sysadmins are a jack of all trades who can resolve issues with both ingenuity and a refreshing lack of perfectionism when stabilizing a situation. Also: many sysadmins are not in the job they envisioned when they graduated, must work with the equivalent of the town ‘character’, and have gleeful management who are thrilled someone else deals with the local mess of bad horses, bad men and bad decisions.

One challenge for a sysadmin is that like the sheriff, when things are going smoothly, neither of them is front of mind for day-to-day operations. You only notice them when unwelcome changes occur, such as a core software platform provider gets bought out by an aggressive organization operating like a PE who then squeezes the clientele for more juice. As a sysadmin you have no good choices: costs might suddenly triple, but you built your disaster recovery plan on this technology. You’re “encouraged” to move to new products if you know what's good for you – can you really afford to walk away from the new owner who decided to weaponize their newly held asset against you, their customer?  

Even the low-grade annoyances can flare up out of the blue. Accepting a service providers’ half-hearted developer experiences, inscrutable services catalogs, or byzantine documentation because they offer stability in reliability, identity and service boundaries seems like reasonable tradeoff. Until the service provider starts surprising you with issues in authorization checks and access controls across your VMs, then responds to legitimate customer sysadmin concerns with a statement: “No one has really found that exploit in a real world situation.”

Notice that even mature, decades-old technologies from suppliers who had been stable, if not accommodating, business partners can be a new wrinkle for sysadmins. A new singularity in the form of AI springs new leaks in what had been a relatively steady-state environment. Today, sysadmins have a trifecta of fleet-wide technology changes that are necessary to get the most from AI, bad actors who use AI to launch attacks at a scale human teams can’t match, and a core technology that no one has yet mastered the art of setting or meeting budget projections for. A recent KPMG report states that nearly 1/3 of corporate leaders are finding it difficult to understand the operating costs, much less control them when it comes to AI implemented at enterprise scale. It’s not just financial operations either: there are new potential legal exposures related to copyright ownership, new supplier contract terms and/or unintended regulatory missteps.

For the AI-plagued sysadmin, here are a few shortlist areas for your triage efforts.  

First, protect the areas where the entire company could be put at risk:

• For product development, consider a private AI implementation or open weights model, small language models or industry-specific models you fine tune. This can help ensure your use of AI in product development isn’t a two-way channel that trains your supplier’s AI on how you develop your products as the basis for future competitors. This is especially important if your product happens to be software and is currently protected through copyright.  
• Ensure that any new tools or technology suppliers can meet your insurability requirements through contract negotiations or vendor selection processes
• Employ AI for threat detection including fraud detection while being very deliberate and prescriptive about where AI is allowed to take action to remediate.

Next, use “multi-cloud” principles to ensure company operations can run independent of specific models to avoid vendor API lock-in and build in finOps modeling to anticipate budget “tops and tails."

No CFO has ever been excited to hear sysadmins tell them “I have no idea what budget I need, or whether I can hit the budget within 5% of my estimates”. Like cloud services as they matured, AI tooling prices have experienced wild swings in the last three years, and model retirement has already caused headaches for what were once stable enterprise workflows.  

Finally, look for real-time operational governance capabilities in your AI solutions.

The regulatory landscape for AI is broad, varies greatly between states and nations, and can carry significant penalties for missteps. In the old west, the town charter and the town council were necessary but not sufficient to tackle the opportunities and challenges that came from a sudden influx of gold prospectors. Similarly, AI governance committees are necessary, but not sufficient in a playing field that changes at the speed of compute.

In the end, the role of the sysadmin hasn’t changed nearly as much as the environment around them. Like the reluctant sheriff, they step in not because the job is easy or glamorous, but because someone has to keep order when complexity and chaos collide. As AI reshapes the frontier yet again, the tools may evolve and the stakes may rise, but the mission remains the same: protect the town, steady the systems, and make sure everyone gets through the day without noticing just how close things came to going off the rails. So, this Sysadmin Day, remember—the quiet stability you rely on is no accident; it’s the work of someone who’s always on watch, even when you don’t see them.