Welcome to
the Forum

Executive Summary

As AI infrastructure continues to grow in complexity, the way it’s measured has simplified to the cost and quality of the tokens it produces. Over the summer, TechArena’s editorial team sat down with more than 20 companies building different components of the AI infrastructure stack to talk about the challenges they face and how to make the cost of a token predictable. A pattern surfaced in our discussions: Each layer that solves its own bottleneck hands a harder problem to the layer above, from compute to memory to the network to the interconnect, until the constraint turns physical and the integrated rack becomes the unit of design, with power, cooling, and firmware as partners to the silicon. Operators who run their infrastructure as one integrated machine can forecast their own cost per token with a confidence the stitched-together alternative cannot match. Every seam between vendors and layers hides variance someone has to absorb. Integration removes the uncertainty operators create for themselves, and silicon roadmaps, model efficiency, and demand will keep moving on schedules nobody controls, which is exactly why no operator can afford to stack self-made uncertainty on top. In this era of AI infrastructure, integration has become the quiet requirement underlying all the others.

Opening: The Unit of Value

Every currency needs a mint, and every mint has one job: strike the same coin, at the same weight, at a cost the treasury can predict. The AI economy is building its mints now. Its coin is the token, struck each time a model answers a question, writes a line of code, or completes a step in an agent’s chain of work. And the mints striking it hold thousands of parts from dozens of vendors, similar to a rocket or a jetliner, with a key difference: This machine has to hit its numbers every hour, for years, at a cost promised in advance.

The AI buildout is well underway. By Dario Amodei’s math, the AI industry will stand up 10 to 15 gigawatts of computing capacity this year alone at a cost near $10 billion per gigawatt. Meta is planning a single site, Hyperion, with plans to scale to 5 gigawatts over several years, on a footprint that would cover a significant part of Manhattan. Sundar Pichai said in Alphabet’s Q3 2025 earnings call that Google was “processing over 1.3 quadrillion monthly tokens, more than 20x growth in a year.” Commitments of this size get made years before the first token ships, so every buyer in the market is underwriting a forecast. An operator that cannot forecast its own cost per token hands that uncertainty to whoever finances the buildout, and capital charges for uncertainty.

Neocloud providers already operate by this rule: CoreWeave reported its average customer contract stretching from four years to five as its backlog grew, and long commitments at known economics are what make a gigawatt bankable. Gigawatts and dollars measure the size of the machine. Its nature is harder to hold in one view. An AI data center brings together silicon, memory, storage, network fabric, optics, power trains, and cooling loops, thousands of parts from dozens of vendors that all have to work in concert. The mint’s discipline of same coin, same weight, same cost holds only when every one of those parts does its job. So, every AI budget conversation eventually lands on the same two numbers: what a token costs to produce and whether its quality holds. The question buyers now put to every vendor, every operator, and every architecture has become: Who can make the price and quality of a token predictable over time?

The industry sells itself in superlatives: the biggest cluster, the fastest chip, the largest model. Next to those, predictability sounds like a modest goal. It is the harder one to deliver.

Lynn Comp, TechArena advisor and Intel’s head of global sales and go-to-market, and vice president of its AI Center of Excellence, has watched enterprises cross from experimentation into production, and she hears the question in its rawest form.

“No CFO is happy when they are told ‘I don’t know how much budget I will need for AIOps at scale, nor can I guarantee I can hit that budget’ by their sysadmins and IT architects,” she said.

A proof of concept can tolerate surprise. A production line cannot.

The people who buy the most compute frame the goal the same way. Satya Nadella told investors on Microsoft’s Q1 FY2026 earnings call that the company’s fleet objective is “maximizing tokens per dollar per watt,” a metric that compresses the whole infrastructure stack into one ratio. Sam Altman put the demand side on the record in early 2025: “The cost to use a given level of AI falls about 10x every 12 months, and lower prices lead to much more use.” Those two statements together explain the paradox this report unpacks. Token prices keep falling, and budgets still surprise their owners, since demand, context lengths, and workload mix all grow faster than any one component improves. Falling unit prices make predictability harder to deliver, and the operators who can promise it hold the scarcest product in the market.

Diane Bryant has watched the industry rebuild itself around this kind of prize before. Her career runs from Intel, where she served as CIO and led the data center group, through Google Cloud and board seats including Broadcom. She describes the current buildout as the unwinding of the standard data center she helped create. The industry spent 20 years converging on 19-inch racks, common heights, Ethernet, PCIe, DDR, and one processor architecture. By 2016, she noted, x86 held 99.5% of server CPUs. Homogeneity kept operations simple and markets competitive, and enterprises demanded it, accepting the efficiency any general-purpose design gives up on a specific workload.

AI broke the bargain. Data center demand now concentrates in a handful of companies rather than thousands of enterprises, and those companies own their software stacks, so no processor architecture holds them.  

“With scale, these companies can afford to customize their infrastructure, tuned to run a particular workload,” she said. Her math on why is stark. Working from Anthropic’s public statements on revenue and capacity, she calculates that the company generates “$30B in revenue per 1GW of data center power.” An operator who tunes infrastructure to extract more compute from that gigawatt lifts revenue and cuts operating cost at once.  

“The ROIC is massive,” she said. Custom silicon, custom racks, custom networks, and custom orchestration all follow from that arithmetic.

Bryant rejects the alarm that usually attaches to this story.  

“It used to be Intel that had the power-on infrastructure. It’s shifted to the hyperscalers and AI model players. It’s naive to think this is new,” she said.  

Power over infrastructure has moved before. What matters for everyone downstream is what the new holders of it now require. The operators one tier below the giants feel those requirements without hyperscale budgets to absorb mistakes. David Driggers, CEO and founder of  Cirrascale Cloud Services, lives on the difference between a well-matched deployment and a wasteful one.  

“Inference is forever, as we like to put it, and those costs compound,” he said.  

Training is an event with an end date. Serving tokens is a permanent operating condition, and every inefficiency baked into the infrastructure repeats with every request, forever.

So the stakes are set. Tokens are the unit of value, predictability is the prize, and the customers with the most buying power are rebuilding everything in pursuit of both. What follows is a walk up the stack in the order the industry built it, one silo at a time. Every layer that solves its own bottleneck hands a harder one to the layer above. The constraint starts at the chip.

Part I. The Silo World: Components in Order

1. Compute: The Driver of AI Infrastructure Performance

A token’s cost starts in silicon, and the numbers on accelerator spec sheets no longer predict it. Delivered compute now depends on process node, advanced packaging, memory bandwidth, and interconnect reach, and the gap between peak performance on a slide and sustained throughput in a rack has become the first place token economics go wrong. The interviews for this report converge on a replacement metric from three separate directions: useful work per watt.

Eddie Ramirez, vice president of marketing for Arm’s infrastructure business, starts from the constraint that reframes everything else.  

“AI infrastructure is increasingly constrained by fixed power, cooling, and rack density rather than demand for compute,” he said.  

Once the power envelope is fixed, efficiency stops being a virtue and becomes the whole game.

“Efficiency therefore isn’t simply about lowering TCO, but it is about how much useful AI work can be delivered within a fixed power envelope,” Ramirez added. On that accounting, a chip that wastes watts costs its owner twice, once in the power bill and again in the tokens those watts never produced.

The merchant silicon leaders have adopted the same arithmetic. At GTC 2026, NVIDIA CEO Jensen Huang put it in revenue terms: “AI factory revenues are equal to tokens per watt, so with power constraints every unused watt is revenue lost.” He named the output in the same breath, declaring that “tokens are the new commodity,” and committed the company to a new architecture every year to keep chasing the ratio. When the company selling the most accelerators in the world describes its own product in tokens per watt and revs it annually, the metric has finished its migration from engineering detail to industry scoreboard.  

Qualcomm’s entry into data center inference tests a specific bet: that two decades of thermally constrained phone design taught the company the right instincts for a power-constrained rack. “Designing for efficiency first enables us to optimize the entire rack around cost per token and power efficiency rather than chasing benchmark peaks,” said Tony Pialis, executive vice president and general manager of data center. The phrase to notice is “the entire rack.” Even the silicon vendors have stopped talking about chips in isolation, a shift this report returns to at its fulcrum.

Manuel Botija, vice president of product at Axelera AI, applies the harshest version of the discipline. His chips go where the resources run out.  

“Data center architectures are built for abundant power, abundant budget, and abundant time. None of that is available at the edge,” he said.  

Axelera's answer is digital in-memory computing. Inference runs mostly on matrix multiplication, and the chip performs that math directly inside the memory cell, so the data barely has to move at all.

“A unified architecture that handles both on the same silicon keeps cost per token stable as workloads evolve from vision to generative AI, which is the discipline that will define the platforms operators can build a business on two years from now,” he said. His premise, that moving data now costs more than computing with it, is the deeper truth the next section takes up.

Buyers, meanwhile, have learned that no single accelerator wins everywhere.  

“No single chip serves a 1B-parameter model and a 400B-parameter model economically, so our customers get the accelerator that fits their workload instead of bending their workload to fit whatever one vendor is selling,” Driggers said. “In training, you can run a thousand GPUs and barely notice a network hiccup. In inference, that same hiccup is a failed request in front of a customer.”

Matching workload to silicon is itself a lever on cost per token, and it only works when the rest of the stack can feed whatever chip wins the assignment.

One more gap defines this layer: the distance between the performance a buyer purchases and the performance a rack delivers. Peak throughput on a launch slide assumes a fed accelerator, and every section that follows in this report describes a way accelerators go hungry. Supply adds its own tax, with packaging capacity, HBM allocation, and lead times shaping the performance actually available to buy in any given quarter.

That is where the trouble begins. Push the compute ceiling higher and the chip starves. The constraint moves off the die and into memory.

2. Systemic Performance: Storage and Memory

The memory wall stopped being a research topic and became a line item.  

“Model sizes are growing far faster than memory bandwidth and capacity, creating what the industry increasingly describes as the memory wall,” Pialis said. He carries the point into the power budget. Memory architecture matters, he said, because “data movement increasingly consumes more energy than computation itself.”

Inference made the wall taller. Randy Kreiser, field CTO at Graid Technology, draws the distinction that defines the new constraint.  

“Training is limited by how fast you can process data; inference is limited by how much conversational state you can keep available,” he said. Every active request carries a KV cache, the model’s working memory of a conversation, and that cache grows with context length and concurrent users rather than model size. Agentic workloads compound it, since a long-running agent accumulates state across every step it takes. “The constraint is not the GPU; it is the memory available to hold and reuse state,” Kreiser said. HBM cannot rescue the economics on its own. It arrives soldered to an accelerator, and buying memory by buying compute you do not need is the waste Lynn Comp pointed to in the Opening — the one CFOs refuse to fund.

Storage vendors read the same shift as a change in their job description.  

“The shift is from ‘store it cheaply and reliably’ to ‘keep the GPU from ever waiting,’” said Jeniece Wnorowski, director of content strategy and industry expert programs at Solidigm. “Every idle GPU second caused by a stalled data loader, slow checkpointing operation, storage-network congestion, or memory bottleneck directly increases training costs and degrades economics.”  

The company’s high-density NVMe now plays a role no storage roadmap predicted a decade ago.

“SSDs increasingly act as a memory-expansion tier, storing model weights or KV cache data that can be swapped in and out of GPU memory on demand,” she said. Capacity still matters. It just stopped being the headline.

The economics of failure here are unforgiving, said Andy Pernsteiner, field CTO at VAST Data.

“Every cycle spent waiting on I/O is a cycle that’s gone for good. You can’t bank idle GPU time and use it later,” he said. “For model builders, GPUs are the most expensive resource in their pipelines. For nearly everyone else, GPUs are the scarcest resource they have, outside of skilled staff.”

Agentic workloads deepen the dependency. An agent’s retrieval step rarely ends at a vector search; it usually needs the structured or unstructured source data the vector points back to. That puts the data platform in the serving path of every request.

Turning storage into a memory tier only works if the tier is fast enough, and Kreiser’s benchmark numbers show how narrow the margin is. In a controlled vLLM and LMCache test on a 235 billion parameter mixture-of-experts model across four NVIDIA H200 GPUs, conventional Linux software RAID made things worse, stretching time to first token from 29.4 seconds with no offload to 36.6. Graid’s GPU-accelerated approach, which runs RAID’s parallel parity math on a sliver of an installed GPU, cut it to 9.0 seconds, 3.27 times faster than no offload at all.
‍

“The business case is simple: Fetch must beat recompute. If it does not, the offload tier works against you,” Kreiser said. “Adding protection to an insufficiently fast storage path can make inference worse, not better.”

Graid organizes its agentic storage portfolio by deployment scale, server to rack to platform, aligns the platform tier with NVIDIA’s STX reference architecture, and plans native execution on BlueField-4 DPUs in the second half of 2026. He expects buyers to grade storage on new metrics.  

“Over the next two years, storage will increasingly be evaluated in inference outcomes rather than raw capacity: cost per million tokens served, cache-hit rate, and time to first token,” Kreiser said. “Terabytes remain necessary, but they stop being the headline metric.”

The pattern of the silo walk is now visible. Memory and storage achieved their assignment, keeping the accelerator fed, and the solution multiplied the traffic between nodes. A fed node is fast. A thousand fed nodes have to talk to each other, and the constraint jumps to the fabric.  

3. Systemic Performance: Network

AI clusters fail as teams, and the network is where the teamwork happens. Training and large-scale inference run collective operations that force thousands of accelerators to exchange results and wait for the slowest participant, so one congested link or one mistuned switch taxes every GPU in the job. Tail latency, an afterthought in traditional networking, becomes the governing statistic. Operators respond the expensive way, overprovisioning bandwidth they will rarely use to insure against the percentile that ruins the job. The relevant number is job completion time, and it belongs to the fabric.

The technology contest at this layer has settled into a familiar shape. InfiniBand held the early AI clusters on the strength of its latency discipline, and purpose-built scale-up fabrics hold the frontier. Ethernet absorbs capability from both, and its ecosystem and economics keep widening its share of new deployments. Buyers push the market toward open standards for the same reason they resist single-vendor racks: A fabric commitment outlasts several generations of the chips it connects.

A fabric commitment outlasts several generations of the chips it connects.

Brandon Draeger, chief marketing officer at Cornelis Networks, makes the case for purpose-built over general-purpose.

“The network is not simply moving bits; it is determining how efficiently the entire machine works,” he said.

Draeger puts a number on what a congested fabric costs: A sustained 1% utilization improvement across 10,000 GPUs, at an assumed cost of $4 per GPU-hour, works out to roughly $3.5 million in annualized compute capacity.

“At scale, a small fabric inefficiency becomes a very large infrastructure bill,” he said.  

Marc Austin, CEO and co-founder of Hedgehog, built a company on the observation that most operators cannot spend what hyperscalers spend to get job completion time right.  

“The network defends it or destroys it,” he said. Hedgehog packages the hyperscaler playbook, open networking and cloud-style abstractions, into software an ordinary operator can run.

“It turns the network from a monthslong engineering project into a product you deploy,” he said. “Any one misconfigured switch quietly taxes every job on the cluster.”

Aanchal Sharma, senior director of product management at Astera Labs, prices the same failure in the report’s currency.  

“Put the fastest chip in the world behind a slow, high-latency fabric, and you’ve built an expensive space heater, with GPUs sitting idle waiting on data instead of generating tokens,” she said. Astera’s connectivity silicon spans the three directions the industry now scales in, up within the rack, out across the cluster, and across sites, and Sharma’s point holds at every one of them.  

“Faster chips buy more compute. They don’t buy less waiting,” she said.  

“Tokens per watt and tokens per dollar get decided in the fabric long before anyone reads a chip’s spec sheet,” she added.

The fabric is also where the industry’s standards politics now play out. Bryant catalogs the fragmentation: Scale-up connectivity has split into competing options, with NVLink and UALink on one side and the Ethernet-based SUE and ESUN on the other.

Sharma makes the case for keeping those interfaces open.  

“An open fabric means an operator never has to bet the entire rack on one company’s roadmap,” she said.

The network carries one more assignment in the integrated machine, and it is the one the arc of this report depends on: composability. Infrastructure that gets pooled and reassembled around a workload, rather than fixed at the moment of purchase, needs a fabric that can redraw the machine’s boundaries in software. Austin sees a solved problem waiting to be borrowed.  

“Composability is fundamentally a network abstraction problem, and hyperscalers already showed the answer: open networking plus VPC abstractions,” he said.  

Operators learned in the cloud era to treat compute as fungible. AI infrastructure brings the same lesson one layer down, with the fabric as the instrument.

Every fix at this layer raises the load on the physical medium underneath it. Faster fabrics push more bits through copper that must carry them farther, at a higher density, and on a shrinking power budget. Copper is running out of room, and the constraint drops into the interconnect itself.

4. The Interconnect: Optical and the Limit of Copper

As data rates climb, electrical signaling loses reach, bandwidth density, and power efficiency at the same time, and the industry’s answer is to move light closer to the silicon until it arrives inside the package. Co-packaged optics is the structural version of that answer, and it changes what the interconnect is: a component of the chip rather than a cable between boxes.

Vishal Chandrasekar, director of product management at Ayar Labs, states the goal in system terms.

“They are trying to connect thousands of accelerators so they can operate as a single unified system, with the bandwidth and latency needed to support increasingly large AI models,” he said of the operators driving demand. Copper forces a choice between bandwidth and distance at the moment AI needs both.

“CPO removes that tradeoff by using light to extend high-bandwidth, low-latency connectivity across tens of meters,” he said, citing up to 10 times higher bandwidth, 10 times lower latency, and three to five times better power efficiency than copper and pluggable alternatives. “A successful demonstration matters, but customers also need confidence in reliability, supply, packaging, fiber attachment, thermal performance, and production yield.”

Under the optics sits a materials problem, and Robert Blum, senior vice president of sales and marketing at Lightwave Logic, works at the layer where it gets solved. The materials that carried the industry to 200 gigabits per lane, improved III-V compounds and silicon photonics, are reaching their ceiling.  

“New materials are required for the next modulator generation where 400 Gbps speeds are needed,” Blum said. His company’s electro-optic polymers compete for that generation.

“EO polymers have really improved in performance and reliability, thanks in part to the lessons learned from the OLED industry, and are now ready for deployment,” he said.

Foundries favor the polymers, Blum said, since they integrate into standard silicon photonics processes more easily than lithium niobate.

Tying this back to the token, moving a bit costs energy, and the interconnect moves more bits than any other layer. Optics attacks the joules per bit directly, and every picojoule saved in transit returns to the power budget as compute.

Co-packaging also rewrites the manufacturing contract between the optics and the silicon they serve, and Blum’s description of the change is the integration thesis in miniature. Optical assemblies for pluggable transceivers tolerate standard solder reflow and wire bonding.  

“For CPO, you need to integrate much more tightly with the switch ASIC, CPU, or GPU,” Blum said, which pulls in hybrid bonding, higher processing temperatures, and more complex assemblies. “All this requires much closer collaboration with system integrators.”

A component that once shipped in a box now gets engineered alongside the chip it feeds.

“Optics tends to be more complicated than copper,” he said, with high-fiber-count detachable connectors still a bottleneck and optical engine form factors not yet standardized.

What remains is the least glamorous part of any technology transition.  

“The real race is about ramping production capacity and getting the 400G ecosystem in place,” Blum said. Lightwave Logic has five Fortune Global 500 customers engaged in prototype testing and targets high-volume production in 2027, with the remaining gates spread across foundry process maturity, back-end qualification, and an ecosystem of DSPs, SerDes, connectors, and laser sources that all have to arrive together.

That phrase, arrive together, is the hinge of this report. Optics solves the last movement problem inside and between racks, and in doing so, it removes the excuse every silo had for engineering alone. Once the parts can move data as one machine, someone has to build them as one machine. The problem stops being electrical. It becomes physical.

Part II. The Turn

5. Rack-Scale: The Unit of Compute Becomes the Building Block

Every voice in Part I described a different layer and ended in the same place.

“AI infrastructure is increasingly a systems problem rather than a chip problem,” Qualcomm EVP Pialis said.

The unit of compute is no longer the chip or the server. It is the rack, designed, sold, and bought as a single product, with the row not far behind.

“AI infrastructure performs best when power, cooling, controls, and software are designed as one integrated technology system instead of assembled piece by piece,” said Steven Carlini, chief advocate for AI and data centers at Schneider Electric.

Operators using the catalog model, picking servers from one vendor, power from a second, cooling from a third, and management software from a fourth, could once assume the interfaces between those purchases were forgiving. At AI densities, no interface is forgiving. A megawatt rack punishes every assumption its designers did not share.

Adam Morton, CTO of data center infrastructure at Flex, makes the same case from the manufacturing side. “Designed as a system from the outset, [integrated AI racks] are generally more efficient, cost-effective, and scalable than racks that come together as a collection of components,” he said.  

“Customized data centers have been the industry standard for decades. That era is coming to an end,” Morton said. “‘Snowflake’ projects with bespoke designs, supply chains, permitting paths, and commissioning plans don’t scale, and certainly not at an accelerated pace. … The future AI factory will be designed once and repeated many times.”

Diane Bryant tracks the same turn from the buyer’s side.

“Racks are all now custom built to the dimensions that optimize power and cooling for the custom xPUs,” she said. The 19-inch rack survived every previous platform shift in computing. It did not survive this one. When the companies with the deepest pockets abandon the most durable standard in the data center to win thermal and power headroom, the message to the rest of the market is unambiguous.

The merchant vendors heard it. AMD CEO Lisa Su highlighted the company’s Helios platform during an earnings call with investors, describing it as an integrated rack-scale solution featuring Instinct MI400-series GPUs, Venice EPYC CPUs, and Pensando NICs. Built on Meta’s double-wide Open Rack Wide standard, Helios was designed and “optimized for the performance, power, cooling, and serviceability required for the next generation of AI infrastructure,” Su said. She reported “a lot of interest in the full rack-scale solution.”

NVIDIA sells its flagship as a rack. The neoclouds pour the same logic into concrete: Chase Lochmiller, CEO of Crusoe, calls the data center “the new unit of compute.” Each phrasing moves the boundary of the product outward, from the chip to the rack to the building.

Moving the boundary moves the accountability with it, and the industry has not finished deciding who holds it. A rack that spans one vendor’s silicon, a second vendor’s power train, a third’s cooling loop, and an integrator’s assembly has to answer an old question at a new scale: When the machine underdelivers, whose machine is it? Our interviews suggest the market is answering with engineering rather than contracts. Thermal, power, and management vendors describe co-design relationships that start at the silicon roadmap, years before a purchase order. The alternative is discovering incompatibility at commissioning, when every idle day burns the most expensive depreciation schedule in the industry.

Component vendors now design for that boundary or design themselves out of the market. Randy Kreiser of Graid sees it from the storage layer.

“Once the rack is the unit of deployment, storage cannot be treated as a component to integrate afterward,” he said. Graid organizes its roadmap by deployment scale rather than by SKU for exactly that reason, and some version of that reorganization appears in nearly every interview in this report.

Rack-scale architecture also delivers the promise the cloud made and AI briefly broke: composability. A machine designed as one system can be pooled, partitioned, and reassembled around a workload in software rather than fixed at the moment of purchase. The fabric abstractions that Hedgehog’s Marc Austin described in the network section are the mechanism. The rack designed as a product is the precondition.

“The technology is converging faster than most organizations are,” said Lakecia Gunter, TechArena advisor and former global CTO and corporate board director. “Companies cannot operate integrated AI infrastructure through disconnected technology, facilities, finance, security, and sustainability teams. The operating model must evolve with the architecture.”  

Treating the rack as one product also knocks down a wall inside the operator’s own organization. The teams who run servers and the teams who run power and cooling grew up in different professions, with different tools, different budget lines, and different definitions of an emergency. A rack designed as one machine makes them tenders of one converged system. Facilities engineers now read GPU telemetry, and IT architects sit in utility interconnection meetings. The hardware merged first; the people are catching up. The chip and the facility used to be separate conversations. From here on, they are the same one, and the rest of this report walks the layers those merged teams share. The first is the one gating the entire industry: power.

Part III. The Integrated Machine: Systems

6. Power as a Design Partner

Power used to enter the conversation after the IT was specified. It now opens the conversation, and often ends it. “Today, speed is driven by time to power,” Carlini said, and the sentence explains more of the current market than any chip roadmap. Utility interconnection queues, substation lead times, and grid capacity now gate AI deployment harder than silicon supply. Satya Nadella made the same point from the buyer’s chair.

“You may actually have a bunch of chips sitting in inventory that I can’t plug in. In fact, that is my problem today,” he said. The scarcest input to an AI factory today is the energized shell around the accelerators.

Operators are answering the queue with every tool that shortens it. Some buy their way into existing capacity, and some build behind the meter. The energy-first developers invert the old site-selection logic, putting the data center where the power is and running the fiber to it. Crusoe built its business on that inversion. Whichever way the shell gets energized, the arithmetic that follows is unforgiving: An unpowered rack has no cost per token, since it has no tokens, and a partially powered facility pays full capital cost for partial revenue.

The premium on an energized megawatt now shows up on income statements. Nebius founder and CEO Arkady Volozh told analysts in August 2026 that the company’s pipeline makes it one of just a few players able to build more than a gigawatt of new capacity a year, and the unit economics reported from that call, tens of millions of dollars in annual revenue per megawatt, make each stranded watt a measurable loss. Carlini frames the resulting agenda in exactly those terms.

“The next challenge is not only securing more power; it is reducing the time it takes to bring that power into service and making every available kilowatt work harder to generate tokens and intelligence,” he said.

Scarcity of that order forces discipline on every watt that does arrive, and the discipline starts inside the rack. Steve Thorne, chief commercial officer at CelLink, makes power delivery a peer of the components it feeds.

“Designing at rack scale forces power delivery to be considered as a first-class constraint alongside compute and cooling, rather than an afterthought bolted on at the end,” he said. “At each step down in voltage, current rises sharply and I²R losses compound accordingly.”

CelLink’s flat, flexible harnesses attack those losses and reclaim tray space, and Thorne’s roadmap points at the same convergence Carlini described, power delivery “co-designed and co-assembled with liquid cooling cold plates as a single unified subsystem.”

Flex’s Adam Morton ties the shift to a specific electrical transition. “Within 800 VDC environments, electrical, mechanical, and thermal management decisions are increasingly interdependent, even as power, cooling, and IT equipment gets disaggregated to accommodate more computing capacity in the rack,” he said. “A co-designed approach to the rack, power infrastructure, and cooling system maximizes performance.”

AI also changed the shape of demand, not just its size. Brandon Smith, vice president of global sales and product management at ZincFive, describes a load profile no facility engineer trained for.

“An AI workload can swing from idle to full draw and back in milliseconds, then repeat it thousands of times an hour,” he said. Batteries designed to sit idle until a blackout solve a problem different from the one Smith describes.

“Rather than sitting idle until the grid fails, the system absorbs the spikes and releases energy as the workload calls for it, shaping load in real time before it travels through the facility and out to the grid. Power moves from passive backup to active stabilization,” Smith said. ZincFive’s case for nickel-zinc chemistry rests on that repetitive, high-rate duty cycle.

“A site that smooths its own demand internally reads as a better neighbor, and that can shape how much capacity gets allocated and how quickly a project breaks ground,” he said. In a market gated by time to power, good citizenship at the meter converts into schedule, and schedule converts into tokens.

Every watt the power team wins arrives in the rack with an obligation attached: It all comes back out as heat.

7. Cooling as a Design Partner

Every watt that enters a rack leaves it as heat, and operators now plan for heat with the same care they give power.  

“Cooling is now a foundational design decision for AI infrastructure,” said Rich Whitmore, president and CEO of Motivair by Schneider Electric.  

“As rack power increases, the challenge shifts from simply removing heat to delivering reliable, efficient, and scalable thermal management across an entire facility.”

The industry crossed the density line where air alone stops working, and the liquid systems replacing it cannot be bolted on after the fact. Scale changed the job, too. Whitmore sets the bar for production readiness in operational terms.

“Production environments require more than excellent thermal performance. They require repeatability, uptime, ease of service, and seamless integration with the rest of the infrastructure,” he said.

Paul Quigley, chief strategic relations officer at AIRSYS, connects cooling to the constraint running through this whole report: the power budget. His company has spent three decades in precision air cooling and now builds hybrid systems that add liquid where density demands it, which gives him an unusual vantage on how the two share the load.

“Power has become a primary constraint, so every infrastructure decision comes back to how much provisioned capacity can ultimately be allocated to compute,” he said. “The first question should not always be, ‘How do I get more power?’ It should also be, ‘Am I making the best use of the power already provisioned?’”

AIRSYS proposes measuring exactly that with a planning metric it calls Power Compute Effectiveness, which evaluates how provisioned electrical capacity gets structurally allocated at design time, a complement to PUE’s operational lens. Quigley expects liquid to carry a growing share of the thermal load and air to remain necessary for portions of the IT load, which makes the design question one of proportion.

Ryan Brown, director of data center product management at Phononic, pushes the thermal case down to the component.

“As AI infrastructure becomes more tightly integrated, performance is increasingly determined by the weakest thermal link in the system,” he said.  

The weakest link has moved past the GPU under its cold plate. Optical transceivers, memory, and NICs now throttle systems whose headline silicon is perfectly cooled, and Phononic’s solid-state thermoelectric coolers target those hotspots directly. Precision at the component pays off at the facility.

“By actively and dynamically controlling temperature at the component level, operators can maintain tighter thermal margins on the devices that matter most, even as rack densities continue to climb,” Brown said.

Tighter margins let operators run warmer coolant and cut the cooling headroom they overprovision out of fear.

“Thermal intelligence turns cooling from a supporting utility into an operational tool that helps improve reliability, efficiency, and asset management,” Brown added. A cooling system that senses and reports becomes part of the machine’s telemetry, the subject of the next section. One more wall between operations and IT comes down.

Simon Jesenko, CEO and CFO of Iceotope, closes the spectrum at the whole board. The company’s precision liquid cooling seals the entire server and delivers dielectric fluid exactly where heat appears.

“Cooling the whole server with dielectric fluid allows the operator to cool all heat-generating components within the server: GPUs, CPUs, memory, networking, and PSUs,” he said. His efficiency case rests on first principles. Every heat exchange between fluids gives up energy, so, in his words, “the most efficient system is one which has fewer (or no) heat exchangers.” He also treats the heat itself as inventory rather than exhaust.

“Captured heat can be reused in other deployments. For example, a precision liquid-cooled data center in a hotel basement could use the excess heat to heat the hotel pool,” Jesenko said.  

Two pressures from outside the machine now shape every choice inside it. Buyers apply a sustainability lens with real procurement weight. They ask about water consumption, energy reuse, and the carbon attached to each token, and liquid systems that run warmer coolant, reject heat efficiently, or hand it to someone who can use it perform better on all three counts than air. The installed base pushes back from the other side. Most of the world’s data center square footage was built for air, so operators choose between retrofitting live facilities toward liquid and reserving the densest AI for greenfield builds.

A rack full of power and coolant is still a body without a nervous system. Someone has to make the machine knowable.

8. The Control Plane: Firmware, Telemetry, Software

Integration remains an aspiration until something operates it, and the layer that does the operating spent decades beneath anyone’s notice. Firmware booted the server, reported its health, and stayed out of sight. AI ended that obscurity, said Colin Brix, vice president of marketing at AMI, a Lattice Company. Operators now demand granular telemetry on GPU performance, power draw, thermal behavior, and interconnect health, feeding orchestration systems that act on it automatically.

“Firmware is no longer just reporting health status. It’s becoming the trusted source of telemetry that drives automated decisions around workload placement, power balancing, cooling optimization, and predictive maintenance,” Brix said. “In short, firmware is evolving from a management layer into a data and control layer for the AI factory.”

The stakes of getting that layer wrong scale with the cluster. In a traditional environment, a misconfigured node inconveniences a workload. In an AI factory, Brix said, “a node that is misconfigured, running an inconsistent firmware level, or failing attestation checks can prevent thousands of GPUs from operating at full efficiency.” The boot-to-workload chain, invisible when it works, becomes the foundation of the fleet’s economics. Secure provisioning, firmware integrity validation, and hardware attestation moved from compliance checkboxes to operating requirements.

Automation is only as good as the telemetry beneath it.

“At AI factory scale, telemetry must be accurate, consistent, and synchronized,” Brix noted, since operators now correlate events across thousands of systems at once, and decisions about workload placement, power allocation, and thermal management are only as good as the data feeding them. Heterogeneity works against all three properties. A fleet whose compute, power, cooling, and networking each expose different interfaces and telemetry formats makes automation fragile and lets operational overhead grow with every added vendor.

At fleet scale, the case for unification stops being aesthetic. AMI’s MegaRAC OneTree pulls management of compute, power, cooling, and networking into one open codebase across different silicon, so a patch propagates once instead of dozens of times and every subsystem speaks the same telemetry language. The endgame is a fleet that runs itself.

“The reality is that nobody manually operates a 50,000-node AI factory,” Brix said. “The telemetry and automation architecture must be designed so the fleet effectively manages itself, with humans focusing on policy and optimization rather than individual device administration.”

Predictable cost per token, Brix said, “requires three things: trusted data, automated optimization, and strong security.” The security leg reaches all the way down to power-on.

“The control plane must be built on a common hardware root of trust that validates systems from initial power-on through workload execution,” he said, so an operator knows every node in the fleet runs authorized firmware, trusted software, and verified hardware before it earns a workload.  

The market has started pricing this layer accordingly. Lattice Semiconductor closed its acquisition of AMI on July 27, 2026, in a deal trade press valued at $1.65 billion. A sum that size for a firmware company measures what control of the control plane is now worth. Eddie Ramirez of Arm sees the same promotion happening in silicon, where “the CPU becomes the control plane for the entire rack, coordinating data movement, scheduling work, feeding accelerators efficiently, and ensuring entire system resources are fully utilized.”

A fleet an operator can see is a fleet a customer, a regulator, or a board can ask hard questions about, and those questions are the final layer of the machine.

“Boards should not view AI infrastructure as a technical procurement decision,” Gunter said. “It is a capital allocation, resilience, and competitive-positioning decision that will shape how quickly the company can turn AI ambition into enterprise value.”

9. Privacy and Policy: The Outermost Layer of Trust

The last requirement is the one no rack diagram shows. A token’s price means nothing if its buyer cannot trust where it was made, what data fed it, and who can be held accountable for it. Lynn Comp, whose vantage at Intel spans the enterprises crossing into production, watches trust move from the legal department to the architecture review.

“I’m seeing harder questions being asked about private and hybrid AI as the true costs of frontier models hosted in hyperscaler data centers become more obvious,” she said. Cost and control turn out to be the same question, and buyers have begun answering it with deployment decisions rather than policy memos.

The economics run deeper than compliance, since the data that privacy rules protect is the same data that makes AI worth buying.

“The AI is generic until it applies your business information and context, but it is difficult to see ROI from AI that spends the majority of tokens on the data input processes rather than getting insights from the underlying data,” Comp said. An enterprise’s proprietary data is its entire differentiation in AI. Where that data can safely go becomes an infrastructure requirement rather than a legal afterthought.

Diane Bryant maps the same boundary from the CIO’s chair she once occupied at Intel. In her assessment, an enterprise CIO today should move everything possible to the cloud, since no internal operation competes with hyperscale efficiency, and keep on premises only the workloads with compliance or privacy concerns. The exceptions define the territory. Sovereignty rules, data residency requirements, and sector regulations now shape where tokens get produced as surely as grid capacity does, and they are the reason private and hybrid AI keeps its seat at a table the economics would otherwise clear.

Purpose-built machinery for those exceptions is arriving. Confidential computing has moved from research program to procurement checkbox. Buyers ask whether inference can run without exposing plaintext the model should never see. The hardware root of trust Brix described in the control plane section turns out to be the foundation for every trust claim above it: The same attestation that tells an operator a node runs authorized firmware tells a regulator, a customer, or a court where a token came from and what touched it along the way. Integration pays a dividend here too. A machine designed as one system can answer for itself as one system.

Agentic AI raises the stakes on all of it, and Comp’s recommendations are design constraints rather than aspirations.

“Infrastructure needs to have very tight design constraints for a given AI-based agent, avoiding the ability of an agent to exercise functionality beyond its intended purpose,” she said. “Any decision that would be difficult to defend in a current regulatory framework needs to have a human in the loop.”

Dana Bos, TechArena co-founding advisor and founder and principal of Bos Solutions, sees that gap as the one most AI stack diagrams don’t show.

"Every AI stack diagram stops at the application layer, but the layer that actually determines ROI sits above it: the humans deciding whether to believe the output. Skip that layer and you've built a very expensive system nobody uses correctly," she said.

Trust, in Comp’s telling, is an engineering deliverable with an audit trail, and it pays like one. A token a customer can trust, produced where the rules require, from data that never left its owner’s control, commands a premium over an identical token that cannot show its provenance. Trust is the only layer of the machine the buyer experiences directly, and the whole integrated machine underneath exists to earn it.

This report has traced a single constraint from the chip through memory, fabric, and light, into the physical realities of the rack, and up through power, heat, and telemetry to the question of trust. One question remains, the one this report opened with.

Close: The Requirement

Who can make the price and quality of a token predictable over time?

The operators who run their infrastructure as one integrated machine, and the vendors who design for that machine, hold the pricing power. The ones still stitching a machine together from catalog parts inherit every seam as a variance they cannot forecast. Theirs are mints that strike a different coin every day. The market pays for the mint that strikes the same one.

The evidence assembled here points one direction. Satya Nadella compresses the hyperscaler’s entire strategy into tokens per dollar per watt. Steven Carlini’s power, cooling, controls, and software perform best “designed as one integrated technology system instead of assembled piece by piece.” Aanchal Sharma’s tokens get decided in the fabric, Randy Kreiser’s in the cache tier, Brandon Smith’s in the milliseconds between power spikes, Ryan Brown’s at the hottest component on the board.
‍

“Thinking of agent design like building an appliance helps keep the architecture clean, which then has the byproduct of being more deterministic and predictable,” Comp noted.

An appliance is the humblest thing engineering produces, and the most trusted. It does what it says, every time, at a cost its owner can state. The AI industry, for all its scale and speed, is working its way toward the appliance’s virtue.

The constraint will keep moving. The voices in this report point to where it travels next: trust and energy, the two inputs no engineering team can manufacture alone. The next year offers clean markers for anyone tracking the turn. Watch whether the scale-up fabric standards consolidate or keep splintering, and whether co-packaged optics crosses from prototype to volume on the timelines vendors like Lightwave Logic have staked. Watch how fast the firmware layer consolidates now that Lattice has priced it, and whether time to power shortens or keeps stretching. Each is a proxy for the same underlying question: how quickly the industry finishes becoming one machine.

What will not change is the shape of the winning response, which every section here has traced. Watch the constraint, follow it across the old silo walls, and design the next layer with the last one instead of after it.

Everything this report has argued comes down to a single payoff: the determinacy dividend. It accrues to the operators who treat integration as the requirement underneath all the others, who co-design power with silicon and cooling with both, who can see every node and trust what they see. Those operators will quote a customer a price per token two years out and hit it, and they will collect the capital, the customers, and the trust that follow. In this era of AI infrastructure, integration is the requirement. Everything else is a line item.

PORTLAND, OR – Sept.10, 2026 – TechArena today welcomed data center and AI leader Lynn Comp to TechArena Advisory as an AI governance advisor. Having driven go-to-market strategy and business operations across some of the most consequential product cycles in the history of compute, Comp currently leads Intel's enterprise go-to-market for the company's CPU business and is one of the few leaders to have run both product strategy and the revenue behind it, at Intel and AMD.

Comp has led data center businesses through their hardest transitions, launched first-of-their-kind products, and built go-to-market engines. Her range runs from silicon to services and from cloud to edge, the full sweep of the modern compute stack. Today, as an IAPP certified AI Governance Professional and enterprise GTM leader, she sits at the center of enterprise AI adoption, working with some of the largest organizations in the world as they turn ambition into deployment.

"Lynn brings an invaluable combination of deep product strategy, industry activation, and AI operational governance to TechArena Advisory," said Allyson Klein, founder and CEO of TechArena. "Most conversations about responsible AI stay abstract. Lynn makes them actionable. Beyond her unquestioned acumen and earned experience, she brings incredible ethics to the nuances of governance for organizations. I could not be prouder to welcome her."

Comp's areas of practice include responsible enterprise AI governance. She also leads TechArena Advisory's AI governance workshops, which help leadership teams move from responsible-AI principles to operating practice.

About Lynn Comp

Lynn Comp builds the commercial infrastructure that turns data center technology into market leadership. At Intel, she spent more than a decade in product marketing and data center strategy before becoming Vice President and General Manager of the Visual Cloud data center business, where she drove go-to-market for cloud graphics, CDN platforms, and video delivery infrastructure, and launched the company's first discrete GPU for data centers built on the Xe architecture. She then moved to AMD, running the cloud P&L for the EPYC processor and leading server marketing through a period of industry-reshaping competitive growth. She returned to Intel in 2023 and now leads enterprise and go-to-market for the company's CPU business, working with enterprises in the trenches navigating the AI-driven transformation. Comp holds the IAPP AI Governance Professional certification and serves on the boards of NeuReality and Napatech.

She holds a BS in Electrical Engineering from Virginia Tech and an MBA from the University of Phoenix. At TechArena Advisory, she helps companies align product roadmaps with commercial strategy, govern AI responsibly, and build the go-to-market infrastructure to win in AI and data center markets.

About TechArena

TechArena brings together business advisory, world-class marketing with tech domain expertise, and trusted insights for companies shaping the AI era. The platform covers AI, data centers, semiconductors, cloud-native infrastructure, networking, edge computing, and sustainability.

Yunshui Chen often recommends the book “The Lessons of History,” by Will and Ariel Durant, because it reminds him that technology changes much faster than human nature. As founder and CEO of Airsys, he has spent 31 years testing that idea in the business of keeping computers cool. 

Chen formed Airsys in 1995, when China was building telecom networks faster than conventional cooling could protect the digital equipment going into them. Conventional comfort cooling could not deliver the reliability those demanding environments required. Chen could see that thermal management was about to become a fundamental constraint on the industry’s growth, and he built a company to address it.

Three decades later, Airsys’ mission remains the same, Chen says: “Remove the physical barriers that stand in the way of digital progress.”

In the AI era, the barriers have changed. Today’s workloads require concentrated computing power at densities far beyond those of most telecom facilities, and cooling, electricity and water have become limiting resources.

Chen’s way of looking at the problem has not changed.

“I can think simultaneously as an engineer, an entrepreneur and a student of history,” he says. “It helps me connect technical details with a much longer view of where an industry is going.”

That view is visible in how Airsys is built. The company develops air cooling, single-phase liquid cooling, two-phase technologies and heat-rejection systems, and that breadth, Chen says, means the company “can design around the customer’s complete computing environment rather than force every load into one technology.” That portfolio reflects Chen’s belief that the transition to liquid cooling will take years and will be hybrid rather than binary.

This same long view shapes how he leads. Asked how he wants to leave his mark on the people who work for him, Chen says he wants them “to become more capable, more confident and more independent than when we began working together.”

“A leader’s greatest legacy is not a group of followers, but a new generation of leaders,” he says.

Hybrid, Not Binary 

No one cooling technology is ideal for every application, Chen says. Each has limits. Immersion cooling delivers excellent thermal performance, but servers have to be installed and serviced inside fluid-filled tanks, which changes the data center’s physical layout, its maintenance procedures and its equipment ecosystem. Conventional cold-plate direct-to-chip systems are more familiar to the industry, but they place conductive, water-based fluids close to high-value electronics and require carefully engineered cold plates, manifolds and connections. The resulting systems can carry significant equipment and installation costs, he says.

His picture of the next several years is a facility running all of it at once.

“A data center may have conventional servers, liquid-cooled AI racks, low-temperature support loads and extremely high-density processors operating in the same facility,” he says.

Air handles the lower-density and residual loads. Single-phase liquid can support many current CPUs and GPUs, and two-phase cooling goes where heat flux is most demanding. Each connects to the high-, medium- or low-temperature heat-rejection loop that suits it.

Operators who assemble that mix from several vendors inherit the seams between them. Chen lists the risks that collect at those boundaries: controls integration, water chemistry, hydraulic balance, commissioning, maintenance and performance accountability. Owning every layer lets Airsys take responsibility for the whole system. 

“We are not trying to sell one cooling product everywhere,” he says. “We are trying to optimize the entire infrastructure around the customer’s computing requirements.” 

The Megawatts That Never Reach the Servers 

Chen would rather talk about Power Compute Effectiveness, or PCE, than power usage effectiveness, the industry’s standard PUE metric.

PUE measures the relationship between a data center’s total energy use and the energy consumed by its IT equipment. It does not, by itself, reveal unused provisioned capacity. A facility might have a 100-megawatt utility allocation but still be able to deliver only 60 or 70 megawatts to IT equipment. The remaining capacity may have been reserved and supported by facility investment yet cannot be converted into computing output.

Airsys defines PCE as the usable power available to IT and computing equipment divided by the total power capacity provisioned for the facility. It is a capacity-allocation metric, not an established replacement for PUE.

“It reveals how much of the power that an operator has purchased, financed and built can actually produce computing output,” Chen says.

He sees the two metrics as complementary. PUE measures operating efficiency. PCE measures utilization of the facility’s most constrained and valuable resource. For an existing data center, improving PCE can release substantial capacity without waiting years for a new utility connection or constructing a new facility. That is the version of the problem an entrepreneur sees, and it is the one Chen keeps returning to: how much productive computing the industry can get from every available megawatt.

Beyond the Chiller 

Chen has said publicly that mechanical chillers will eventually be eliminated from data center infrastructure. He treats the prediction as a specification.

Chillers are necessary, he explains, when IT equipment needs coolant colder than the outdoor environment can reliably supply. Eliminating them means moving the entire thermal chain to higher operating temperatures. Servers and cooling components have to be designed and warranted for warmer coolant. Liquid cooling must capture nearly all of the high-temperature heat at its source, so that very little load still needs low-temperature air. Then dry coolers, two-phase systems, heat exchangers and controls get engineered as one integrated high-temperature system. Extreme weather is handled through properly sized heat-rejection equipment, thermal storage or limited trim cooling during exceptional hours.

“Chillers will not disappear from every data center at the same time,” Chen says. “However, I believe mechanical refrigeration can cease to be the default solution for high-density AI facilities. As liquid temperatures rise and heat capture improves, many data centers will be able to reject heat directly to the outdoor environment for most — or eventually all — of the year.”

The same higher temperatures also bear on the question communities are asking about water. Zero onsite water consumption for heat rejection is technically achievable today, Chen says, through dry coolers, air-cooled systems and closed-loop liquid cooling. He is careful with the term. A closed-loop system still needs an initial fluid charge and occasional maintenance. Electricity generation carries its own upstream water footprint. In very hot climates, avoiding evaporative cooling can require more heat-exchanger surface, higher airflow or more energy.

“The warmer the liquid leaving the servers, the easier it becomes to reject that heat directly to outdoor air,” he says. 

Airsys works from the chip outward: capture heat at a higher temperature, minimize low-temperature loads and design the dry-cooling and control systems around those conditions.

“Zero onsite water consumption for heat rejection is not simply a choice of outdoor equipment. It is a complete system architecture,” he says. “Properly designed, it can support large-scale AI development without placing an unsustainable demand on local water resources.”

An Energy Company 

Chen has called Airsys an energy company rather than a cooling company. His reasoning starts with what the customer is buying.

“Customers do not ultimately purchase cooling; they purchase productive computing within a limited power envelope,” he says. “Cooling is one of the places where power is either preserved for computing or consumed as overhead.”

Thinking like an energy company changes the engineering objective. The team considers how every watt moves through the facility, how much becomes useful computing capacity and how much is lost to conversion, cooling and distribution. It also weighs water consumption, equipment life, heat reuse and the embodied resources in any new construction.

The company’s phrase for this is “Balance the Environment.” Chen defines it as balancing the growth of the digital economy against the limits of the electrical grid, the natural environment and the communities hosting these facilities. He insists it be expressed through engineering: higher-temperature cooling, reduced fan and pump energy, water-free heat rejection, longer equipment life and more computing output from existing infrastructure. Sustainability that does not improve the customer’s economics, in his view, does not scale.

“The most durable solutions are those that reduce environmental impact while increasing the return on invested power,” he says.

Made in Woodruff 

In May, Airsys opened the first phase of a 60-acre global headquarters campus in Woodruff, South Carolina. It will house a large-scale 3D-printing operation for cooling components.

Chen says U.S.-based manufacturing brings engineering, production and customer deployment closer together. Customers building critical infrastructure need traceability, consistent quality, secure supply and responsive technical support, along with confidence that their cooling partner can scale with them. Working directly with Airsys engineering and production teams during development, testing and qualification shortens the loop between field performance and product improvement, and it strengthens lifecycle service after installation.

The printers also change what Airsys can design. Complex internal flow paths, heat-transfer geometries and customized components that would be difficult or impossible to machine conventionally can be produced additively, avoiding some of the constraints of conventional machining. That shortens the path from design to prototype, qualification and scaled production.

“The Woodruff campus is more than a factory,” Chen says. “It is a commitment to technological development, manufacturing accountability and long-term participation in the North American data center ecosystem.” 

Inside the Server 

Airsys’ server-level liquid cooling architecture, LiquidPack, delivers dielectric fluid directly to the principal heat sources inside the server through a spray mechanism rather than immersing the whole rack. It captures heat at the chip and preserves much of the familiar rack configuration and service workflow. Properly designed, Chen says, it combines high heat-transfer capability with a relatively small fluid inventory and strong electrical safety.

The tradeoff is precision. Spray distribution, sealing, fluid management and system controls all have to be engineered exactly. He sees particular value in brownfield retrofits, mixed server fleets and facilities that need more computing density without rebuilding their white-space infrastructure.

Two-phase cooling goes further. It uses the latent heat absorbed when a fluid changes from liquid to vapor, allowing the fluid to remove substantial heat at a nearly constant temperature. Compared with single-phase cooling, it supports much higher heat fluxes, reduces temperature differences across the chip and can operate with lower fluid-flow requirements. That can mean smaller components, lower pumping energy and more uniform chip temperatures as processors move toward multi-kilowatt thermal loads. The engineering challenge is greater. Boiling stability, pressure, vapor transport, condensation, material compatibility, fluid containment and long-term serviceability all have to be controlled.

Chen does not see two-phase replacing what came before. He describes the cooling roadmap as an integrated spectrum. Air will keep handling legacy equipment and lower-density components, and single-phase liquid will be the workhorse for much of the market. Two-phase will matter most at the leading edge, where conventional cold plates approach their practical limits. 

Looking Ahead 

Asked what has to be true for AI infrastructure to keep scaling at the pace the industry expects, Chen answers like someone who has watched a buildout before. Cooling technology must become standardized, manufacturable and serviceable at scale, “not merely thermally impressive in a laboratory.”

“Servers, electrical systems and cooling infrastructure must be designed as one system,” he says. “The industry will need higher coolant temperatures, standardized liquid interfaces, clear warranty responsibilities, water-conscious heat rejection and practical upgrade paths for existing data centers. It must also develop the manufacturing and service capacity required to deploy these technologies across thousands of sites.”

Airsys’ role is to connect those layers: air, single-phase liquid, two-phase and heat-rejection technologies, together with system controls, manufacturing and lifecycle support. Chen wants to help operators unlock stranded capacity in existing facilities and design the next generation of high-temperature, zero-water AI infrastructure.

“If we succeed, cooling will no longer be the constraint that determines how quickly AI can grow,” he says.

Technology changes much faster than human nature. That is the theme Chen takes from “The Lessons of History.” Every technology in this story, from the telecom equipment Airsys first protected to the two-phase components it is developing now, has a shelf life. The things that outlast them are human: the need for reliable computing, the limits of the grid and the communities that host the facilities, and the judgment of the people who decide how to work within them. Thirty-one years in, Chen is still building for the slow-moving side of the equation.

Garrett McKibben of Graid Technology joins TechArena's "Data Insights" podcast to unpack KV cache overflow.

For more than four decades, AMI has written the firmware that boots, manages, and secures the world's servers. That kind of longevity in a business long defined by proprietary code makes the company's latest strategic shift notable. On a recent episode of The Control Plane, Allyson Klein, CEO of TechArena, sat down with Zachary Bobroff, VP of product management at AMI, a Lattice Company, to talk about why open source has become central to AMI's firmware strategy, and how the company is untangling what he calls firmware's "five-headed hydra."

As VP of product management, Zachary spends much of his time gathering requirements from AMI's two core customer bases: system manufacturers, including OEMs and ODMs, and the hyperscalers that deploy that hardware at scale. Neoclouds and other large-scale consumers of systems are increasingly entering the conversation too, he said, reshaping how the industry defines its firmware needs.

Historically, OEMs owned platform architecture, Zachary explained. As hyperscalers rose to prominence, influence moved first to ODMs and then to the hyperscalers themselves. Now, with end customers looking to take control of firmware and vertically integrate it, open source has become the best way to reach that goal. In Zachary’s words, "Open source is the best path to innovation, ecosystem adoption, and having everybody on the same page."

OpenBMC and One Tree

AMI has increased its contributions to OpenBMC, the Linux Foundation project that originated at IBM as a lighter-weight alternative to full-featured BMCs. AMI has also built MegaRAC OneTree Community Edition, its own distribution of OpenBMC. "Our OneTree product is OpenBMC at its core, and you can kind of view it as a distribution of OpenBMC. This is a tagged, well-validated version of OpenBMC. We're testing it across multiple silicon vendors, multiple ODM platforms and OEM platforms,” Zachary said.

Customers who need more can still license AMI's proprietary version, which provides long-term direct customer support and SLAs, plus IP packages for advanced needs.

The name OneTree comes from a direct response to a problem Zachary discussed at a recent OpenBMC meetup AMI co-hosted with Meta: the proliferation of firmware forks across silicon vendors and manufacturers, a challenge he calls a five-headed hydra.

Given finite engineering resources, it’s not scalable to continue to work across these multiple trees. “What we've done is take information from those different forks and merge them back into a common tree today,” he said.

Security and Sovereign Data

Security threads through all of it. "We've taken a strong approach to security by putting out advisories to our customers and going through the effort of having third-party audits of our code base," he said, pointing to AMI's annual OCP SAFE audits of its community edition release through the Open Compute Project.

Sovereign data requirements are adding another layer of complexity. As new regional cloud providers emerge to meet local data rules, local players are staring to emerge within these regions, and many are encountering firmware at scale for the first time.

AMI backs its open source strategy with a long-term customer relationship that spans the entire hardware life cycle. "We work pre power on, well before silicon is available, all the way through 10 years to 15 years after launch," Zachary said, positioning AMI as an economy-of-scale partner that spares customers the cost of building their own in-house firmware teams.

For developers who want in, Zachary pointed to AMI's developer portal for MegaRAC OneTree Community Edition, its community portal, upcoming OpenBMC meetups, and the Open Compute Project as entry points. "We were very open to having discussions with customers and industry experts," he added.

Key Takeaway

AMI's open source pivot is a direct response to real fragmentation risk, as multiple forks across silicon paths and rising sovereign requirements threaten to make firmware unmanageable at data center scale. By feeding its OneTree CE distribution back into OpenBMC and layering proprietary IP packages on top where customers need them, AMI is betting that a single, secure, well-audited tree beats a forest of one-off forks. With AI infrastructure evolving from servers to racks to pods to rows to rooms, that bet on convergence looks well timed.

‍

Modern AI infrastructure runs on components pulled from dozens of vendors, and each arrives with its own silicon, its own firmware and its own idea of what security should look like. That patchwork is the problem I set out to unpack on the latest episode of “The Control Plane,” an AMI-sponsored TechArena podcast. I sat down with Alex Williams, founder of The New Stack, and Stefano Righi, security and industry advisor at AMI, a Lattice company. We discussed what’s needed to trust a chip nobody can fully see inside.

A Familiar Fracture

Alex told us that elements of our conversation felt familiar. He spent a decade watching cloud infrastructure mature after founding The New Stack in 2014, when early cloud teams treated workloads as interchangeable since so little real state touched the infrastructure itself. At a recent KubeCon in Atlanta, Alex got an idea of the challenges AI infrastructure presents while sitting in on a session with a senior Uber engineer who described standing up a multicloud GPU environment.

“They had to configure every GPU from every different vendor,” Alex said. “And that became a nightmare for them. And it still is.”

The Chip as a Black Box

Alex drew a direct line to software’s own fragmentation problem three decades earlier. When the 1990s produced a dozen competing versions of Unix, the industry converged on Linux kernel. By 2014, containers ushered in the age of Kubernetes and open source tools such as Cilium, eBPF and Open Policy Agent emerged to manage workloads. All were built on the Linux kernel. No comparable tool ecosystem exists yet for AI hardware, he said. For example, Alex cited “the attestation issue,” one that becomes paramount when a chip does not get verified. The instinct is to blame a hallucination, he said, when the problem could be caught early with early detection of rogue behaviors.

‘Nobody Can Be the Cop of Himself’

Stefano said that modern AI systems assemble components from vendors that each bring their own silicon, firmware, update process and security model, and that variety is what creates the visibility gaps Alex had described. The industry’s identity conversation used to stop at the user logging into a system, Stefano said, but machines now need an identity of their own. His prescription is a genuine root of trust: hardware that carries a cryptographic identity, supports secure boot, and can produce its own measurement and attestation, all resting on a foundation that can be independently verified.

“We used to say that nobody can be the cop of himself,” he said, adding that an open, community-verified standard such as Caliptra, the Open Compute Project-backed silicon root of trust project, is the shared foundation the ecosystem needs.

Firmware Becomes the Control Plane

Alex said that firmware is already the closest thing the industry has to a software control plane, a shift he has watched play out inside the platform engineering teams that emerged once DevOps proved necessary but not sufficient on its own. He said those teams turned their attention to the cloud and to firmware, the layer that orchestrates data movement, manages power and configures interconnects, and that firmware’s reach has extended past traditional x86 and ARM CPUs over the past two or three years. Stefano tied that shift back to AMI’s own history: four decades of boot firmware experience now feed a combined offering spanning the data plane, control plane and security plane, aimed at building the chip-to-cloud chain of trust.

The TechArena Take

Our conversation made the case for treating chip trust as a shared industry problem rather than a single company’s responsibility. The language of cloud native computing is already finding a second life in hardware: Pets vs. cattle, control planes, attestation and identity are concepts platform teams once fought to standardize, and now they have to relearn them one layer down, closer to the metal, where data, Alex said, has been promoted from second-class citizen to first-class citizen with AI infrastructure. Stefano’s argument that no single vendor can police its own hardware reframes chip trust as a problem the industry must solve through open standards, not a feature any single company can attach and sell. The vendors that get there first will be the ones already fluent in both worlds, which is exactly the bet AMI is making with decades of firmware history behind it.

To hear the full conversation, listen to the podcast episode or visit AMI.com.

‍

Investment in AI infrastructure is now measured in the trillions of dollars, and power demand at the largest sites rivals the electricity needs of entire cities. According to Ken Sun, Corporate Vice President AMI, a Lattice company, these numbers indicate why today’s AI buildout is one of the most consequential engineering undertakings in modern history. Ken joined host Allyson Klein on The Control Plane, a TechArena podcast, to talk about why firmware, for long the quietest layer of the data center, has become central to building AI data centers.

Ken came to AMI after roles spanning service providers and cloud infrastructure, and he described the opportunity as a rare moment to help the entire ecosystem, from silicon to OEMs, ODMs, and hyperscalers, collaborate and innovate in new ways. As Corporate Vice President, his focus is go-to-market strategy, ecosystem alignment, and customer success across that effort.

Firmware, the Glue Across Five Layers

The industry commonly describes the modern tech stack in five layers: energy, silicon, cloud, models, and applications. What rarely gets discussed is the layer connecting all five.

“I really see firmware, and platform software to some extent, as critical pieces to help secure, scale, manage and also enable sustainability of your entire AI infrastructure state. Firmware works with all five layers to ensure there is a coherent connection across the layers and across the ecosystem,” said Ken. That positioning also explains why AMI sits across so much of the value chain, working with power and cooling vendors, silicon providers, OEMs, ODMs, hyperscalers, neo-clouds, and open source communities alike.

The Five-Headed Hydra

Ken credited Sanjoy Maity, SVP and Business Unit Leader, at AMI, a Lattice Company, with a phrase he has embraced: the five-headed Hydra.

“You can’t tackle any one of these challenges on its own, it seems like whenever you have control over one of them, another problem arises,” he said, referring to challenges posed by fragmentation, security, power and thermal management, and fleet management. AMI’s answer leans on firmware, transparency, and open standards.

The company is among the leading contributors to the Open Compute Project and to OpenBMC, work Ken sees as double-edged: open source accelerates innovation on a common stack, but can lead to more fragmentation if left unmanaged. AMI’s response is its MegaRAC OneTree Community Edition, an effort to streamline and upstream disparate open source contributions into a more unified base, paired with the services and support customers need across design, deployment, and maintenance.

Lessons From Computex

Ken’s team recently returned from Computex in Taipei, and he pointed to several trends that stood out. He was impressed at seeing the pace of innovation in action, as systems became more modular, more compact, and easier to build, operate, and deploy than just a year earlier. Rack-level boot and management is a growing challenge as operators now boot and manage hundreds of nodes per rack, and power and cooling have become dynamic rather than static, which Ken called one of the toughest parts of the equation the industry is solving for. He also highlighted predictive failure as an emerging goal: managing fleets, including aging ones, with less human intervention through self-awareness and self-healing capabilities. Modular, plug-and-play design at both the hardware and firmware level, he added, is increasingly aimed at accelerating time to market.

Security, Regulation, and the Trusted Partner Model

“Security has shifted from being a design check box to just board level mandate,” Ken said, showing up in every part of the stack, from hardware up through the models. In his view, every component now needs hardware root of trust attestation, with crypto agility increasingly required to prepare for post-quantum threats. Regulation adds another layer of complexity: the EU Cyber Resilience Act extends requirements well beyond secure coding practices into vulnerability management, firmware update processes, documentation, and incident response across a product’s lifecycle.

The customer conversation has shifted as a result. “Rather than debating which technology or which route to trust to pick, customers are asking if they have a trusted partner to work with and not just a supplier,” he said. That distinction is the role AMI wants to play across its 40 years of relationships with silicon partners, ODMs, and operators.

Key takeaway

Ken Sun’s conversation makes a clear case that firmware has moved from background utility to strategic layer. As AI infrastructure investment climbs into the trillions and power needs rival small cities, the five-headed Hydra of fragmentation, security, power and thermal management, and fleet management can no longer be solved piecemeal. AMI’s bet is that customers will find value in their open source contribution paired with services, security IP, and lifecycle support, over point solutions. For data center leaders navigating rising regulatory demands and dynamic power and cooling requirements, the real question Ken poses is not which technology to pick, but which partner can be trusted with the full lifecycle.

‍

MLCommons released results Tuesday for MLPerf Storage v3.0, the industry benchmark that measures how storage systems handle machine learning workloads. Version 3.0 adds two tests aimed at AI inference and opens the suite to S3 object storage for the first time, extending a benchmark that had measured only training and checkpointing.

The update pushes MLPerf Storage past training data delivery and into how storage supports AI systems already serving users. Nineteen organizations submitted 144 performance results this round, including 11 first-time entrants such as Azure, NVIDIA and Nebius. The results also reveal a wide spread in power efficiency among competing systems, evidence that AI storage architecture remains unsettled even as adoption grows.

New Tests Target AI Inference

Version 3.0 adds a KV cache test, which measures how storage systems handle the read/write operations behind LLM inference. KV caching lets a model reuse key-value vectors it already computed instead of recalculating them on every conversation turn, a common technique in transformer-based AI inference. The benchmark simulates multi-turn conversations that write a context once and read it repeatedly, with transfers ranging from 64 MiB to about 3 GiB and a median workload near 24,561 contexts totaling 13 TiB of data.

The suite also adds a vector database test, which measures performance for the indexing and query workloads behind RAG pipelines. The test uses the Milvus database with 1 million vectors at 1,536 dimensions and a DiskANN index, generating a stream of small, random read-only queries whose accuracy is checked against brute-force ground truth.

“These new additions to the benchmark suite round out the test collection, covering a larger range of AI inference workloads that drive storage needs,” said Brian Belgodere, MLPerf Storage working group co-chair.

He added: “Including tests that decompose monolithic AI systems and focus on specific storage uses and patterns, such as checkpointing, KV caching and vector databases, gives stakeholders a much clearer idea of how to engineer and provision AI systems to minimize storage performance bottlenecks.”

David Kanter, founder of MLCommons and the head of MLPerf, said the KV cache test differs from the suite's earlier training and checkpointing tests in one respect. Those benchmarks were derived from MLPerf's own industry-standard training workloads, while KV cache runs on an emulation the working group built itself, since no comparable industry-standard agentic workload existed in time for this release. He said the working group wants to align future KV cache rounds with MLPerf's own agentic inference benchmark once that work matures, replacing the purpose-built emulation with a workload derived from a documented standard.

Object Storage Joins the Comparison

Version 3.0 also adds support for S3 object storage as an access layer alongside the existing POSIX file system standard, letting submitters run training and checkpointing workloads against object storage instead of through a file system. About one-sixth of this round's submissions used the S3 layer.

“As the scale of AI contexts reaches into the trillions, we expect object-based storage systems to emerge as a viable, and possibly preferred, alternative to filesystem-based storage,” said Curtis Anderson, working group co-chair. “By enabling S3 support now, we are ensuring that stakeholders will have the performance information they need to make smart decisions.”

In a news briefing, Anderson put that scale in concrete terms. “If you look at the math for a KV cache environment, a billion iPhones or 5 million iPhones, every iPhone user has 1,000 contexts. Now you’re talking trillions of contexts that need to be stored. That’s an object problem, not a file system problem.”

Submitters including Nebius, NVIDIA and OpenLake used S3-compatible object storage for training and checkpointing workloads in this round.

Measuring Power Efficiency

The results also gave MLCommons its first broad look at power efficiency across submitted systems. On-premises submissions for the checkpointing write test posted a median of 14 GB/second per watt, with the top result reaching 201. The UNet3D read test showed a median of 34 GB/second per watt and a top result of 277.

“There is a wide range of power efficiencies represented in the results,” Anderson said. “It also shows that there is ample room for further improvement, and we encourage all suppliers to optimize for that metric.”

The working group also reframed how it wants benchmark customers to read the numbers. Rather than raw bandwidth, the training benchmark scores how many accelerators a system can keep above 90% utilization. Checkpointing scores duration. KV cache scores the number of conversations a system supports. Anderson said the comparisons that matter to data centers are performance per rack unit and performance per watt of provisioned power, since a working data center's space and power budgets are the constraints operators cannot expand on demand.

The TechArena Take

This round marks an expansion of what MLPerf Storage measures, from a single moment in a model's life to its full arc. Training and checkpointing belong in a model’s build phase. KV cache and vector database capture how it performs once deployed and serving real conversations and queries. That shift to covering a model's full life cycle tracks where AI investment is going as more organizations move models into production.

The economics are shifting to match. Anderson's accelerator-hours framing, storage judged by GPU time saved as well as dollars per terabyte, reflects a broader repricing of infrastructure happening across the AI stack, not just storage. Object storage's arrival alongside parallel file systems tells a similar story: As AI context volumes climb toward the trillions, storage architectures built for a smaller era are being tested by systems built for hyperscale conversational AI.

As AI infrastructure keeps changing shape, from training clusters to inference fleets to the agentic systems MLCommons is only beginning to benchmark, MLPerf Storage remains one of the few places buyers can compare vendors on equal footing.

Engineering teams building with generative AI are facing a challenge: How do you move faster while making sure customers can trust your technology? Recently, Solidigm’s Jeniece Wnorowski and I sat down with Priya Sawant, senior vice president of engineering at ASAPP, to talk through that tension. ASAPP builds the chat and voice AI agents that power contact centers for large enterprises, and Priya’s teams sit at the center of a shift redefining how engineers work and what they build. ‍

Speed Without Sacrificing Trust

Priya described the change as twofold: Her engineers now write code with a different set of tools, and the products they ship look different, too. Generative AI is probabilistic by nature, she said, and you have to make sure that your customers believe that your technology is doing exactly what you’re saying it’s doing. Building evaluation outputs and visibility into agent behavior became a priority for her teams. Speed matters at ASAPP, she said, but not at the cost of the hygiene that earned the company its customers in the first place.

Golden Pathways’ to Adoption

Priya said her platform teams design tools that make engineers’ jobs easier, creating “golden pathways that everyone else can follow.”  

She breaks the discipline into three phases:

1. ‍Design: Teams interview internal users to understand their problems and craft solutions to address them. ‍
2. Build: Teams pull out the backlog item most needed now. ‍
3. Adoption: A surge of early interest settles into steady, lasting use.  

A platform built without attention to developer experience creates friction, Priya said, and friction kills adoption.  

Look for the Common Denominator

Priya said that when addressing internal engineering problems, you have to identify “the common denominator of the challenges that you want to solve for” and provide flexibility on top of that. Security and reliability, for example, stay fixed, while other components are more pliable. She pointed out that bottom-up adoption, spread through engineer-champions who find real value early and convey that enthusiasm to their peers, works better than top-down mandates. ASAPP borrows a page from open-source communities here, running user groups where engineers weigh in on different tools and the build plan.  

The Expanding Surface Area

Platform teams still do what Priya calls “glue work”: stitching disparate systems into a coherent experience for internal customers, and AI tools are giving these teams the same productivity boost that product engineers already enjoy. But with this boost comes more complexity: The surface area that platform teams need to cover has grown fast, spanning new models, inference platforms, frameworks and experimentation tools that shift by the week. ASAPP’s own tooling shows that shift in miniature. Engineers used to build language-based frameworks that baked in the company’s best practices, making it easier to spin up a new service. Now the team builds a base agent framework so that anyone plugging a new agent into ASAPP’s platform starts from a shared foundation rather than building one from scratch. That same instinct shows up in the product itself, as an observability suite letting enterprise customers verify that their deployed agents hold to business policy, paired with what Priya calls “the agent flywheel”: a system that mines usage data to surface the next automation opportunity. Human-in-the-loop design rounds out the approach, keeping a person in the workflow wherever regulation or customer preference calls for one.

The TechArena Take

Priya’s conversation offers a lesson for engineering leaders navigating the AI transition. Successful teams will build evaluation, observability and developer experience into the product at the start instead of patching them on after something breaks. Underneath all of it sits a storage and compute layer that has to keep pace with what these agents demand. ASAPP’s golden-pathway approach to platform engineering, paired with attention to what internal engineers need, offers a working model for enterprises that want to move fast on AI without losing the trust that took years to build.

To hear the full conversation, listen to the podcast episode or visit asapp.com.

Global data center power demand is projected to rise 27% in 2026 to 132 GW, en route to roughly 290 GW by 2030, per recent industry forecasts.  

In the latest edition of The Control Plane, guest Zane Ball, chief technology officer of the Open Compute Project Foundation (OCP), joined hosts Allyson Klein, CEO of TechArena, and Colin Brix, Vice President of Marketing at AMI, a Lattice company, to discuss the growth of open hardware and what Zane calls the upcoming “golden age of firmware.”

Zane spent almost 30 years at Intel, most recently leading its data center and AI engineering team, before retiring and joining OCP. OCP was founded 15 years ago after Facebook and other hyperscalers started designing their own servers and going directly to Taiwan’s ODM ecosystem for manufacturing. What began as a gathering of a few hundred people has grown into a community of more than 500 member companies collaborating across over 150 technical projects, grown to more than 10,000 attendees at OCP’s most recent global summit.

AI and Open Hardware

The conversation centered on why open hardware matters more, not less, as AI scales. Zane explained that AI’s tightly coupled nature, where chip design, model design, software, the data center, and the power grid all influence one another, creates real pressure toward proprietary, vertically integrated stacks. That approach pays off short term, he said, but grows costly once an operator needs to adapt to a different geography, climate, or energy profile.  

As Zane put it, “Fragility is the reason proprietary solutions can be challenged.” Finding the right interface points is what allows competition and innovation to flourish both above and below them, he added.

That thinking carries directly into what Zane sees as a defining shift for the industry: firmware’s rising importance. He said the industry is entering “a golden age of firmware,” driven by AI data centers that bring together battery energy storage systems, low-voltage DC power, liquid cooling units, and IT racks, all of which need to read the same telemetry and act in concert.  

A major benefit of interoperable systems is improved reliability. “AI systems are not super reliable. With cloud systems, you can isolate every piece of the machine from every other piece. AI systems aren't like that. If one GPU goes down over here, it affects everything else. If you have better manageability, better firmware, and better collaboration across all these systems, guess what? You can build more reliable systems,” said Zane.

Power grid opportunity

According to Zane, citing studies from the Duke University Nicholas Institute, if the power grid could curtail demand for just half a percent of the year, roughly 44 hours, close to 100 gigawatts of capacity that already exists in North America would become available today. This statistic illustrates the other big opportunity for open source. If data centers could flex their consumption during rare grid stress events, the industry could unlock enormous latent capacity without new infrastructure.  

Colin brought the firmware vendor’s perspective to the discussion. AMI has spent 40 years managing the disaggregated hardware that makes platforms work, and the company is stepping out of a purely behind-the-scenes role.  

“We’ve been this quiet player that just makes things work,” he said, adding that AMI wants to bring the voice of the end customer upstream, so reliability and security get designed in early rather than addressed after deployment.

Asked whether operators will start demanding open source firmware in their RFPs, Zane said many already do. Most cloud companies use open BMCs rather than proprietary manageability firmware, largely because they want visibility into the code running in their own data centers.  

Not every layer needs to be open, he added. A BIOS configuring registers close to the silicon can reasonably stay proprietary, while the bigger opportunity lies in the layer above it, where firmware delivers security and manageability features across the whole data center. Taking inspiration from Robert Frost’s famous line ‘good fences make good neighbours’ which his old colleague Jim Keller often quoted, Zane said the goal is to build clear interfaces that let each layer of the stack innovate independently.

Key takeaway

Zane’s conversation makes a clear case for open innovation in firmware. The bigger opportunity extends past the data center walls: if operators, chipmakers, and firmware vendors like AMI agree on the right open interfaces, the industry can unlock existing grid capacity, improve reliability, and avoid the fragility of fully proprietary stacks.

With AI deployments fundamentally changing the data center infrastructure conversation, firmware has become the trusted source of telemetry driving automated decisions across workload placement, power balancing, cooling optimization, and predictive maintenance.

For our last 5 Fast Facts Q&A in our summer series on AI infrastructure requirements, we sat down with Colin Brix, vice president of marketing at AMI, a Lattice Company, to discuss the critical role firmware plays in managing the components of the AI stack and AMI's emphasis on an open, unified codebase. Here's what we learned.

Q1: AMI's firmware runs from the BIOS and BMC on a single server up to fleet-level management across a data center. What are operators asking that firmware to do today that they weren't a year ago?

A: A year ago, operators were primarily focused on server health, uptime, and traditional infrastructure monitoring. Today, AI deployments have fundamentally changed the conversation.

Operators are demanding much deeper visibility into the resources that directly impact token production and infrastructure efficiency. That means granular telemetry around GPU performance, accelerator utilization, power consumption, thermal behavior, interconnect performance, and rack-level power dynamics. They are also asking firmware to provide increasingly sophisticated power management capabilities, allowing them to optimize performance-per-watt without sacrificing workload throughput.

Just as importantly, operators are looking for real-time data that can feed higher-level AI factory orchestration systems. Firmware is no longer just reporting health status. It's becoming the trusted source of telemetry that drives automated decisions around workload placement, power balancing, cooling optimization, and predictive maintenance.

In short, firmware is evolving from a management layer into a data and control layer for the AI factory.

Q2: The boot-to-workload chain is invisible when it works and everything when it doesn't. Where in that chain do AI deployments most often run into trouble?

A: The biggest challenges emerge at the intersection between rapidly evolving hardware and increasingly complex software stacks.

AI infrastructure combines CPUs, GPUs, accelerators, networking fabrics, storage, power systems, and orchestration software that often come from multiple vendors. Any mismatch in firmware versions, configuration settings, security policies, device initialization, or hardware inventory can create failures that are difficult to diagnose and expensive to resolve.

What makes AI environments unique is that issues that might only affect a single server in a traditional environment can impact an entire training cluster. A node that is misconfigured, running an inconsistent firmware level, or failing attestation checks can prevent thousands of GPUs from operating at full efficiency.

This is why operators place such a premium on consistency and trust throughout the boot chain. Secure provisioning, firmware integrity validation, hardware attestation, and automated fleet-wide lifecycle management have become critical because the cost of a single misbehaving node is dramatically higher in an AI environment.

From AMI's perspective, success comes from creating a trusted and repeatable path from power-on to productive workload execution across the entire fleet.

Q3: MegaRAC OneTree pulls management of compute, power, cooling, and networking into one open codebase across different silicon. What changes for an operator when those sit in one place instead of separate tools?

A: MegaRAC OneTree's unified codebase enables operators to focus on maximizing the token output of their AI factory instead of spending time maintaining dozens of independent management stacks.

With separate management domains, every platform, subsystem, and vendor often comes with its own codebase, update cycle, security process, telemetry model, and operational workflow. That creates tremendous operational complexity and introduces risk every time an update or security patch must be deployed.

A unified codebase changes that equation. When a vulnerability is discovered or a feature enhancement is required, it can be addressed once and propagated consistently across the heterogeneous fleet. Operators gain a common operational model regardless of whether they're managing compute nodes, accelerators, power infrastructure, cooling systems, or networking equipment.

Equally important, every component speaks the same language. Telemetry becomes normalized, automation becomes simpler, and fleet-wide optimization becomes practical. By creating uniformity across a heterogeneous AI factory, OneTree removes friction from daily operations and allows teams to focus on efficiency, performance, and scale rather than infrastructure complexity.

Q4: As fleets scale into AI factories, what has to be true about telemetry and automation for one team to actually run tens of thousands of nodes?

A: At AI factory scale, telemetry must be accurate, consistent, and synchronized.

Decisions about workload placement, power allocation, thermal management, and capacity planning are only as good as the data feeding those decisions. Accurate telemetry combined with precise timestamps is essential because operators are increasingly correlating events across thousands of systems simultaneously.

Beyond accuracy, standardization and unification become absolute requirements. AI factories are inherently heterogeneous environments containing compute platforms, accelerators, networking fabrics, power infrastructure, cooling systems, and storage resources. If each subsystem exposes different management interfaces and telemetry formats, automation becomes fragile and operational overhead grows exponentially.

Successful operators require systems that expose common telemetry models, common APIs, and common lifecycle management processes. Only then can automation safely aggregate fleet-wide data, identify anomalies, trigger remediation actions, and maintain optimal performance at scale.

The reality is that nobody manually operates a 50,000-node AI factory. The telemetry and automation architecture must be designed so the fleet effectively manages itself, with humans focusing on policy and optimization rather than individual device administration.

Q5: What does an operator need from the control plane to be able to promise a predictable and trustworthy cost per token?

A: A predictable cost per token begins with complete visibility and control across the entire AI factory.

Operators must continuously optimize compute utilization, networking efficiency, power consumption, and cooling performance on a second-by-second basis. Any blind spot or inconsistency directly affects infrastructure efficiency and drives up token costs.

To achieve this, operators need a control plane that is fast, reliable, and unified. They need a single operational framework that provides trusted telemetry, consistent automation, and coordinated management across all infrastructure domains. Their engineering teams should spend their time optimizing AI production rather than reconciling conflicting data sources or maintaining multiple management stacks.

Trust is equally critical. The control plane must be built on a common hardware root of trust that validates systems from initial power-on through workload execution. Operators need assurance that every system in the fleet is running authorized firmware, trusted software, and verified hardware.

Ultimately, predictable cost per token requires three things: trusted data, automated optimization, and strong security. AMI's role is to provide the foundational management infrastructure that enables all three at AI factory scale.

As AI inference workloads shift toward agentic AI, memory capacity increasingly determines the cost to serve them.

This summer, TechArena has been asking the companies that build AI infrastructure how their requirements are changing as deployments scale. We sat down with Randy Kreiser, Field CTO at Graid Technology, which has developed a portfolio of storage performance and resilience products. Using GPU-accelerated RAID for KV cache offloading, he said, makes time to first token (TTFT) roughly three times faster than running with no cache offload at all.

We also discussed what a storage tier has to deliver to keep pace with the GPU and where storage fits into the industry's shift toward rack-scale AI systems designed as a whole. Here's what we learned.

Q1: AI infrastructure conversations used to start and end with FLOPS, but inference has moved the constraint to memory. From your vantage point, how did memory become the gating factor for inference economics, and why can't GPU memory alone solve it?

A: Training is limited by how fast you can process data; inference is limited by how much conversational state you can keep available. Every active request carries a KV cache, the model's working memory of the conversation, and cache demand grows with context length and concurrent users, not simply with model size.

That matters especially for agentic AI. A long-running multistep agent can create substantial cache demand even when GPU compute is not the limiting resource. The constraint is not the GPU; it is the memory available to hold and reuse state.

GPU memory alone cannot solve the problem economically. HBM comes attached to an accelerator, so adding memory often means buying compute you do not need. When cache capacity is exhausted, the system evicts state and recomputes it later. GPU utilization may still look healthy, but useful work declines because the system is regenerating context it already processed.

Q2: KV cache offloading essentially turns NVMe storage into an extension of the memory hierarchy. What has to be true of the storage layer for that tiering to work at inference speed rather than becoming the next bottleneck?

The cache must be faster to retrieve than it is to recompute. If it is not, offloading simply adds latency to the inference path.

A: The threshold is higher than many teams expect. In a controlled vLLM and LMCache benchmark using a 235 billion-parameter MoE model across four NVIDIA H200 GPUs, Linux MD RAID5 increased TTFT from 29.4 seconds with no offload to 36.6 seconds. Adding protection to an insufficiently fast storage path can make inference worse, not better.

For NVMe to serve as an effective inference-memory tier, it needs:

• A direct path into GPU memory, avoiding unnecessary host-DRAM copies.
• Predictable tail latency, because inference SLAs are governed by p99 behavior, not average bandwidth.
• Sufficient capacity in a unified namespace, avoiding manual tier management.
• Resilience without a runtime performance penalty; a rebuild cannot cut production throughput in half.

Miss any one of these, and the storage tier becomes the next bottleneck. Graid Technology's innovative volume management capabilities deliver all of these.

Q3: Traditional RAID architectures were built for a CPU-centric world and burn the very cycles and PCIe lanes AI servers can't spare. What convinced Graid Technology that RAID logic belonged on the GPU, and what does that unlock in an AI data center that a hardware RAID card or software RAID can't?

A: RAID is fundamentally parallel parity math: XOR and Galois-field operations. Traditional architectures placed that work on a RAID-controller ASIC because CPUs were poorly suited to it and there was no better parallel processor in the server.

That is no longer true. Modern AI servers already contain massively parallel GPUs, while NVMe has exposed the limits of conventional RAID designs. A hardware controller can become a bandwidth ceiling as Gen5 NVMe arrays scale up. Software RAID avoids that controller bottleneck but consumes host CPU cores and PCIe resources that AI workloads need for feeding accelerators.

GPU-accelerated RAID changes the tradeoff: SupremeRAID AE uses a small portion (approximately 4% or 6 SMs) of an installed GPU rather than a dedicated RAID card, preserves host CPU resources, and enables a direct storage-to-GPU-memory path. The result is protected, high-throughput storage designed around the AI server rather than bolted onto it.

Q4: You've positioned SupremeRAID AE around KV cache offloading, citing roughly 3x improvement in TTFT. Walk us through how GPU-accelerated storage changes the inference pipeline in practice. Where do those gains actually come from?

A: Correct, we measured 3.26x faster TTFT than no offload and 4x faster than Linux MD RAID5, reducing mean TTFT from 29.4 seconds to 9.0 seconds.

The gain is not merely faster storage. It comes from avoiding unnecessary GPU computation. TTFT is heavily influenced by prefill, the work required to process the full prior context before the model can generate its first token. In long-context agentic workflows, that prefill work is expensive and often repeated.

When the KV cache already exists, the system can retrieve it instead of recomputing it. A sufficiently fast storage read is far cheaper than rerunning attention across a large context window. That releases GPU cycles for decoding and serving additional requests. The business case is simple: Fetch must beat recompute. If it does not, the offload tier works against you. SupremeRAID is proven to be faster than recompute.

Q5: The industry is moving from components bought off spec sheets to rack-scale systems designed as a whole. As storage becomes a design partner to the GPU rather than a peripheral, how does Graid Technology's roadmap fit into that integration story, and what does storage look like in the AI data center two years out?

A: Our Agentic AI Storage Portfolio is organized by deployment scale, KV Cache Server, KV Cache Rack and KV Cache Platform, rather than by individual SKUs. That reflects a shift in AI infrastructure design: Once the rack is the unit of deployment, storage cannot be treated as a component to integrate afterward.

KV Cache Platform aligns with NVIDIA's STX reference architecture. The roadmap includes native BlueField-4 DPU execution in H2 2026 and expanded drive-count support, allowing one SupremeRAID instance to span multiple CMX chassis and present a virtualized pool to an entire rack of STX nodes.

Over the next two years, storage will increasingly be evaluated in inference outcomes rather than raw capacity: cost per million tokens served, cache-hit rate, and TTFT. Terabytes remain necessary, but they stop being the headline metric. RAID and I/O processing will follow the available parallel compute, from GPUs today to DPUs in the next phase.

Co-packaged optics change manufacturing and integration requirements for electro-optic polymers, said Robert Blum, senior vice president of sales and marketing at Lightwave Logic, which develops EO polymers. That means much closer integration with the switch ASIC, CPU or GPU, he said, along with hybrid bonding processes that run at higher temperatures than optical assemblies for pluggable transceivers.

This summer, TechArena has been asking the companies that build AI infrastructure how their requirements are changing as deployments scale. We sat down with Robert, who discussed where interconnect materials science needs to go next, what separates EO polymers from competing approaches, and what's standing in the way of faster industry-wide adoption of optical solutions over copper. Here's what we learned.

Q1: Copper, pluggables and now co-packaged optics have each marked a step change in how data moves inside the data center. Where does interconnect materials science need to go next to keep pace with AI bandwidth demand?

A: There has been tremendous progress in both the materials used for lasers and modulators. III-V materials have improved, enabling higher-power lasers and electro-absorption modulators up to 200 Gb/s per lane. Silicon photonics has improved, achieving 200 Gb/s with micro-ring modulators and highly doped p-n junctions.  

But new materials are required for the next modulator generation where 400 Gb/s speeds are needed. That's where thin-film lithium niobate and electro-optic polymers come into play. We see strong momentum behind EO polymers, because they can be easily integrated into standard silicon photonics foundry processes.

Q2: Electro-optic polymers are one of several competing approaches to improving modulator performance, alongside indium phosphide and lithium niobate. What does the industry still get wrong about how these materials compare?

A: EO polymers have really improved in performance and reliability, thanks in part to the lessons learned from the OLED industry, and are now ready for deployment. They integrate much more easily with silicon photonics than lithium niobate, so the foundries love these materials. And customers like them because they can enable much more compact modulators with lower drive voltages than lithium niobate. The real race is about ramping production capacity and getting the 400G ecosystem in place.

Q3: Given copper's physical limitations in the interconnect layer as data rates climb, what is the bottleneck to faster adoption of optical solutions industry-wide?

A: Optics tends to be more complicated than copper. High-fiber-count detachable optical connectors are one of the bottlenecks. Form factors for optical engines are not standardized, so many solutions are proprietary or require custom designs and packages.

Silicon photonics foundries have ramped capacity at astonishing rates, but lead times can still be quite long. On the other hand, there is a large payback for going to optical at these higher data rates, so many suppliers are eager to make the transition.

Q4: Lightwave Logic is developing electro-optic polymers that can be used for co-packaged optics (CPO) in AI and hyperscale data centers. What does co-packaging change about the requirements for your polymer, versus a pluggable transceiver?

A: Optical assemblies that go into pluggable transceivers tend to have more forgiving process requirements. Going through standard solder reflow or wire bonding is one of the main requirements.  

For CPO, you need to integrate much more tightly with the switch ASIC, CPU or GPU. So you potentially start to look at hybrid bonding, which has higher processing temperatures and also much more complex assemblies in general. All this requires much closer collaboration with system integrators.

Q5: With five Fortune Global 500 customers actively engaged in prototype testing, what's the biggest technical gate left between prototype and the high-volume production you're targeting for 2027?

A: There are several items that are required for true high-volume production: Silicon photonics foundries need to mature their slot waveguide processes, which means that the doping profiles and slot dimensions, for example, are within their required tolerances.  

The back end of line process transfer needs to be completed and qualified. And customers need to complete the design and qualification of their own module.  

Finally, the rest of the 400G per lane ecosystem needs to be ready, which typically is focused on DSPs and SerDes but may also include new optical and electrical connector assemblies and higher-power laser sources.

This summer, TechArena has been talking to the companies building the power and cooling infrastructure behind AI data centers.  

We caught up with Rich Whitmore, president and CEO of Motivair by Schneider Electric, which designs and manufactures liquid cooling systems, including coolant distribution units, for high-density AI and HPC deployments.

As rack densities climb and heat removal becomes as critical to AI deployment as power delivery, liquid cooling has become a foundational part of data center design from the outset. We talked about how Motivair builds cooling into AI projects from the beginning, how far liquid cooling can scale as rack power keeps rising, and what the company is engineering next as AI infrastructure demands continue to grow. Here’s what we learned.

Q1: Cooling has moved from the back of the house to the center of the design. How does Motivair help operators build cooling in from the start rather than add it at the end?

A: Cooling is now a foundational design decision for AI infrastructure. Motivair works alongside Schneider Electric’s power and digital infrastructure portfolio to help customers design liquid cooling into the project from day one. That integrated approach supports higher-density computing, simplifies deployment and helps customers reduce time to power by avoiding costly redesigns later in the project.

Q2: As rack power keeps climbing, how far can liquid cooling scale, and what changes along the way?

A: Liquid cooling is designed to scale alongside the growing demands of AI, from today’s high-density racks to tomorrow’s AI factories. As rack power increases, the challenge shifts from simply removing heat to delivering reliable, efficient and scalable thermal management across an entire facility. Motivair by Schneider Electric is advancing this next generation of liquid cooling with solutions such as coolant distribution units designed to support multimegawatt AI deployments and beyond. The future of cooling requires deeper integration with power, controls and digital infrastructure so operators can deploy faster, optimize performance and support the next wave of AI workloads.

Q3: What separates cooling that works for a supercomputer from cooling that works across thousands of racks in production?

A: Production environments require more than excellent thermal performance. They require repeatability, uptime, ease of service and seamless integration with the rest of the infrastructure. Motivair combines decades of liquid cooling expertise with Schneider Electric’s end-to-end infrastructure capabilities to help customers deploy AI at scale while reducing operational complexity and accelerating deployment.

Q4: When an operator chooses Motivair over another cooling option, what usually tips the decision in your favor?

A: Customers are looking for proven technology backed by deep engineering expertise and the ability to deploy quickly. Motivair offers high-performance liquid cooling that is part of Schneider Electric’s broader AI infrastructure portfolio, giving customers confidence that power, cooling, controls and services are engineered to work together. That integration helps reduce project risk and improve time to deployment.

Q5: What are you engineering for next, beyond today’s deployments?

A: The industry is moving toward even higher-density AI infrastructure with greater demands for efficiency, flexibility and speed of deployment. We’re focused on advancing liquid cooling technologies, expanding manufacturing capacity, and developing integrated solutions that help customers deploy next-generation AI infrastructure with greater confidence, scalability and operational efficiency.

The cost of serving a model increasingly comes down to how fast accelerators can move data between one another as AI inference workloads shift toward multistep reasoning and mixture-of-experts models.  

Faster chips are not the answer, said Aanchal Sharma, senior director of product management at Astera Labs, because they buy more compute, not less waiting. Put the fastest chip in the world behind a slow fabric, and you have built what she calls “an expensive space heater.”

We sat down with Aanchal as part of our series about how companies that build AI infrastructure are seeing requirements change as deployments scale. Astera Labs builds fabric switches and connectivity products that let AI accelerators from different vendors work together inside a rack.  

She talked about what it takes to prove multi-vendor interoperability before deployment, why Astera Labs builds Scorpio and Taurus as open rather than proprietary interconnects, and where the fabric has to scale next as clusters grow toward hundreds of thousands of accelerators. Here’s what we learned.

Q1: Rack-scale AI infrastructure now depends on many vendors' silicon working together inside one system: accelerators, memory and fabric all from different suppliers. What does that interoperability require in practice, and where does it tend to break down?

A: At rack scale, physics doesn't care whose logo is on the silicon. A GPU from one vendor, a CPU from another, memory from a third: They all have to hold a stable low-latency link under real production traffic every hour of every day for years. That's the bar.  

Most failures show up exactly where you'd expect: signal integrity breaking down across longer or noisier channels, firmware that was never tested together choking on link training, and faults that only appear once hundreds of accelerators fire the same collective operation at the same instant.  

We stress-test these exact combinations in our Cloud-Scale Interop Lab before a customer ever racks a single unit so customers can deploy with confidence and focus their engineering resources on AI innovation rather than infrastructure integration challenges.

Q2: Fabric switches and retimers now have to work across GPUs, CPUs and memory sourced from different vendors within the same rack. What does proving that kind of multi-vendor interoperability involve before a customer can put it into production?

A: It involves validating the rack as a complete system, not just each component in isolation.  

We re-create the customer's topology with the specific CPUs, GPUs, memory devices, fabric switches and retimers, then test across operating systems, software stacks, workloads and protocols. Working with ecosystem partners, we use co-emulation, test automation and continuous regression testing to identify compatibility issues and confirm seamless integration before deployment.

The result is a validated configuration that reduces integration risk and accelerates time to deployment. That's the entire point of an interop lab: catch every integration risk on our bench months before a customer ever has to.

Q3: Astera Labs builds Scorpio and Taurus as open multi-vendor scale-up fabric rather than a single vendor's proprietary interconnect. What does an open fabric let operators do that a closed single-vendor interconnect can't?

A: An open fabric means an operator never has to bet the entire rack on one company's roadmap. Mix the best GPU with the best CPU with the best memory; swap a supplier when lead times blow out; carry hardware forward across generations instead of tearing it out.  

Taurus supports this model across Ethernet, UALink and ESUN, while Scorpio provides an open software-defined fabric architecture that supports diverse accelerators, system topologies, and both open and platform-specific protocols.  

Compared with a closed interconnect, this keeps architecture and supplier choices open as performance, availability and platform requirements change.

Q4: As inference moves into production, a growing share of what it costs to serve a model comes from moving data between accelerators, not from the accelerators themselves. Why can't faster chips alone fix that?

A: Faster chips buy more compute. They don't buy less waiting.  

As inference workloads move toward multistep reasoning and mixture-of-experts architectures, more of the total job becomes accelerators talking to each other rather than accelerators doing math. Put the fastest chip in the world behind a slow, high-latency fabric and you've built an expensive space heater, with GPUs sitting idle waiting on data instead of generating tokens.  

That's exactly why Scorpio builds acceleration for collective operations, like Hypercast, directly into the fabric; why Taurus keeps those links clean and low power across Ethernet, UALink and ESUN as clusters scale; and why COSMOS gives operators the visibility to find and kill communication bottlenecks in real time. Tokens per watt and tokens per dollar get decided in the fabric long before anyone reads a chip's spec sheet.

Q5: As clusters grow toward hundreds of thousands of accelerators spread across racks and rows, where does the connecting fabric have to go next? How do open standards and products built on them fit that path?

A: The fabric has to scale in three directions at once: scale up inside the rack, scale out across racks and rows, and increasingly scale across between clusters and sites.  

Scale-up keeps pushing toward higher radix and lower latency inside the rack. Scale-out has to connect racks and rows without the distance tax of added latency and cost eating the gains. Scale-across is the newest of the three, holding performance together across distances that scale-up and scale-out were never built to cover.  

That means more optical connectivity, higher-density switching, and fabric that carries intelligence, not just bits, accelerating collective operations instead of passively relaying them. Open standards like CXL, Ethernet, PCIe, UALink and ESUN give customers one way to build that path without betting the company, but that's only part of the story. Some of the largest deployments we support run on NVLink Fusion, and others need a fully custom link built around one customer's architecture. Products like Taurus and Scorpio are built to make all three paths work.

As new AI cloud providers appear, many compete on a single number: how many GPUs they can rent by the hour. Dan Brown, who leads hardware engineering at DigitalOcean, sees the real contest elsewhere. In a recent TechArena Data Insights episode, Solidigm’s Jeniece Wnorowski and I talked with Dan and Solidigm account executive Ty MacAdam about building and delivering holistic data center infrastructure solutions for AI workloads. Dan's argument was consistent throughout: raw GPU capacity means little if data cannot reach it quickly, and that is where DigitalOcean is placing its bet.

The Data Velocity Problem

Dan frames the core AI infrastructure challenge around three factors: where and how data sets are captured, how quickly that data can travel from its source into an AI cluster, and how fast systems can exchange tokens between systems in the AI cluster to reduce the data down into a meaningful set that can be used for inference. When any of those steps drags, the entire pipeline slows.

“The AI solution designs are limited by what we call data velocity,” Dan said. “How quickly can you get your data in, transform it, and get it back out for next useful step in the pipeline.”

Much of the friction, he noted, comes from customers trying to run modern AI against aging storage. Many arrive with legacy arrays that have been in service for a decade or more, then discover that infrastructure cannot keep pace. Moving that data into DigitalOcean’s high-performance block and object storage, or directly onto the AI machines, is where the gains show up. Solidigm supplies the high-speed NVMes, including PCIe Gen5 drives, that sit close to the GPUs and keep the read and write cycle tight.

More Than a GPU Landlord

Dan is direct about how DigitalOcean differs from the wave of providers rushing into AI hosting. Many, he argues, do one thing only.

“They’re taking GPUs and servers, shoving them into boxes and then selling them by the hour to customers,” Dan said. “We call it a GPU landlord.” With those providers, he explained, customers must bring their own data scientists, hardware engineers, and expertise to make the hardware useful.

DigitalOcean’s answer is a full suite of compute and storage products that surround the GPU and share the same backbone network and data center, tightly coupled to improve velocity. Rather than selling raw capacity by the hour, Dan said, the company is building a unified cloud where every resource around the AI cluster exists to make that cluster faster and more cost effective to run.

Preference-Aware Routing

A clear expression of that approach is DigitalOcean Inference Router, which reads the intent behind each request and matches it to the model that best fits the developer’s task and priorities, whether they are optimizing for quality, cost, or latency, without any application-side routing logic. The approach is grounded in years of preference-aware routing research from the team behind the open-source Arch-Router work, now part of DigitalOcean. Dan described the result as token as a service, and he said the productivity effect has been striking.

Dan said the productivity gain has been dramatic: teams can now prototype a working product for a few hundred dollars in minutes, rather than standing up entire teams of administrators and data scientists to build the back end first.  

Density and the Nanosecond

Asked what organizations still overlook, Dan pointed to density and reached for a piece of computing history. Admiral Grace Hopper famously wore an 11.8-inch-long coil of copper wire. She used this to illustrate that for every such length of copper in a system, a nanosecond of delay is introduced in that system. Across a large AI cluster, Dan noted, miles of copper and fiber add up to real delay.

DigitalOcean’s response is to build the densest possible clusters, adopt liquid cooling, and keep local Solidigm storage on each server so data moves host to host in microseconds rather than milliseconds. That translates directly into shorter compute time, faster time to first token, and lower customer costs.

Ty reinforced the point from the storage side. “Storage used to be a passive layer where everybody just thought of it as where the data would sit. That doesn’t hold up anymore,” he said. “Storage isn’t where the data sits. It’s more about how fast intelligence moves.” He added that Solidigm’s high density quad-level call (QLC) NAND puts more data closer to compute at lower power, which reshapes the total cost of ownership math at deployment scale.

The TechArena Take

DigitalOcean’s message to technology decision makers is a useful reminder to a market fixated on GPU counts. Buying accelerators and fast networking is only part of the equation. If data cannot reach those GPUs quickly, or if teams overspend on oversized models for narrow jobs, the investment underperforms. Refreshing legacy storage, right-sizing a model to a task, treating proximity between data and compute as a design decision rather than an afterthought are practical places to start to get the most out of an AI infrastructure investment. As agentic workloads raise the pressure further, the providers building for velocity, not just capacity, look best positioned for what comes next.

For more information, listen to the full podcast and visit DigitalOcean.com.

The same chip architecture that Axelera AI built for the power, cost and latency constraints of edge vision AI can now also handle multiuser generative AI workloads at the edge, using the same Metis and Europa chip family and toolchain.

We sat down with Manuel Botija of Axelera AI, which builds AI chips for edge devices. Manuel said that expanding capability matters as more AI workloads move beyond the data center and edge devices are asked to run increasingly complex models. Europa, he said, is built to run vision-language models and agentic AI directly on edge hardware.

We also talked about what digital in-memory computing changes about chip performance; how customers worldwide are using Axelera AI’s technology; and what it takes to offer a predictable cost per token as inference workloads grow.  

Here’s what we learned.

Q1: Much of the progress in AI to date has come from scaling the data center, but only a handful of players can build at that scale. Where did Axelera AI see a different path?

A: Most of the industry has tried to scale AI from the data center out to the edge, and that direction has never really worked. Data center architectures are built for abundant power, abundant budget and abundant time. None of that is available at the edge. So the chips and systems built for one environment were getting forced into a location they were never designed for, which meant serious tradeoffs had to be made.

We believed there was a different way and started by holding those “limits” as real limits. We built an architecture that solves for small power envelopes, limited budgets and time from the beginning, because at the edge, those constraints are real.  

Our first architecture, the Metis AIPU, had to hit real performance within a tight power envelope and a real price point, and that discipline is exactly what let us scale to the Europa AIPU without compromising the efficiency we perfected. We have even taken that edge architecture with both Metis and Europa and built server-class products supporting customers globally with full-length, full-height PCIe cards built around multiple AIPUs.

Q2: Axelera AI made its name at the edge with the Metis platform. What are customers using the products you’ve brought to market to solve?

A: Customers are putting Metis to work solving real problems in places most people never think to look.  

A large global convenience store chain uses it for real-time shelf monitoring, catching stockouts as they happen. An agritech company in New Zealand built it into automated apple sorters, and a healthcare technology company in Europe uses it inside an automated pill sorting machine, both cases where accuracy and speed directly affect people’s safety.  

The utilization keeps expanding. A drone company in Croatia deploys Metis in a drone pack built for search and rescue missions. In India, a company built a cargo monitoring system for the country’s busiest port, bringing computer vision to logistics at scale.

These are just a handful of the deployments in production, with new uses for vision and localized models emerging around the world. What stands out is how different these use cases are from one another, and how each one found its way to edge AI because it was the only architecture that could meet their constraints on power, cost and latency at once. It’s a reminder of how much innovation is happening at the edge, in every corner of the world.

Q3: At the heart of your chips is digital in-memory computing on a RISC-V core. What does moving the math into memory change about performance and efficiency?

A: Traditional processors spend most of their energy and time moving data back and forth between memory and compute, especially for the matrix-vector multiplication that makes up roughly 80% of AI inference workloads.  

Digital in-memory computing performs that math directly inside the memory cell, so the data barely has to move at all. That single change removes the biggest bottleneck in conventional architectures and turns into real gains in both speed and power. The RISC-V controlled dataflow is what makes this programmable and precise rather than a fixed-function shortcut. It gives us four independently programmable cores that can run parallel model execution with deterministic, digital accuracy, unlike analog in-memory approaches that trade precision for efficiency.  

That combination is why the Metis architecture delivers up to 15 TOPS per watt and three times better performance per watt than GPU-based solutions, all while keeping accuracy indistinguishable from a full floating-point model. The result is an architecture that scales cleanly. The same principles that made the Metis AIPU efficient at 214 TOPS carry through to the Europa AIPU at 629 TOPS, which means moving the math into memory isn’t just a one-time efficiency trick. It’s the foundation of our roadmap.

Q4: Axelera AI is scaling one architecture from Metis at the edge up to Europa for multiuser generative AI. What does that move open up for your customers?

A: The Europa AIPU is built on the same digital in-memory computing and RISC-V dataflow foundation as Metis, but with meaningful additions: twice the AI cores, native video decode, and 16 vector cores dedicated to pre- and post-processing, all of which make Europa perfectly suited for larger and more complex models and analytics. Those additions are what make native transformer support and multiuser generative AI workloads possible at the edge.

What stays constant for customers is the development experience. The same Voyager toolchain carries across both chips, so a team that built a pipeline on Metis isn’t starting over on Europa. That continuity is what turns an architectural leap into a growth path instead of a rebuild.  

Metis remains an efficient powerhouse for computer vision use cases at the edge, including embedded systems where power and space are tightly constrained. Europa extends that foundation into a highly flexible AI accelerator built for a broader range of inferencing needs, from vision-language models and agentic AI to dozens of concurrent high-resolution video streams running natively at the edge. For customers, that means the same tooling and philosophy they already trust, now available across an even wider set of use cases.

Q5: Two years out, what will separate an inference platform that lets an operator promise a predictable cost per token from one that can’t?

A: Digital in-memory computing removes the data movement bottleneck that makes costs scale unpredictably as workloads grow, so the economics hold steady whether an operator is running a handful of streams or scaling to dozens of concurrent workloads.

The other separator is architectural flexibility. Operators locked into separate chips for CNNs and separate chips for transformer-based models face a cost structure that multiplies every time they add a new model type. A unified architecture that handles both on the same silicon keeps cost per token stable as workloads evolve from vision to generative AI, which is the discipline that will define the platforms operators can build a business on two years from now. Europa delivers that flexibility to customers.

Lastly, when models are run on one’s own infrastructure, the cost per token is much more manageable than running models in the clouds through providers that change cost structures at their discretion. With the Axelera Voyager toolchain, customers can use open-source models tuned to their weights and deploy them in-house. This includes Voyager Wingman, an agentic platform that can build pipelines based on natural language prompts.

Solidigm and io.net discuss neo-cloud economics, GPU pricing, decentralized AI compute, and inference cost cutting versus hyperscaler cloud infrastructure.

We sat down with David Driggers, CEO and founder of Cirrascale Cloud Services, a neocloud that runs dedicated hardware for AI workloads across NVIDIA, AMD, Qualcomm and Tenstorrent accelerators. Cirrascale’s approach is relevant now as enterprises move from proof-of-concept AI deployments toward production and need infrastructure that matches each workload to the chip built for it.

We talked about how Cirrascale matches workloads to the right accelerator as models grow, how its billing model creates cost predictability, and what separates neoclouds built for low-latency inference from those built for training alone. Here's what we learned.

Q1: Why run every accelerator on dedicated hardware?

A: Cirrascale gets classified as a neocloud, but we were building this hardware before the category had a name. We designed the industry’s first 8-GPU server back in 2012 working directly with NVIDIA, and we were infrastructure partners to OpenAI when they were still an 8-10 person team. That hardware DNA is why we can run NVIDIA, AMD, Qualcomm and Tenstorrent side by side, alongside our work with Google on Google Distributed Cloud, and actually know what each one is good for. No single chip serves a 1B-parameter model and a 400B-parameter model economically, so our customers get the accelerator that fits their workload instead of bending their workload to fit whatever one vendor is selling.

Q2: How does billing create predictability?

A: We don’t charge for data ingress or egress, which removes one of the biggest hidden cost swings customers deal with elsewhere. We also give customers the choice between token-based pricing for intermittent workloads and GPU-hour billing for anything running 24/7, so the bill follows how they use the platform. And because we track token consumption at a granular level, customers get real insight into where they’re burning input tokens inefficiently. That data helps them optimize the model, not just the invoice. Inference is forever, as we like to put it, and those costs compound. That’s why we built a cost calculator: Customers can model their own workload before they commit to anything.

Q3: What’s standing between customers and more capacity, and what’s changed?

A: Most enterprises are still in the POC phase with models like Gemini, GPT and Claude, trying to prove utilization before they can get more approved budget. Nobody’s handing out incremental AI spend without proof it moves the needle. That’s part of why we run Private Gemini on Google Distributed Cloud. It gets a frontier model into production inside the customer’s own security boundary instead of leaving it stalled in evaluation. On the hardware side, we’re pushing into multimodal, larger-context inference. What’s changed in the last year is that hybrid deployment, on-prem for steady workloads plus neocloud for real-time workloads across regions, is becoming the norm. I expect that to be standard by the end of this year and accelerate hard through 2027.

Q4: What have you learned matching workload to chip?

A: We built the load balancing in the Cirrascale Inference Platform ourselves, specifically because a model’s ideal hardware changes as it grows. An 8B model that becomes a 70B model isn’t cost-optimal on its original chip anymore, and that kind of migration is really hard to do on-prem. The single biggest cost lever is capacity utilization. The platform puts each workload on the most efficient capacity that still clears its latency target, so customers aren’t paying premium rates for headroom they don’t need.  

Q5: What separates the neoclouds that thrive from the ones that fade?

A: Training and inference are completely different animals. In training, you can run a thousand GPUs and barely notice a network hiccup. In inference, that same hiccup is a failed request in front of a customer. The neoclouds that survive are the ones built for real-time workloads and low latency from the ground up, not the ones bolting inference onto training infrastructure. Carrier-hotel-grade connectivity is a big part of that, which is why we partnered with Telehouse to deploy the Cirrascale Inference Platform inside their data centers. It also means being able to serve regulated industries, where FedRAMP High and CMMC 2.0 are the price of entry rather than a roadmap item. We also give customers real ownership flexibility. Some customers own their GPUs outright while we run the networking and storage around them. And we’re expanding our data center footprint this year to put that capacity and failover closer to where the inference actually happens.

Nicklas Frahm of Corti on domain-specific AI, built-in compliance, and the Kubernetes Engine powering sovereign healthcare infrastructure.