For most of the last two years, the GPU conversation was about cost. How do we bring down the bill? Should we commit or stay on-demand? Is this inference workload worth what it costs to serve?
Those questions still matter. But in 2026, a more basic question has taken precedence: can you get the GPUs at all?
Across the market, the constraint has shifted from price to access. On-demand capacity is sold out. Contract rates are climbing on four-year-old hardware. Capacity that hasn't been built yet is already spoken for. For AI teams, the bottleneck is no longer the budget. It's the queue.
That shift changes what good GPU infrastructure management looks like. When you can't buy your way out of a capacity problem, the GPUs you already have become your most valuable, and most underutilized, asset.
The market has run out of spare GPUs
The evidence comes from every layer of the supply chain, and it points the same way.
Rental capacity is gone. SemiAnalysis, which tracks GPU rental pricing across more than 100 market participants, reported in early 2026 that on-demand capacity was sold out across every GPU type. When it went looking for just 64 H100s or H200s, half the providers it asked had none. Capacity scheduled to come online through August and September 2026 was already booked.
Old hardware is getting more expensive, not less. The H100 launched in 2022, and the market spent 2025 expecting its rental price to keep falling as Blackwell shipped. Instead, SemiAnalysis's one-year H100 contract index climbed nearly 40%, from $1.70 to $2.35 per GPU-hour, between October 2025 and March 2026. Some H100 contracts are now renewing at the same rates signed two to three years ago, with terms running through 2028.
Providers are still raising prices. On October 1, 2026, Nebius raised its on-demand H100 price by 17% and its H200 price by 20%. A week later, AWS raised EC2 Capacity Blocks prices by roughly 15%. That's a sharp reversal from June 2025, when AWS cut its H100 instance prices by 44%. (Source: Uvik GPU Price Index)
Relief isn't coming soon. The bottleneck has moved upstream into memory, which every AI accelerator depends on. On its October 1 earnings call, Micron CEO Sanjay Mehrotra said, "In calendar 2027 as well as 2028, we see demand exceeding supply." He added that Micron has no line of sight to when the market will balance. (Source: Tech Insider)
Power is a second obstacle. Even when chips arrive, there often isn't power to run them. Microsoft CEO Satya Nadella said in late 2025 that his biggest constraint wasn't chip supply but power and ready data center space, leaving him with "a bunch of chips sitting in inventory that I can't plug in." Microsoft CFO Amy Hood told analysts the company had been short on space or power for many quarters, with demand still rising. For everyone else, that means new capacity can be delayed even after it's been bought. (Source: DCD)
Put together, the picture is clear. For most AI teams, getting more GPUs means long waits, rising prices, and multi-year commitments, if the capacity is available at all.
Scarcity turns every team into a hoarder
When GPUs are hard to get, the rational move is to never let go of the ones you have. SemiAnalysis observed exactly that: teams holding on-demand capacity won't release it back to the pool, even as prices climb. Some renters are even subdividing their clusters and subletting the spare capacity.
The same behavior plays out inside companies. An ML team that waited months for a GPU allocation will keep it reserved through quiet weeks, because giving it back means rejoining the queue. A platform team will provision for peak and leave headroom idle, because running short during a launch is worse than paying for slack.
Each of those decisions makes sense on its own. Together, they create an organization where GPUs are simultaneously scarce and underused. The queue for new capacity grows while capacity the company already pays for sits partly idle.
This is why buying more GPUs rarely fixes the problem. New capacity gets absorbed into the same allocation patterns, and the waste scales right along with the fleet.
The hidden fleet: GPUs you already own, running cold
Here's the paradox. While the market fights over every available GPU, the GPUs companies already have are mostly sitting idle.
In a June 2026 VentureBeat Pulse Research survey, 83% of enterprises that operate their own GPUs reported utilization of 50% or less. Nearly half (49%) reported 25% or less. Only 12% cleared the halfway mark, and 8% didn't measure utilization at all.
The same survey found that the most common plan for the year ahead is evaluating more compute. VentureBeat described the result as enterprises buying more infrastructure while the capacity they already own sits largely unused.
The survey skews toward mid-market companies and is best read as directional. But the direction matches what infrastructure teams see in their own clusters, and the reported numbers likely flatter reality. Standard utilization metrics, like the ones reported by nvidia-smi, show whether a GPU is running any work at all, not how much of the GPU that work actually uses. A GPU can report high utilization while most of its compute and memory sit idle.
That's the opportunity hiding in plain sight. If most fleets run at half capacity or less, the fastest source of new GPU capacity isn't a cloud provider or a procurement cycle. It's the fleet you already have.
Get More from Your GPUs
Sedai reclaims idle allocations & packs more workloads onto each GPU. So you can get more jobs done.
How to get more from the GPUs you have
The goal isn't to give GPUs back. In this market, no one wants to. The goal is to put more work on the GPUs you're already holding. Five moves make the biggest difference.
- Measure what GPUs actually do. Look past the single utilization percentage. Track compute activity, memory use, and throughput per workload, so you can tell a busy GPU from one that's merely occupied.
- Reclaim idle allocations. Find workloads that hold a GPU but rarely use it, such as notebooks left running, dev environments, and services sized for a peak that never comes. Freeing those allocations lets the next job in the queue run.
- Share GPUs where workloads don't need a whole one. Many inference, dev, and test workloads use a fraction of a modern GPU. Partitioning with NVIDIA Multi-Instance GPU (MIG) or time-slicing lets several workloads share one device safely.
- Pack workloads to avoid stranded capacity. In Kubernetes, scattered workloads leave fragments of GPU capacity that no single job can use. Better bin-packing consolidates that space, and the scheduler places pending workloads on it.
- Make it continuous and safe. Workloads and models change weekly, so a quarterly cleanup goes stale fast. Teams also hoard GPUs because they fear losing them, so any change needs guardrails, validation, and a clear view of the impact before it happens.
None of these moves requires buying anything new. Together, they turn GPU waste into GPU capacity: more models served, more experiments run, and shorter queues on the same hardware.
Ask a different question
The GPU shortage isn't going away soon. Memory makers see demand outrunning supply into 2028, rental prices are still climbing, and new capacity is booked before it comes online. Teams that plan around buying their way out will spend 2027 waiting in line.
The teams that pull ahead will ask a different question. Not "how do we get more GPUs?" but "how do we get more from the GPUs we have?"
That's the problem Sedai was built for. Sedai models true GPU utilization across Kubernetes clusters, reclaims idle allocations, packs more workloads onto each GPU, and does it safely and continuously, so your existing fleet takes on more work. See how it works.