Author: Zach Horn, Senior Marketing Manager, Akash Network
Bio: With a background in economics and a passion for decentralized AI applications, Zach Horn is the Senior Marketing Manager for Sequoia Scout-backed Overclock Labs, the creators of Akash Network, a distributed peer-to-peer marketplace for compute resources. Prior to joining, Zach published the Vault Newsletter with insights on decentralization, AI, and the future of the internet.

Access to GPUs is becoming increasingly tight amid the AI boom, with the waitlist for some Nvidia models now at three years. While big tech is squeezing the market, smaller companies are tapping into alternative GPU sources and cloud networks to train and scale their AI apps. Greg Osuri, founder of open-source cloud Akash Network, explores these questions and sheds light on the current realities of AI and machine learning development and where the industry is headed.

Here’s the full interview, lightly edited for clarity:

• What can be done to help ease this squeeze on GPU resources?
  • Osuri: Right now, the traditional providers – AWS, Microsoft Azure, and Google Cloud – are unable to meet demand from developers. Distributed and permissionless networks can help sustain the AI boom and mitigate concerns that more established players will dominate access to these resources. Big tech companies are looking to purchase chips, while the smaller networks are repurposing underused chips that are more accessible at a cheaper price point. By giving permissionless access to compute resources including Nvidia A100s and H100s from a range of providers – from independent to hyperscale – these computing platforms are uniquely positioned to mitigate inefficiencies.

• Big tech seems to be trying to take on Nvidia by making its own chips. Do you think this will alleviate the chip shortage?
  • Osuri: I think the question that begs to be asked is: can supply catch up to the demand curve? It’s extremely difficult to predict the demand curve for AI given the varying degrees of sophistication. There’s a general demand for chips, but we don’t have a great system to calculate which are in the highest demand. The big challenge for Nvidia and big tech companies is understanding the demand for a certain type of chip. If they can crack that code, they’ll know where to focus their time and money, which can alleviate the shortage of chips in the long run.

• Do other companies have AI training software similar to Nvidia that can compete in the near term?
  • Osuri: There’s always the possibility that a bright developer will enter the industry and crack the software code. Is there evidence of that? Not yet. Right now, I don’t think other companies have the resources necessary to support an AI training facility.

• Since access to chips is so tight, is there any hope? What sort of alternatives exist for smaller players to compete in this environment?
  • Osuri: It depends, since AI is so broad. More powerful models like GPT4.5 won’t be able to run on consumer-grade chips like Nvidia 4090s, but the older, task-specific AI models should be able to. It’s too early to say which model is going to win. For general purpose, large models are great, but for more specific tasks like coding, GPT 4.5 is tough. There will be different markets for different chipsets, with various types of AI running on less sophisticated chips, while the more specialized models will require high-performance chips.

• In that same vein, chips are not cheap. Do you think model diversity will drive down those prices?
  • Osuri: Sure, diversification driving down chip prices is the industry’s goal. But the issue today isn’t the supply of chips. It’s that we don’t have the right type available for AI workloads.There are chips in the hands of consumers already, like Nvidia 4090s and 3090s which are leveraged in gaming consoles like the Playstation or Xbox, or Apple M1s and M2s in new MacBooks. We have a ubiquitous supply of GPUs in Macs and gaming consoles, but until now, there wasn’t a way for these types of chips to come online.

• How can companies leverage diverse chips to power their AI workloads?
  • Osuri: An increasing number of organizations have turned to distributed and permissionless cloud networks to gain access to GPUs, including less sophisticated chips that in many cases sit idle. By using less intensive data set requirements, these teams will deploy more efficient techniques like Low-Rank Adaptation (LoRA) to train language models and distribute workloads in a parallel manner.This involves deploying clusters of lower-tier chips to accomplish tasks equivalent to a smaller number of A100s and H100s. A new era of cloud computing will emerge, one in which power is decentralized and not in the hands of just a few.

• Is the current cloud market sustainable when it comes to the increasing demand for compute-to-power AI investments?
  • Osuri: No, and I don’t think it will be in the near future. AI companies are already struggling to gain access due to larger companies with more resources purchasing high-performance chips directly from chip manufacturers, rather than going through a middleman like the cloud. Think about when you’re purchasing a car – would you purchase it from a rent-a-car company like Hertz or visit a car dealership? AI companies are using this logic when purchasing AI chips, though chip makers can’t just ramp up supply to create sustainability as only a limited number manufacture and sell their own chips, like Intel.Cloud companies are now a lower priority for chip manufacturers who prefer to sell directly to their customers. This contributes to the squeeze for compute, restrictive long-term contracts, and sky-high price points in the cloud that force upstarts out.

• What are some of the biggest challenges companies face when adopting a serverless architecture?
  • Osuri: The big challenge with serverless is that while it looks good on paper architecturally, people haven’t thought through the overhead it adds when it comes to the reauthentication of every individually hosted service. It’s great for getting started, but scaling becomes challenging because the overhead is so high. Every time you use a different service, you need to re-authenticate, and authentication is cumbersome and expensive when it comes to computation costs.For serverless companies, it’s challenging to achieve profitability as the majority of users are hobbyists or hackathon developers rather than dedicated users. For example, Amazon Prime reduced its infrastructure costs by 90% when it adopted a monolithic architecture.

To learn more about Kubernetes and the cloud native ecosystem, join us at KubeCon + CloudNativeCon Europe in Paris from March 19-22.