Elon Musk’s startup xAI has revealed its newest creation: a supercomputer named Colossus.
This large AI coaching system presently boasts 100,000 Nvidia Hopper H100 processors.
Situated in Memphis, Tennesse, Colossus is ready to be expanded with 50,000 of Nvidia’s new, extra superior H200 sequence chips, roughly twice as highly effective because the H100s.
That may virtually definitely make Colossus essentially the most highly effective laptop on the planet if it wasn’t already.
For context, Meta had introduced plans to build up 350,000 H100 GPUs by the tip of 2024, whereas OpenAI was believed to make use of simply round 25,000 GPUs for GPT-4, although it may very well be many extra.
Musk introduced the launch of Colossus himself on X over the Labor Day weekend, stating, “From begin to end, it was carried out in 122 days. Colossus is essentially the most highly effective AI coaching system on the planet. Furthermore, it can double in measurement to 200k (50k H200s) in a number of months.”
This weekend, the @xAI group introduced our Colossus 100k H100 coaching cluster on-line. From begin to end, it was carried out in 122 days.
Colossus is essentially the most highly effective AI coaching system on the planet. Furthermore, it can double in measurement to 200k (50k H200s) in a number of months.
Wonderful…
— Elon Musk (@elonmusk) September 2, 2024
Powering the way forward for Grok
The speedy goal of Colossus is to coach xAI’s massive language mannequin (LLM) referred to as Grok.
At the moment, Grok is barely out there to paying subscribers of Musk’s X social media platform. The corporate has already launched an early beta of Grok-2, which was educated on round 15,000 Nvidia H100s.
Regardless of this comparatively small coaching information set, Grok-2 is already thought-about among the many most succesful AI massive language fashions in line with aggressive chatbot leaderboards.
It’s additionally open-source, bucking the development of closed-source mannequin releases from opponents OpenAI, Anthropic, and Google.
Musk’s ambitions for Grok are extravagant. He’s already looking forward to Grok-3, slated for launch in December 2024.
In a July interview with Jordan Peterson, Musk boldly claimed, “Grok-3 must be essentially the most highly effective AI on the planet at that time.”
The huge enhance in GPU rely for coaching Grok-3 means that Musk is critical about this declare.
What about xAI’s different targets?
Based in July 2023, xAI was launched with a mission that goes far past the AI trade’s typical remit.
In Musk’s personal phrases, xAI in the end desires to search out out, “What the hell is absolutely happening?” together with probing elementary questions on actuality, darkish matter, the Fermi Paradox, and different cosmic mysteries.
The xAI group was picked to share that imaginative and prescient. Igor Babuschkin, who beforehand labored on the Massive Hadron Collider at CERN, acknowledged that the corporate desires to “actually make a dent in our understanding of the universe.”
Jimmy Ba, an AI researcher from the College of Toronto, defined the objective of constructing a “normal goal problem-solving machine” to deal with humanity’s most difficult issues.
To gasoline its future, xAI secured $6 billion in a Collection B funding spherical in Could 2024, backed by enterprise capital companies like Andreessen Horowitz and Sequoia Capital, in addition to deep-pocketed traders like Constancy and Saudi prince Alwaleed bin Talal’s Kingdom Holding.
Many Tesla specialists consider that Grok may finally energy the AI behind Tesla’s humanoid robotic, Optimus. Musk claimed Optimus may generate $1 trillion in annual earnings for Tesla.
Musk has additionally hinted at probably proposing a $5 billion funding from Tesla into xAI, which some shareholders have welcomed.
With Colossus now on-line, would possibly xAI be poised to transition past creating extra LLMs and try one thing daring and new?
Positive, Grok-3 will in all probability be very highly effective. It’ll in all probability solely be barely extra highly effective than its friends, and just for a short while till one thing overtakes it.
It’s a must to query the elemental long-term targets right here. AI firms are grimly competing with related fashions educated with just about equivalent {hardware} stacks. It’s costly, and innovation dangers plateauing.
This leaves the sphere open for one of many major gamers to see past the period of LLMs.
With Colossus, will xAI be the primary one to step up as Musk hinted it could when it was based?
