Rusticl Capable Of Running Tinygrad For LLaMA Model

[phoronix]

Phoronix: Rusticl Capable Of Running Tinygrad For LLaMA Model

Mesa's Rusticl OpenGL implementation written in Rust it turns out can already run the Tinygrad open-source software with its OpenCL back-end for running the LLaMA model...

https://www.phoronix.com/news/Rusticl-Runs-Tinygrad-LLaMA

[Mathias]

Maybe I should put "Georg Hots makes AMD compute usable" on my 2024 Bingo card.

[dawid.mlyn]

Michael typo: Mesa's Rusticl OpenGL implementation written in Rust

[Quackdoc]

Maybe I should put "Georg Hots makes AMD compute usable" on my 2024 Bingo card.

yeah it's not something I thought would happen

[vancha]

How is one person currently doing a better job than an entire company that has actively been trying to enter that segment of the market? He's right, the amd drivers are a mess for compute workloads. Not something i can say from experience, but i've heard it numerous time from people that do.
The guy's a genius, so I wonder what he'll end up with, hopefully he reaches his goal.

[brucethemoose]

How is one person currently doing a better job than an entire company that has actively been trying to enter that segment of the market? He's right, the amd drivers are a mess for compute workloads. Not something i can say from experience, but i've heard it numerous time from people that do.
The guy's a genius, so I wonder what he'll end up with, hopefully he reaches his goal.

You are talking about Hotz?

He is not. There are a couple of frameworks that can run fast OpenCL or Vulkan LLaMA (among other models) right now, some of them with CUDA rivaling performance running on Nvidia. Last I checked, tinygrad is so slow in LLaMA (5s/token?) that its basically not functional, though I can't find any recent benchmarks and I can't find a single interface that is using it as a backend.


But this Hotz guy has attracted tons of attention on Twitter an YouTube... Honestly, I don't understand all this hype around tinygrad and Hotz.

[kiffmet]

brucethemoose George Hotz is kinda famous for having made the first iPhone jailbreaks, aswell as for being the first person to break into Sony's PS3. As a result of the latter, Sony sued him for damages because "he enabled software piracy", which in turn caused "Anonymous"/hacktivists to break into their infrastructure, which caused quite a bit of media uproar. In the end, a settlement was achieved with Hotz promising to not break into Sony consoles anymore.

When he says will program or reverse engineer 'insert difficult, technical thingy here', he is absolutely going to follow it through.

[brucethemoose]

brucethemoose George Hotz is kinda famous for having made the first iPhone jailbreaks, aswell as for being the first person to break into Sony's PS3. As a result of the latter, Sony sued him for damages because "he enabled software piracy", which in turn caused "Anonymous"/hacktivists to break into their infrastructure, which caused quite a bit of media uproar. In the end, a settlement was achieved with Hotz promising to not break into Sony consoles anymore.

When he says will program or reverse engineer 'insert difficult, technical thingy here', he is absolutely going to follow it through.

Until I see real world tinygrad benches (not toy test scripts), I am still extremely skeptical. Building and maintaining a performant ML framework is a different order of magnitude than finding exploits.


One particular thing I don't understand: why reinvent the wheel? There are some excellent industry efforts that he could have built on top of, at the very least.

[kiffmet]

why reinvent the wheel?

The goal is to make AI accessible to everyone, regardless of the device. CPUs, GPUs, DSPs, custom accelerators, … basically everything with an FPU. I.e. think about PyTorch or TensorFlow - it took ages until the former finally received ROCm support, while the latter had to be forked and always lags a few versions behind upstream. Tinygrad aims to make the programming aspect of creating a NN easier aswell.

[Amaranth]

brucethemoose George Hotz is kinda famous for having made the first iPhone jailbreaks, aswell as for being the first person to break into Sony's PS3. As a result of the latter, Sony sued him for damages because "he enabled software piracy", which in turn caused "Anonymous"/hacktivists to break into their infrastructure, which caused quite a bit of media uproar. In the end, a settlement was achieved with Hotz promising to not break into Sony consoles anymore.

When he says will program or reverse engineer 'insert difficult, technical thingy here', he is absolutely going to follow it through.

fail0verflow hacked the PS3, they even demoed their work at CCC. They refused to make the private key they found available though because that would enable piracy. Hotz then used their hack to get the same key and published it which triggered the lawsuits. The members of fail0verflow were even named defendants in those lawsuits.