AirLLM: Run 70B Models on a 4GB GPU — Hype vs Reality

[Alex]: Welcome, tech enthusiasts, to another exciting episode of the Nerd Level Tech AI Cast, where we dive deep into the world of technology and pull out the gems for you. I’m Alex.

[Jamie]: And I’m Jamie! Today, we’re tackling something that sounds almost too good to be true — running 70 billion parameter models on a puny 4GB GPU. It’s all about AirLLM today, folks. Hype or reality? We'll find out.

[Alex]: Exactly, Jamie. But before we get into the nuts and bolts, let’s set the stage. Imagine you're trying to run a marathon, but you’ve only got flip-flops on your feet. Sounds tough, right?

[Jamie]: [LAUGHS] Definitely not the footwear of choice for a marathon, Alex!

[Alex]: Well, that’s kind of the situation with running massive language models like the ones we’re discussing today. These models usually need a powerhouse of a GPU with tons of VRAM. But here comes AirLLM, claiming to let these heavyweights run on, well, flip-flops.

[Jamie]: 
So, how does AirLLM pull off this magic trick?

[Alex]: It’s all about something called layer-wise inference. Traditional model inference wants all the data — or model layers — loaded at once. AirLLM changes the game by loading one transformer layer at a time, processing it, and then moving on to the next one.

[Jamie]: So, it’s kind of like reading a book one page at a time instead of opening every single page on a giant table. Sounds memory efficient but slow?

[Alex]: Spot on, Jamie. It’s memory efficient, but there’s a tradeoff — speed. From the tests, it’s like getting 0.7 tokens per second on the faster setups and down to minutes per token on the slower ones.

[Jamie]: Minutes per token? I could probably make a sandwich in that time!

[Alex]: [LAUGHS] Exactly! You could start a small sandwich shop in the time it takes for some setups to crank through a model. So, while AirLLM makes it possible to run huge models on small GPUs, it’s not the speedy solution you’d use for, say, real-time applications.

[Jamie]: Got it. But how does AirLLM stack up against other solutions?

[Alex]: Well, compared to something like llama.cpp, which uses quantization — basically simplifying the data to make it fit — AirLLM runs these large models at full precision. No shortcuts. But remember, llama.cpp can process up to 15 tokens per second. That’s a lot faster than AirLLM’s best case.

[Jamie]: Right, so if I’m just playing around at home, trying to understand how big models work, AirLLM could be my go-to. But if I’m building something like an interactive chatbot, maybe not so much.

[Alex]: Exactly! AirLLM is fantastic for researchers or hobbyists who need access to big models without the big hardware. But for production-level speed, it’s not quite there yet.

[Jamie]: And what about the hardware compatibility?

[Alex]: AirLLM is pretty versatile. It runs on Linux, Windows, macOS, even on Apple Silicon. Just remember, it’s a beast when it comes to disk space. You’ll need about 70GB free after its initial setup.

[Jamie]: So, before you download, maybe check if you've got enough space, or you might need to delete a game or two!

[Alex]: Or maybe a dozen, depending on your gaming habits, Jamie! [LAUGHS]

[Jamie]: Guilty as charged. 
Well, it sounds like AirLLM is a bridge to the future — making the impossible possible, just not instantaneously.

[Alex]: Beautifully put. It’s about pushing boundaries, testing limits. For those on the frontier, AirLLM is a tool that opens new doors.

[Jamie]: And for the rest of us, it’s a peek into what might soon be more accessible. Thanks, Alex, for breaking down the tech wizardry of AirLLM for us and our listeners!

[Alex]: Always a pleasure, Jamie. And thank you, everyone, for tuning in. Remember, the future is just a podcast away!

[Jamie]: Don’t forget to subscribe for more episodes from Nerd Level Tech AI Cast. Catch you next time! [OUTRO MUSIC FADES IN]