Sizing hardware for running models locally
Technique
Size hardware for running an AI model on your own machine by checking that the whole model fits in the graphics chip's memory. Memory amount, not chip speed, decides what you can run.
Theo's current stance
Theo says splitting a model across two consumer graphics cards doesn't really work, because the link between them is far too slow to share memory.
Try it
- Before buying hardware to run a model, check the model's size against the memory of a single machine.
- Don't plan on linking two consumer graphics cards to fit a bigger model.
- Look at machines with large unified memory (shared by processor and graphics), like the Mac Studio.
Where he said it
7:25
You can't just plug the chips in together. If you have a model that is 40 gigs, hell, if you have a model that's 33 gigs and it doesn't fit on your 5090, you now have to split the data across the two GPUs, which means ideally you…
Theo Daily is unofficial and not affiliated with Theo or t3.gg. Written by Claude Opus 5.5 from each video's transcript.