This is a weird question to ask here, but I thought I would start here and then get pointed to the right place. I am an academic researcher who regularly does computational text analysis. Much of what I do is small scale, and so I can do almost all of it on my M1 MBA. Sometimes things will take half an hour or hour to run, but that’s fine.
Transformers change all that. Running a build of even a simple GPT using 1000 small texts can take an hour per epoch and I want to run several dozen of those epochs. And it uses all 8 of my MBA’s cores almost fully. (Like 790%.)
I clearly need to off-load this work, and so I am wondering what the most cost efficient way of doing this is: external GPU seems like it won’t offload enough; external hardware means I have to build a separate software stack on that computer to run things; or just move everything to the cloud and pay for using cycles there.
If anyone has had a similar experience and run the numbers and/or built something. I would love to know about it.
I’d consider going the cloud route if you can, unless your data is proprietary. Lots of options across the computing spectrum for pennies to a few dollars per hour.
On the build front, the up front cost and hassle is one thing, then there’s the noise and heat and you’d be well served to also look at the power consumption beforehand; it might surprise you in a bad way depending upon where you live.
Like Artisan said, cloud is almost definitely the way to go. Something like a Mac Studio or Macbook Pro M3 Max would definitely run better than your Air, but I’m not sure how much. Since you’re an academic, does your institution have any kind of cloud service they can set up for you to use and provide support for?
I use Azure through work and they have pretty robust support for spinning up LLMs of any kind. Training small ones is not expensive at all, but I don’t pay the bill so I don’t have any exact numbers.
My thanks to you both for confirming what I suspected. Sadly, I work at a regional public university in the Deep South that could not care less about supporting faculty like me, so I’ll have to see what’s possible in terms of funding some processing time “in the cloud.”
Thinking laterally, is this a one off thing or a regular thing?
If it’s a one off, could you (for example) buy a higher powered Mac (Maybe a Mac Studio) on a credit card and then return it within the return window for a full refund and apart from some time to set it up, effectively no cost at all.