💥 That One Time I Ran an AI/ML GPU Farm in My House
April 2022.
We were in the middle of a massive network latency and SQL optimization phase at Select Data. Our Data Science MicroK8s cluster (2 x Dell PowerEdge R740s with 2TB of RAM and 4 x Nvidia Tesla V100s) was acting up — constant I/O errors during experiments.
I had a hunch the latency issue might be the culprit. So, I made a call:
Shut it down and ship the hardware from our Phoenix colocation to Irvine.
Only one problem… I missed the call from remote hands asking how much to insure the shipment.
So they insured $60,000 worth of servers for $200.
(Yes, you read that right.)
The next call I got was from our Irvine colocation:
“Brian, you need to see this.”
I jumped on a plane the next morning. Catastrophe doesn’t even begin to describe it. The boxes looked like they had gone through a war zone. The servers were trashed. The GPUs, somehow, survived.
So, I did what any sane person would do…
I rebuilt the entire AI cluster in my garage.
At the time, I was already mining Ethereum, so I had a 30-amp circuit and a mini datacenter setup. Between spare parts, scrap hardware, and determination, I rebuilt the cluster:
- Ubuntu Server nodes (128GB RAM each)
- A mini-PC with 1TB NFS storage
- Four Tesla V100 GPUs running side by side
That cluster ran for months while we finished building out our new Dallas colocation.
Eventually, I migrated it — first physically, then virtually into our VMware ESXi environment.
That cluster is still running to this day.
Legal handled the reimbursement.
And I’ll never forget the sound of that garage humming like a jet engine while deep learning models trained through the night.
Sometimes innovation doesn’t happen in a boardroom or a lab — it happens in your garage, fueled by faith, duct tape, and a little bit of crazy.
All Systems (Are) Go. 