Hacking 32GB VRAM: The Multi-GPU Turing Cluster on Frugal Hardware
Running local LLMs with 32k+ context shouldn't require renting cloud A100s or spending $10,000 on enterprise accelerators. We engineered a 32GB GDDR6 inference cluster inside a repurposed HPE ProLiant DL380p Gen8 server by pairing a desktop RTX 2060 12GB with two decommissioned NVIDIA CMP 50HX crypto mining cards (10GB each).
Mining silicon famously lacks display outputs and has aggressive power configurations drawing ~240W at idle. Through custom MSI VBIOS flashing, hardware strapping, and an aggressive Linux driver P-state governor, we slashed idle power draw down to a near-silent ~33W.
We compiled llama.cpp with CUDA 12 and NCCL tensor parallelism across all three Turing GPUs, achieving 325–500 tok/s prompt prefill speeds. To complete the stack, we wrote ollama-proxy.py—an intelligent API gateway offering dynamic reasoning/thinking control (think_budget=0 fast chat vs full dynamic chain-of-thought) compatible with Ollama, OpenAI, and Anthropic tooling.
git clone https://github.com/wmantly/turing-multi-gpu-llm-server.git