LLM Hardware Decision Dashboard

Key Insight: Laptop vs Terminal

• If you work with private client data — laptop needs at least 64GB (32B models run locally, no cloud)
• If you travel/work from office — 128GB laptop means no dependency on home rack being reachable
• If you're always home/office with good internet — 48GB laptop + powerful rack saves $4,000+
• The 64GB "sweet spot": Our benchmark shows 16-32GB tier scored 90% — Qwen3 Coder Next runs on 16GB and beat 72B models. A 64GB laptop with smart model selection matches 128GB for most tasks.

RDMA: The Game Changer

Without RDMA, adding cluster nodes makes LLM inference slower on 10GbE (Jeff Geerling: 4 nodes TCP = 15.2 t/s, worse than 1 node at 20.4 t/s). With TB5 RDMA: 31.9 t/s — 2.1x faster.

New data (2026-04): macOS Tahoe 26.2 RDMA + Exo 1.0 — DevStroll 123B dense goes from 9.2 to 22 tok/s on 4x M3 Ultra (2.4x scaling). But Qwen 235B MoE only scales 1.2x (30 to 37 tok/s) — dense models benefit far more than MoE from tensor parallelism.

However: Alex Ziskind showed that upgrading Framework nodes from 10GbE to 25GbE PCIe x4 NICs significantly improves multi-node performance. Framework 4-node cluster achieves ~13 tok/s gen / ~152 tok/s prefill on MiniMax M2 Q6, with ~20% gen drop from 2 to 4 nodes. For teams, Framework + 25GbE is still the best value.

Sources: Jeff Geerling RDMA benchmark | Apple TN3205 | Alex Ziskind benchmarks (2026-04)

B70 Budget Analysis: 128GB VRAM for $4,000

• vLLM >> llama.cpp for professional GPUs — llama.cpp is optimized for consumer/Apple hardware, vLLM leverages tensor parallelism across GPUs properly
• B70 wins on BF16 concurrency — 194 tok/s at C4 beats both RTX Pro 4000 (173) and R9700 (149)
• RTX Pro 4000 wins on AWQ quantization — 89 tok/s vs B70's 72, CUDA quant kernels are more mature
• R9700 AWQ is broken — only 25 tok/s in AWQ mode (ROCm quant support is poor)
• Prefill is fast everywhere — 10K-13K tok/s on all three GPUs, not the bottleneck
• For pure VRAM capacity, 4x B70 at $4K is cheaper than a single Framework node ($3,759) and uses GPU memory bandwidth (600 GB/s per card) instead of CPU DDR5 (256 GB/s total)

Dense vs MoE Cluster Scaling

• Dense models benefit dramatically from tensor parallelism — 2.4x on 4 nodes for DevStroll 123B. Every node does useful compute.
• MoE models scale poorly — only 1.2x on 4 nodes for Qwen 235B. Active expert count is fixed regardless of node count; extra nodes just hold dormant parameters.
• RDMA is the enabler — without it (TCP over 10GbE), multi-node inference is slower than single node. macOS Tahoe 26.2 RDMA over Thunderbolt makes clustering viable.
• Power efficiency — 66W total for a 4-node cluster running 123B dense at 22 tok/s. Compare to GPU rigs at 300-1200W for similar capability.
• Cluster rule: pick dense models for multi-node. If you have enough VRAM on one node, MoE is fine. If you must distribute, use dense architectures.

DGX Spark Takeaways

• Generation scales linearly — 23 to 51 tok/s from 1 to 4 nodes (2.2x), because generation is memory-bandwidth bound and more nodes = more bandwidth
• Prefill regresses past 2 nodes — prefill is compute-bound, and coordination overhead across ConnectX-7 RoCE eats into the gains
• DGX Spark is a reference point, not a recommendation — pricing is datacenter-tier. Framework or Mac clusters deliver comparable per-dollar performance for homelabs.
• ConnectX-7 + RoCE at ~3us latency is the gold standard for GPU clustering — but TB5 RDMA at ~50us is close enough for inference (not training)

Framework Cluster Reality Check

• 13 tok/s generation on MiniMax M2 Q6 — usable for batch/async workloads, not ideal for interactive chat
• Generation drops ~20% from 2 to 4 nodes — network overhead is real, even at 25GbE. This is why RDMA matters.
• Prefill at 152 tok/s is solid — prompt processing is fast enough for most use cases
• 25G NICs via PCIe x4 are essential — Thunderbolt NICs max out at 10G, which kills multi-node performance (Jeff Geerling's original finding)
• For teams, Framework clusters are still the best value — 13 tok/s shared across users with queue management beats paying $50/user/mo for SaaS

Shared Infrastructure Stack

Subscription Replacement Calculator

Replace per-seat SaaS AI subscriptions with self-hosted infrastructure.

User Profiles by Role

For teams: Framework Desktop + 100GbE NICs is the best value. $9,503 for 2-node 256GB cluster serves 30 people. Add $1,600 for 100GbE NICs (2×$800) for near-Mac performance. Total: $11,103 — still 25% cheaper than one Mac Studio Ultra.

Open WebUI handles all roles — admin assigns model tiers per user group. Developers get 70B access, admin staff gets 7B-14B (faster, cheaper). All data stays on-premises.

ROI Example: 20-person team

* Includes hardware ($9,503) + electricity ($1,800). Break-even: 10 months.

Client Sizing: 30-Person Company

Key insight: 70% of roles (admin, accountants, managers) score 90%+ on 7B-14B models. They dont need expensive laptops. One central 128GB server running 70B for devs + 14B for everyone else serves the entire company. Most employees keep their existing PCs and access via Open WebUI in the browser.

Bottom Line

• Effective cost: $130/mo for the entire setup (after resale + tax deduction)
• Less than a Claude Max subscription — and you get 256GB of local compute
• One client win ($5K-50K engagement from a self-hosted LLM demo) pays for everything
• AppleCare+ at $189/yr is non-negotiable — it preserves the 75% resale value that makes the ROI work
• The laptop and rack complement each other — portable private AI + always-on heavy inference. Neither alone is optimal.