Two Qwen3 models on one DGX Spark: the residency math
Running multiple large language models on a single GPU requires precise memory management—the author demonstrates that gpu_memory_utilization is calculated against total GPU memory (not free memory), and co-resident models must sum to ~0.95 or less to avoid crashes and KV cache starvation. For IT organizations deploying local LLM backends, this means careful capacity planning, architectural choices around model selection (reasoning vs. instruct variants), and empirical testing are essential, as theoretical calculations often diverge from actual residency patterns due to CUDA overhead and model-specific state requirements.
Hacker News3 min read
Running multiple large language models on a single GPU requires precise memory management—the author demonstrates that gpu_memory_utilization is calculated against total GPU memory (not free memory), and co-resident models must sum to ~0.95 or less to avoid crashes and KV cache starvation. For IT organizations deploying local LLM backends, this means careful capacity planning, architectural choices around model selection (reasoning vs. instruct variants), and empirical testing are essential, as theoretical calculations often diverge from actual residency patterns due to CUDA overhead and model-specific state requirements.