Blackwell GPUs Achieve 21x Faster RAG Inference for Private LLMs in SMEs

Small and medium-sized enterprises are increasingly exploring alternatives to cloud-based large language model (LLM) APIs due to growing data privacy concerns. Jonathan Knoop from IE Business University and Hendrik Holtmann demonstrate the viability of cost-effective, on-premise LLM inference using consumer-grade Blackwell GPUs. Their systematic evaluation of RTX 5060 Ti, 5070 Ti, and 5090 GPUs, benchmarking models such as Qwen3-8B and Gemma3-27B, reveals significant performance and cost benefits compared to cloud solutions. This research is particularly significant as it demonstrates that self-hosted inference can reduce costs by a factor of 40-200x, with hardware amortisation achievable within four months, offering SMEs a practical pathway to maintain data sovereignty without prohibitive infrastructure expenses.

The research team systematically evaluated these GPUs, benchmarking four open-weight models, Qwen3-8B, Gemma3-12B, Gemma3-27B, and GPT-OSS-20B, across 79 distinct configurations. These configurations varied quantization formats (BF16, W4A16, NVFP4, MXFP4), context lengths ranging from 8k to 64k tokens, and three representative workloads: retrieval-augmented generation (RAG), multi-LoRA agentic serving, and high-concurrency APIs. This comprehensive evaluation employed vLLM and AIPerf to meticulously measure throughput, latency, and energy consumption, providing a detailed performance profile for each GPU and model combination.

The study reveals that the RTX 5090 achieves 3.5 to 4.6times higher throughput than the RTX 5060 Ti, coupled with a substantial 21-fold reduction in latency specifically for RAG applications. However, the team discovered that budget GPUs offer the most cost-effective throughput-per-dollar for API workloads, delivering sub-second latency. Crucially, the implementation of NVFP4 quantization yielded a 1.6-fold increase in throughput compared to BF16, alongside a 41% reduction in energy consumption, with minimal impact on model quality, experiencing only 2-4% quality loss. This optimisation significantly enhances the efficiency of LLM inference on consumer hardware.

Experiments show that self-hosted inference costs range from $0.001 to $0.04 per million tokens, based solely on electricity costs, representing a remarkable 40 to 200times reduction in expense compared to budget-tier cloud APIs. The research establishes that hardware investments can break even in under four months at moderate usage volumes, approximately 30 million tokens per day, making local deployment a financially attractive option for SMEs. The work conclusively demonstrates that consumer GPUs can reliably replace cloud inference for the majority of SME workloads, with the exception of latency-critical, long-context RAG applications, where high-end GPUs remain essential. To facilitate wider adoption, the researchers provide detailed deployment guidance and have released all benchmark data, ensuring reproducibility and enabling SME-scale deployments. This comprehensive study addresses a critical gap in understanding consumer GPU performance for production LLM inference, offering a practical guide for cost-effective and privacy-preserving local deployment strategies for small and medium-sized enterprises. The team’s methodology is structured around four key research questions, focusing on throughput and latency, quantization trade-offs, agentic overheads, and energy/cost analysis, providing a robust foundation for future research and development in this rapidly evolving field.

Blackwell GPU Benchmarking for LLM Inference Workloads

The study systematically evaluated NVIDIA’s Blackwell consumer GPUs, specifically the RTX 5060 Ti, 5070 Ti, and 5090, to determine their viability for production-level large language model (LLM) inference within small and medium-sized enterprises. Researchers engineered a comprehensive benchmarking suite encompassing 79 configurations, meticulously varying quantization formats including BF16, W4A16, NVFP4, and MXFP4, alongside context lengths ranging from 8k to 64k tokens. This detailed analysis spanned three distinct workloads: retrieval-augmented generation (RAG), multi-LoRA agentic serving, and high-concurrency APIs, allowing for a nuanced understanding of performance characteristics across diverse applications. Experiments employed the vLLM framework and NVIDIA’s AIPerf tools to precisely measure throughput, latency, and energy consumption.

The team focused on answering four key research questions, beginning with a comparative analysis of token-per-second (TPS), time-to-first-token (TTFT), and tail latencies across single and dual-GPU setups for each workload. Further investigation explored the trade-offs associated with different low-precision quantization formats, assessing their impact on throughput, memory footprint, and energy efficiency while maintaining acceptable model quality. To simulate realistic multi-agent systems, the study quantified the overhead introduced by frequent LoRA adapter switching, evaluating vLLM’s adapter management capabilities on commodity hardware. Finally, researchers calculated energy consumption (Wh/MTok) and estimated electricity costs per million tokens, providing a detailed cost analysis for SME-relevant local deployments.

The model suite included Qwen3-8B, Gemma3-12B, Gemma3-27B, and GPT-OSS-20B, representing diverse open-weight LLMs from Chinese and US organizations, ensuring broad applicability and representation within the open-weight ecosystem. This methodology enabled the team to demonstrate that consumer GPUs can reliably replace cloud inference for many SME workloads, achieving cost parity with commercial APIs within one to four months at moderate usage levels and subsequently operating at 40, 200× lower cost. All code, configurations, and Docker images were released to ensure reproducibility and facilitate wider adoption of self-hosted LLM inference solutions.

Blackwell GPUs benchmarked for LLM inference performance

The research team systematically evaluated Blackwell consumer GPUs, RTX 5060 Ti, 5070 Ti, and 5090, for production large language model (LLM) inference, benchmarking four open-weight models including Qwen3-8B, Gemma3-12B, Gemma3-27B, and GPT-OSS-20B. Experiments spanned 79 configurations, varying quantization formats (BF16, W4A16, NVFP4, MXFP4), context lengths from 8k to 64k, and three distinct workloads: retrieval-augmented generation (RAG), multi-LoRA agentic serving, and high-concurrency APIs. The RTX 5090 achieved 3.5-4.6x higher throughput than the RTX 5060 Ti, coupled with a 21x reduction in latency specifically for RAG applications, demonstrating its superior performance in demanding scenarios. However, the study revealed that budget GPUs deliver the highest throughput-per-dollar for API workloads, maintaining sub-second latency, and offering a cost-effective solution for less latency-sensitive tasks.

Crucially, NVFP4 quantization provided a 1.6x increase in throughput compared to BF16, alongside a 41% reduction in energy consumption, with only a minimal 2-4% loss in model quality. Measurements confirm that self-hosted inference incurs costs ranging from $0.001 to $0.04 per million tokens, utilising electricity costs only, representing a cost reduction of 40-200x compared to budget-tier cloud APIs. The hardware is projected to break even in under four months at a moderate volume of 30 million tokens per day, highlighting the potential for significant long-term savings. Results demonstrate that consumer GPUs can reliably replace cloud inference for the majority of small and medium-sized enterprise (SME) workloads, with the exception of latency-critical, long-context RAG applications, where the higher performance of high-end GPUs remains essential. The team measured throughput, latency, and energy consumption using vLLM and AIPerf, providing a comprehensive analysis of performance across diverse configurations and workloads. All benchmark data and deployment guidance have been released to facilitate reproducible SME-scale deployments.

Blackwell GPUs enable cost-effective LLM inference

This research demonstrates that Blackwell consumer GPUs offer a viable alternative to cloud-based large language model (LLM) inference for many small and medium-sized enterprises (SMEs). A systematic evaluation of RTX 5060 Ti, 5070 Ti, and 5090 GPUs, combined with four open-weight models, revealed substantial performance differences based on GPU selection, quantization format, context length, and workload type. The RTX 5090 achieved the highest throughput for retrieval-augmented generation (RAG) tasks, while the more affordable GPUs proved cost-effective for API-based workloads. The findings establish that NVFP4 quantization consistently outperforms BF16, delivering increased throughput and reduced energy consumption with minimal quality loss.

Furthermore, the study highlights the significant impact of context length on cost, advocating for semantic chunking to maintain efficiency. Self-hosted inference on consumer GPUs offers a considerable cost reduction, ranging from 40 to 200times cheaper than cloud APIs, with hardware costs recouped within a few months at moderate usage levels. The authors acknowledge limitations including the use of synthetic workloads and the exclusion of CPU/memory/cooling overhead in energy measurements. Future research should focus on evaluating performance with real-world request patterns and exploring alternative inference engines to further optimise deployment strategies.