Case study 03 / ML Engineering · Software Engineering
Understanding the cost of inference.
A reproducible comparison of vLLM serving configurations and a serial HuggingFace baseline on the same GPU and workload.
The problem
How much do batching and quantized kernels change LLM serving throughput, latency, memory use, and cost?
What I built
Built a FastAPI baseline with an OpenAI-compatible streaming endpoint, scripts for concurrency sweeps, a retrospective cost model, and Prometheus/Grafana monitoring. Used vLLM’s vendored load generator for consistent request measurement.
How it works
The same benchmark client calls vLLM or a serial HuggingFace server. Runs compare fp16, AWQ, and GPTQ at concurrency 1, 4, 8, 16, and 32. Reports translate measured output throughput into cost using the recorded GPU rental rate.
- ShareGPT requests
- Concurrency sweep
- Serving engines
- Metrics & cost
Engineering decisions
A fair serial baseline
Corrected dtype and attention defaults that had silently slowed the baseline. Kept serialization and lack of continuous batching as the intended comparison.
Check the actual kernel
Investigated an AWQ kernel-configuration problem. The report documents how using awq_marlin changed the performance ranking.
Show the memory tradeoff
The baseline uses less peak GPU memory because it does not reserve a large KV-cache pool. Higher concurrent throughput comes with a different memory allocation strategy.
Results & evidence
AWQ output throughput
2,187.9 tok/s
Llama-3.2-3B-Instruct, RTX A5000 24 GB, vLLM 0.8.5, concurrency 32, 100 ShareGPT requests; repository-reported result.
Baseline peak memory
6.8 GiB
Compared with 21.1 GiB for AWQ. The serial baseline processes one request at a time; this is not an equal-batching comparison.
Results are documented in the linked project artifacts. They have not been independently reproduced for this portfolio.
Limits & lessons
Measurements describe the recorded hardware, models, workloads, and engine version. Raw run outputs are git-ignored; the published report contains the result tables. Cost estimates use the historical rental price, not a current cloud quote.
Performance work starts with understanding the baseline and the configuration actually executed.