vLLM Cheatsheet - High-Performance LLM Inference & Serving

vLLM differs from llama.cpp or Ollama not in whether a model runs but in its throughput model: PagedAttention manages KV cache in pages and continuous batching packs requests of different lengths into one batch, so the goal is maximum tokens per second inside a fixed memory budget. Three signals justify the operational cost - you need an OpenAI-compatible endpoint shared by several services, your request volume is high enough for batching to pay off, and you can afford the GPU and the upkeep. This sheet covers installation, launching vllm serve, client calls, quantisation and performance tuning. Three things genuinely need care: size the model against --gpu-memory-utilization first, then set --tensor-parallel-size to the card count (tensor parallelism is not data parallelism, and mismatching it slows things down), and finally choose a quantisation format against accuracy. AWQ and GPTQ are a memory decision, not a speed decision - quantised inference often runs slower, so switch only when memory is the binding constraint.

AI CLI·25 commands·Last updated 2026-08-23

Install 5

pip install vllm

Install vLLM (includes the vllm CLI)

pip install vllm --upgrade

Upgrade to the latest

vllm --version

Show installed version

python -c "import vllm; print(vllm.__version__)"

Confirm the package loads

vllm --help

Show all subcommands

OpenAI-compatible Server 5

vllm serve facebook/opt-125m

Start server (default :8000, OpenAI protocol)

vllm serve <model> --host 0.0.0.0 --port 8080

Set listen address and port

vllm serve <model> --api-key <key>

Enable API-key check (or VLLM_API_KEY env)

vllm serve <model> --tensor-parallel-size 2

Multi-GPU tensor parallelism (TP)

vllm serve <model> --gpu-memory-utilization 0.9

Raise VRAM util to fit larger models

Client Calls 5

curl http://localhost:8000/v1/models

List loaded models

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"...","messages":[...]}'

Send a chat request

openai_api_base=http://localhost:8000/v1

Point OpenAI SDK at local vLLM

vllm serve <model> --max-model-len 8192

Cap context length to save VRAM

vllm serve <model> --enforce-eager

Disable CUDA graph for stable debugging

Quantization & Perf 5

vllm serve <model> --quantization awq

Load AWQ-quantized weights

vllm serve <model> --quantization gptq

Load GPTQ-quantized weights

vllm serve <model> --enable-prefix-caching

Prefix cache for repeated prefixes

vllm serve <model> --max-num-seqs 256

Raise max concurrent sequences

vllm serve <model> --dtype half

Use fp16 to save VRAM (default auto)

Tips 5

Server defaults to :8000

Different from OpenAI default — fix client base_url

Big model? raise --gpu-memory-utilization first

Tune util before anything else

TP count = GPU count

Set tensor-parallel by card count

Debug with --enforce-eager

Rule out CUDA-graph weirdness

Quantization trades throughput/VRAM

awq/gptq balance precision vs cost

Tips

vLLM is a high-throughput inference engine (PagedAttention + continuous batching) that exposes open models over the OpenAI-compatible protocol — the default for self-hosted serving.
The server listens on :8000 by default (not OpenAI's :8080); clients just set base_url=http://localhost:8000/v1 to swap in seamlessly.
Low on VRAM? raise --gpu-memory-utilization first; use --tensor-parallel-size per GPU count for multi-card; debug oddities with --enforce-eager.
AWQ/GPTQ trade precision for cost; --enable-prefix-caching speeds up multi-turn/repeated-prefix workloads.

FAQ

How does vLLM differ from loading a model with Transformers directly?

vLLM implements PagedAttention and continuous batching, scheduling requests far more efficiently and delivering noticeably higher throughput under concurrency than naive per-request inference. The trade-offs are more aggressive memory management, a heavier startup, and less flexibility for custom inference logic. Use Transformers for single-request work or research; use vLLM when you are serving traffic.

What if the model does not fit in VRAM?

Try these in order: raise the allocation with --gpu-memory-utilization 0.9, then cap the context with --max-model-len 8192 — either one is often enough. If it still does not fit, load quantized weights with --quantization awq or gptq, accepting some accuracy loss that is usually tolerable. With multiple GPUs, use --tensor-parallel-size 2 for tensor parallelism, where the count generally matches the GPU count.

Should I use --enforce-eager when debugging odd problems?

Yes. --enforce-eager disables CUDA graph, narrowing the problem from "vLLM versus CUDA graph" down to the model itself. If the issue disappears, the fault lies in the CUDA graph layer and you should look at driver and CUDA versions; if it persists, the problem is in the model or your code. Enabling it during debugging saves a lot of time.

What port does the vLLM server use by default?

8000. OpenAI's own API lives in a different range, which is why an unmodified OpenAI SDK client fails to connect. Point base_url explicitly at http://localhost:8000/v1 when using the SDK. You can also choose your own port with --port 8080, and pair it with --host 0.0.0.0 to make the service reachable from other machines on the network.

How do I raise concurrent throughput?

Start with --max-num-seqs 256 to raise the concurrency ceiling, then add --enable-prefix-caching to reuse shared prefixes, which pays off clearly in multi-turn chat. Across multiple GPUs, --tensor-parallel-size 2 shards the model. Raising --gpu-memory-utilization 0.9 also fits more KV cache and indirectly improves concurrency.

Official References

Each command links to its official documentation below, so you can verify the latest usage and read deeper.

Maintained by LaoHand

Publicly updated on Aug 23, 2026, continuously proofread against official docs.

Contact Us

Wrong command or description? Send us corrections, business inquiries or product feedback by email.