SGLang CLI Cheatsheet - sglang.launch_server Inference Full Reference

SGLang is a high-throughput inference and serving framework known for RadixAttention KV-cache reuse. python -m sglang.launch_server is its server entry with tensor and data parallelism plus quantization. This sheet covers launch, quantization, and performance flags so you can deploy an OpenAI-compatible server on a high-VRAM machine.

AI CLI·20 commands·Last updated 2026-09-10

Basic Launch 5

python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct

Minimal launch

python -m sglang.launch_server --model-path model --host 0.0.0.0 --port 30000

Set host and port

python -m sglang.launch_server --model-path model --tp 2

2-GPU tensor parallel

python -m sglang.launch_server --model-path model --dp 2 --tp 2

Data plus tensor parallel

python -m sglang.launch_server --model-path model --mem-fraction-static 0.7

Lower KV cache VRAM share

Quantization and Precision 5

python -m sglang.launch_server --model-path model --quantization fp8

Weight fp8 quantization

python -m sglang.launch_server --model-path model --kv-cache-dtype fp8_e4m3

KV cache fp8 quantization

python -m sglang.launch_server --model-path model --dtype bfloat16

Set compute dtype

python -m sglang.launch_server --model-path model --context-length 32768

Extend context length

python -m sglang.launch_server --model-path model --trust-remote-code

Allow custom model code

Performance and Advanced 10

python -m sglang.launch_server --model-path model --chunked-prefill-size 4096

Chunked prefill for long prompts

python -m sglang.launch_server --model-path model --enable-torch-compile

Enable torch.compile

python -m sglang.launch_server --model-path model --max-running-requests 64

Raise concurrent requests

python -m sglang.launch_server --model-path model --schedule-policy lpm

Longest-prefix-match scheduling

python -m sglang.launch_server --model-path model --config config.yaml

Launch from a config file

python -m sglang_router.launch_server --model-path model --dp 2 --tp 2

Data parallel via Router

python -m sglang.launch_server --help

All flags

python -m sglang.launch_server --model-path model --nnodes 2

Multi-node tensor parallel

python -m sglang.launch_server --model-path model --enable-deterministic-inference

Deterministic inference

python -m sglang.launch_server --model-path model --disable-cuda-graph

Disable CUDA Graph (debug)

Tips

--tp splits across GPUs; if one card is too small, add cards or lower --mem-fraction-static or quantize.
RadixAttention is on by default; repeated prefixes (system prompt, RAG context) are reused automatically.
On OOM, try --quantization fp8 or --mem-fraction-static 0.8 first.

FAQ

How is SGLang different from vLLM?

Both are high-throughput frameworks; SGLang leads with RadixAttention prefix reuse and structured generation, vLLM with PagedAttention memory management. Both serve OpenAI-compatible endpoints.

What protocol does it expose?

OpenAI-compatible /v1/chat/completions and /generate; point a client base_url at it.

Official References

Each command links to its official documentation below, so you can verify the latest usage and read deeper.

SGLang Docs

Maintained by LaoHand

Publicly updated on Sep 10, 2026, continuously proofread against official docs.

Contact Us

Wrong command or description? Send us corrections, business inquiries or product feedback by email.