MLC-LLM CLI Cheatsheet - mlc_llm Compile and Serve Full Reference

MLC-LLM compiles popular open models into efficient local GPU (CUDA, Metal, Vulkan) runtimes. mlc_llm is its unified CLI: chat interactively, serve an OpenAI-compatible endpoint, or convert a HuggingFace model into deployable weights. This sheet covers the three subcommands and common flags so you can launch local inference.

AI CLI·20 commands·Last updated 2026-09-10

Interactive Chat 5

mlc_llm chat HF://mlc-ai/Llama-3-8B-Instruct-q4f16_1-MLC

Chat a precompiled model directly

mlc_llm chat model --device cuda

Run on NVIDIA GPU

mlc_llm chat model --device mac

Run on Apple Silicon

mlc_llm chat model --device metal

Use the Metal backend explicitly

mlc_llm chat model --quantization q4f16_1

Choose a quantization scheme

Serve and Convert 5

mlc_llm serve model

Start a local OpenAI-compatible server

mlc_llm serve --device cuda

Serve with the CUDA backend

mlc_llm convert model --quantization q4f16_1

Convert and quantize a model

mlc_llm convert model --model-lib path

Point at a prebuilt model lib

mlc_llm serve --host 0.0.0.0 --port 8000

Expose the server port externally

Flags and Help 10

mlc_llm chat model --overrides "{}"

Override generation params via JSON

mlc_llm chat model --temperature 0.7

Set sampling temperature

mlc_llm chat model --top-p 0.95

Set nucleus sampling

mlc_llm chat model --max-gen-len 1024

Cap max generation length

mlc_llm chat model --device vulkan

Run with the Vulkan backend

mlc_llm serve --port 8000

Custom server port

mlc_llm convert --revision main

Pick the conversion branch

mlc_llm chat --help

All chat flags

mlc_llm serve --help

All serve flags

mlc_llm --help

Top-level commands

Tips

For local GPU inference prefer mlc_llm chat model --device cuda (or mac); it beats pure CPU by far.
Use mlc_llm serve to expose an OpenAI-compatible API for client tools.
convert quantizes a HuggingFace model into MLC format; convert once, then serve offline repeatedly.

FAQ

How is MLC-LLM different from llama.cpp?

MLC-LLM uses a compilation route (TVM/Relax) for unified cross-hardware deployment; llama.cpp is a hand-written C++ engine. Both run local models.

What protocol does serve expose?

It is OpenAI chat-completions compatible; point a client base_url at the serve address to call it directly.

Official References

Each command links to its official documentation below, so you can verify the latest usage and read deeper.

MLC-LLM Docs

Maintained by LaoHand

Publicly updated on Sep 10, 2026, continuously proofread against official docs.

Contact Us

Wrong command or description? Send us corrections, business inquiries or product feedback by email.