Optimized vLLM inference with MoE, AWQ 4-bit and MTP on the DGX Spark (GB10, Grace Blackwell ARM64).
Run Qwen3.6-35B-A3B (Mixture of Experts) on the DGX Spark at maximum throughput for agents, n8n and OpenWebUI. Tool calling via native OpenAI tool_calls.
cd /srv/vllm
git clone https://github.com/hermes-speedboat/vllm.qwen3.6-35b_mtp_spark.git .
bash setup_vllm.sh # venv + vllm + huggingface-hub
bash download_model.sh # 24 GB model from HF
bash vllm-server.sh # start server on port 8000
Qwen3.6-35B-A3B has 35B parameters total but only 3B active parameters per token.
256 experts total, 8 active per token + 1 shared expert. 40 layers (mixed: DeltaNet Linear-Attention + Full-Attention).
Advantage over a dense 27B model: ~10x less memory bandwidth required → significantly higher throughput on bandwidth-constrained hardware like the DGX Spark.
Accuracy-aware quantization to 4-bit (group_size=32). Model weighs 24 GB total on disk, ~1.5 GB active per forward pass.
vLLM flags: --quantization awq --dtype float16
Reason for float16 over bfloat16: AWQ quantization requires float16 compute dtype. Using bfloat16 (the default with --dtype auto) produces an error.
Qwen3.6 was trained with multi-token prediction heads. vLLM uses this for speculative decoding:
MTP is baked into the model architecture (config.json: "mtp_num_hidden_layers": 1). The relevant vLLM flag is --speculative-config '{"method":"mtp","num_speculative_tokens":2}'.
Note: The feature was renamed from qwen3_next_mtp to mtp in vLLM 0.22.1. The old name is deprecated.
qwen35 GGUF architecture is also unsupported by the transformers GGUF parser.Qwen3.6 generates internal reasoning (thinking tokens) by default. No suppression is applied — the model generates full thinking in responses.
Hermes (and most agent frameworks) handles thinking tokens without issues (same as DeepSeek, etc.).
All parameters are controlled via environment variables in vllm-server.sh:
PORT (8000): Server portMAX_MODEL_LEN (262144): Maximum context length (up to 262144)GPU_MEM_UTIL (0.65): GPU memory utilization fractionLANGUAGE_ONLY (true): Disable vision encoderNUM_SPEC_TOKENS (2): MTP speculative tokens per stepMODEL_REPO (cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit): Hugging Face model repositoryThe model is served as cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit via --served-model-name. Use this name in API requests and Hermes config (hermes config set model.default cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit).
Tool/function calling is fully supported for agent frameworks (Hermes, n8n, etc.):
--enable-auto-tool-choice: Allow automatic tool selection from request--tool-call-parser qwen3_coder: Parse Qwen3-style tool calls into native OpenAI tool_callsNote: Qwen3.6 uses a different tool-call format than Qwen2.5. The hermes parser does not work — use qwen3_coder instead. This was confirmed via NVIDIA DGX Spark forums where other users reported the same issue.
The first request after server start may have a latency spike (CUDA graph compilation). vLLM warns:
causes a latency spike; consider extending warmup to cover this shape/config.
Mitigated with --enforce-eager, which skips CUDA graph optimization entirely. On the DGX Spark the throughput difference is negligible.
D-Bus user session must be available. On headless servers (SSH-only), set up:
# Install dbus if missing
sudo apt install dbus dbus-user-session
# Enable linger for user services
sudo loginctl enable-linger $USER
# Add to ~/.bashrc for persistent session bus
echo 'export DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/$(id -u)/bus"' >> ~/.bashrc
source ~/.bashrc
mkdir -p ~/.config/systemd/user/
cp vllm.service ~/.config/systemd/user/
chmod +x /srv/vllm/vllm-server.sh
# Fix ownership if .venv was created with sudo
[ ! -d /srv/vllm/.venv ] || sudo chown -R $USER:$USER /srv/vllm/.venv
systemctl --user daemon-reload
systemctl --user enable --now vllm
203/EXEC Fehler: Service startet nicht? → chmod +x vllm-server.sh + .venv Ownership fixen, dann systemctl --user restart vllm.
systemctl --user status vllm
journalctl --user -u vllm -f
systemctl --user stop vllm
systemctl --user restart vllm
All setup files are on GitHub:
Repo: hermes-speedboat/vllm.qwen3.6-35b_mtp_spark
| File | Purpose |
|---|---|
setup_vllm.sh |
Install uv, venv, vllm, huggingface-hub |
download_model.sh |
Download model via snapshot_download() |
vllm-server.sh |
Start vLLM with optimized flags |
vllm.service |
Systemd user service file |
test_vllm.sh |
API functional test |
--torch-backend=autovLLM + transformers GGUF parser does not support newer architectures like qwen35. Solution: use PyTorch/safetensors format via snapshot_download().
ValueError: torch.bfloat16 is not supported for quantization method awq
Solution: always set --dtype float16 explicitly. Do not rely on --dtype auto with AWQ models.
Clients (OpenWebUI) probing Ollama-compatible endpoints produce harmless 404s:
GET /api/v1/models 404
GET /api/tags 404
GET /v1/props 404
These are not server errors. Configure OpenWebUI to use an OpenAI-compatible endpoint (vLLM), not Ollama API path. The vLLM server only serves OpenAI-compatible endpoints at /v1/models, /v1/chat/completions, etc.
Triton kernel JIT compilation during inference
One-time warning on first request. Kernel is cached by Triton, warning disappears on subsequent requests.