Set Up DGX Spark with Qwen 3.6 27B Dense

Hardware Overview

• Model: NVIDIA DGX Spark
• GPU: GB10 (Grace-Blackwell, aarch64)
• Unified Memory: 128 GB
• CUDA: 13.0, Driver 580.159.03
• IP: 192.168.50.137 (LAN)
• User: anduin (uid 1001)

Security Configuration

• SSH: YubiKey ECDSA-SK only; password login and root login are disabled
• Firewall: ufw allows only 22/tcp and 9000/tcp
• Sleep: All sleep/suspend/hibernate/hybrid-sleep targets are masked

Model Selection

Why choose this: 128GB of VRAM is more than sufficient for a 27B model in BF16 (~54GB for weights + ~30GB for KV cache @256K + system overhead ≈ 90-95GB). BF16 ensures zero loss in quality for translation, summarization, and tool calling. The license is clean, with no commercial risk.

Deployment Architecture

  Client (LAN) ──→ OllamaGateway (ultra) ──→ vLLM :9000 (DGX Spark)
     │                   │                          │
     │  Ollama API       │  Virtual Model           │  OpenAI-compatible API
     │  /api/chat        │  dgx-qwen → qwen36-27b   │  /v1/chat/completions

• DGX Spark is solely responsible for inference and exposes the standard OpenAI API
• OllamaGateway handles authentication/audit/model aliases/thinking toggle
• The two machines communicate over a local area network with no external dependencies

Environment Preparation

1. Remove the Desktop Environment (GDM)

The desktop environment (Xorg + gnome-shell) uses ~320MB of GPU VRAM, which causes vLLM startup to fail at 95% utilization:

sudo systemctl stop gdm3
sudo systemctl disable gdm3
sudo systemctl set-default multi-user.target

2. Disable Sleep and Hibernation

sudo systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target

3. Firewall

sudo ufw allow 22/tcp
sudo ufw allow 9000/tcp
sudo ufw enable

4. SSH Hardening

Edit /etc/ssh/sshd_config:

PasswordAuthentication no
PermitRootLogin no

Then run sudo systemctl reload sshd.

5. New Virtual Environment

rm -rf ~/llm/.venv ~/llm/models ~/llm/logs
python3 -m venv ~/llm/.venv
source ~/llm/.venv/bin/activate

6. Install vLLM

Use the PyTorch extra index for CUDA 13; use PyPI for vLLM:

pip install --upgrade pip wheel
pip install vllm --extra-index-url https://download.pytorch.org/whl/cu13

Verification:

python -c "import vllm; print(vllm.__version__)"  # 0.21.0
python -c "import torch; print(torch.__version__); print(torch.cuda.is_available())"  # 2.11.0, True

Download Model

huggingface-cli is deprecated; use hf instead:

pip install huggingface_hub[hf_transfer]
hf download Qwen/Qwen3.6-27B-BF16 --local-dir ~/llm/models/Qwen3.6-27B-BF16

Download size ~52 GB (15 safetensor shards + tokenizer/config files). If rate-limited:

export HF_TOKEN="hf_xxx"  # 可选，提升限速阈值

Service Script

~/llm/serve_qwen.sh:

#!/usr/bin/env bash
# Qwen 3.6 27B BF16 — 翻译/归纳/闲聊 API 服务器
# 策略：零量化，聪明优先，128GB 物尽其用

set -uo pipefail
HOME_LLM=/home/anduin/llm
source "$HOME_LLM/.venv/bin/activate"
export PATH="$HOME/.local/bin:/usr/local/cuda/bin:$PATH"

# 停掉旧进程
pkill -9 -f "vllm serve" 2>/dev/null
pkill -9 -f EngineCore 2>/dev/null
sleep 3

MODEL_DIR="$HOME_LLM/models/Qwen3.6-27B-BF16"

exec vllm serve "$MODEL_DIR" \
  --served-model-name qwen36-27b \
  --host 0.0.0.0 --port 9000 \
  --trust-remote-code \
  --dtype bfloat16 \
  --kv-cache-dtype bfloat16 \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.95 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --max-num-seqs 4 \
  --max-num-batched-tokens 8192 \
  --max-model-len 262144 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --enable-auto-tool-choice

Parameter Explanation

Thinking Control

vLLM's Qwen 3.6 reasoning parser is controlled via chat_template_kwargs:

# 不思考 (翻译/归纳/闲聊)
curl http://192.168.50.137:9000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen36-27b",
    "messages": [{"role": "user", "content": "你好"}],
    "chat_template_kwargs": {"enable_thinking": false}
  }'

# 思考 (深度推理)
curl http://192.168.50.137:9000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen36-27b",
    "messages": [{"role": "user", "content": "解这道数学题..."}],
    "chat_template_kwargs": {"enable_thinking": true}
  }'

When Thinking=true, the reasoning process is returned in the choices[0].message.reasoning field; when thinking=false, only content is returned. No model reload is required, as it can be toggled per request.

systemd Auto-Start

~/.config/systemd/user/vllm-qwen.service:

[Unit]
Description=vLLM Qwen 3.6 27B BF16 API Server
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/home/anduin/llm/serve_qwen.sh
ExecStop=/usr/bin/pkill -9 -f 'vllm serve'
Restart=always
RestartSec=10
WorkingDirectory=/home/anduin/llm
StandardOutput=append:/home/anduin/llm/logs/serve.log
StandardError=append:/home/anduin/llm/logs/serve.log
Environment=HOME=/home/anduin
Environment=XDG_RUNTIME_DIR=/run/user/1001

[Install]
WantedBy=default.target

Enable:

sudo loginctl enable-linger anduin    # 开机无登录也启动
systemctl --user daemon-reload
systemctl --user enable vllm-qwen
systemctl --user start vllm-qwen

Start Time

Once the model is resident in memory, API responses are immediate with no warm-up required.

Measured Performance

27B BF16, non-streaming, thinking=off:

The bottleneck lies in the GB10 unified memory bandwidth (~500 GB/s), not in compute capability.

Verified Features

• OpenAI-compatible API (/v1/chat/completions, /v1/models, /health)
• Thinking toggle (per-request, enable_thinking: true/false)
• Tool calling (--tool-call-parser qwen3_coder)
• Streaming (SSE/NDJSON)
• systemd auto-start / auto-restart
• Auto-start on boot (linger enabled)
• SSH YubiKey only (password login/root disabled)
• Firewall (ufw: 22 + 9000)

Debug Commands

# 服务状态
systemctl --user status vllm-qwen
journalctl --user -u vllm-qwen -f

# 端口检查
ss -tlnp | grep 9000

# NVIDIA 状态 (DGX Spark 用 nvidia-smi 可能输出有限)
nvidia-smi

# 内存使用
free -h

# 实时日志
tail -f ~/llm/logs/serve.log

# 健康检查
curl -s http://localhost:9000/health

# 模型列表
curl -s http://localhost:9000/v1/models | python3 -m json.tool

# 杀掉僵死进程
pkill -9 -f "vllm serve"
pkill -9 -f EngineCore

Recovery Process

If the API does not respond after the machine restarts:

1. ssh [email protected]
2. systemctl --user status vllm-qwen — Check the status
3. tail -50 ~/llm/logs/serve.log — View the logs
4. Confirm that GDM is not active: sudo systemctl is-active gdm3
5. Confirm the firewall status: sudo ufw status | grep 9000
6. If GDM is active: sudo systemctl stop gdm3 && systemctl --user restart vllm-qwen
7. Wait for ~150s for the cold start

Memory Budget

The remaining capacity is sufficient, and there is no risk of swap. If longer context or larger batches are needed in the future, there is still room for expansion.