DeepSeek V4 Flash on a single AMD MI300X

DeepSeek V4 Flash on a single AMD MI300X

• ID: 07048389
• 原文链接: https://github.com/ryanzhou/deepseek-v4-flash-mi300x
• 作者 / 日期: Ryan Zhou | 2026-08-05
• 分类: infra
• 来源类型: github
• 标签: deepseek, mi300x, vllm, rocm, inference
• 质量评分: 4/5
• 抓取时间: 2026-08-05T15:45:26.663004+00:00

中文导读

这个仓库给出单张 AMD MI300X 部署 DeepSeek V4 Flash 的生产配置样本，覆盖 vLLM ROCm nightlyAITERFP8DSparkKV cacheCaddySHA-256 pin 与 smoke testClawFeed 摘要记录作者报告单流解码 168.6 tok/s8 并发 542 tok/s aggregate64-stream burst 830 tok/s aggregate，权重 156.67 GiB 放入 HBM，未额外量化或 offload它的价值在可复现配置版本 pin启动日志和 ROCm/MI300X 坑点，而不只是能跑模型

为什么值得关注

这类部署仓库有价值，因为它把 ROCm/vLLM/FP8/KV cache 的生产配置和坑点都摊开了

English Summary

The repository documents a reproducible single-AMD-MI300X deployment setup for DeepSeek V4 Flash using ROCm/vLLM/AITER, FP8, KV cache sizing, Caddy, pinned hashes, startup checks, and throughput notes.

Obsidian Evidence

候选来自 OpenClaw定时任务/ClawFeed24小时高价值一览/2026-08-05-ClawFeed24小时高价值一览.md 的 ClawFeed 精选。

Source Extract / Metadata

ryanzhou/deepseek-v4-flash

原文链接: https://github.com/ryanzhou/deepseek-v4-flash-mi300x

DeepSeek V4 Flash on a single AMD MI300X

#deepseek-v4-flash-on-a-single-amd-mi300x

This repository contains the configuration and patches I use to run `deepseek-ai/DeepSeek-V4-Flash-0731` on one AMD MI300X in production. It includes the Docker Compose stack, SHA-256-pinned file overlays, reference diffs against upstream, and tuning tables. The checkpoint runs as shipped, without additional weight quantization or offload.

Results from the pinned stack (vLLM ROCm nightly 0.26.1rc1.dev229+g124154a88.rocm723, AITER 0.1.19):

| Metric | Result | | --- | --: | | Single-stream decode (median per-stream, DSpark-7) | 168.6 tok/s | | Prefill with tuned kernels | ≈ 7.9–8.5K tok/s (6,988–7,019 tok/s on fresh prompts in the shipping profile) | | 8 concurrent streams | 542 tok/s aggregate, 90.3 tok/s median per stream | | 64-stream burst | 830 tok/s aggregate, no OOM, no engine errors | | Context | 256K validated (the architecture supports 1M) | | Weights in HBM | 156.67 GiB — no additional quantization or weight offload |

The official vLLM recipe targets NVIDIA and newer AMD hardware. Running the model reliably on MI300X required fixes for its FP8 format, MoE routing at high concurrency, causal speculative verification, CPU-KV synchronization, and several untuned kernel shapes. This repository collects those fixes and pins the versions used in production.

• * *

Why MI300X

#why-mi300x

The MI300X has 192 GB of HBM3 and 5.3 TB/s of memory bandwidth, with 2.4× the HBM capacity of an H100 SXM5 (AMD). Doubleword's write-up estimates that it costs roughly half as much at list price. For this 304B-parameter checkpoint, the memory capacity allows a simple single-GPU deployment:

• The entire model fits in HBM without PCIe weight streaming or layer offload.
• There is room for a 20 GB GPU KV pool and a 96 GiB CPU tier for evicted prefix-cache entries.
• One card handles 2–8 typical concurrent streams and bursts of up to 64 streams.

MI300X (CDNA3) implements the AMD/Graphcore fnuz variant of E4M3, while MI325X and newer use OCP-standard FP8 (background). A kernel that assumes OCP semantics on MI300X can be wrong by a factor of two in the scale domain. Correctness on this FP8 implementation was the first priority; performance tuning came afterward.

Prior art, and what this repo adds

#prior-art-and-what-this-repo-adds

Fergus Finn's MI300X worklog and the accompanying Doubleword repository identified the FP8 incompatibility, missing AITER fast paths on gfx942, HIP-graph hazards in sparse MLA decode, and MoE routing bugs. The official vLLM recipe covers NVIDIA hardware and newer AMD GPUs (MI325X at 4K context and MI355X), but not a single-MI300X production configuration for the 0731 checkpoint.

This repository adds:

1. Correctness overlays for the pinned ROCm nightly, including fixes not yet in upstream vLLM. 2. A validated serving configuration with probabilistic DSpark drafting, block rejection, and static K=7. It uses a 2,048-token scheduler budget and a 1,024-token long-prefill cap to prevent a cold prompt from stalling other streams. 3. AITER GEMM tuning tables for the recurring gfx942 shapes the packaged tables were missing, plus a gfx942 OGS geometry override for the MXFP4 experts. 4. A hybrid KV strategy: 20 GB of fp8_ds_mla GPU cache + 96 GiB native CPU offload, with a load-path fencing fix that upstream issue #47282 documents but PR #47291 never merged.

Repository layout

#repository-layout

.
├── compose.yaml         # The production stack (vLLM ROCm + Caddy), digest-pinned
├── Caddyfile.example    # Copy to Caddyfile; set hostname, email, and source CIDR
├── vllm-entrypoint.sh   # Removes stale CPU-KV mmaps from /dev/shm before start
├── SHA256SUMS           # SHA-256 pins for every runtime artifact
├── patches/
│   ├── *.py            # Byte-for-byte production overlays (mounted read-only)
│   ├── diffs/*.patch   # Unified diffs vs. the upstream base revision
│   └── README.md       # Provenance and regeneration instructions
└── tuning/
    └── *.csv           # AITER A8W8 blockscale tuning tables for gfx942

Runtime configuration

#runtime-configuration

The stack uses a digest-pinned official vLLM ROCm nightly with:

• --trust-remote-code and the DeepSeek V4 tokenizer, reasoning, and tool parsers
• fp8_ds_mla KV cache (UE8M0 block-scaled FP8, not generic unscaled FP8) with 256-token blocks
• VLLM_ROCM_USE_AITER=1 and --moe-backend triton; Triton OGS handles the grouped MXFP4 experts, while AITER handles attention and dense linear layers
• DSpark-7 speculative decoding with probabilistic drafting and block rejection
• full/breakable CUDA graph capture, giving one graph launch per token during steady decode
• Caddy as an IP-allowlisted HTTPS proxy

Deploying it

#deploying-it

1\. Host prerequisites

#1-host-prerequisites

One MI300X (gfx942, 304 CUs, ~192 GiB HBM), a working AMD kernel driver, recent Docker Compose, ~235 GiB RAM for the CPU KV tier, and ~500 GB disk (the model cache alone is ~156 GB).

2\. Pull the pinned runtime and model

#2-pull-the-pinned-runtime-and-model

VLLM_IMAGE='vllm/vllm-openai-rocm@sha256:e68d18b2ba50298661bfc49baf01158fbf036645c2362cccf3e8a7a79fe6c69a'
MODEL='deepseek-ai/DeepSeek-V4-Flash-0731'
REVISION='7872f01b1d1fe23eabc4c98b48bffcef5a386062'

docker pull "$VLLM_IMAGE"
docker run --rm --entrypoint hf \
  -v /root/.cache/huggingface:/root/.cache/huggingface \
  "$VLLM_IMAGE" download "$MODEL" --revision "$REVISION"

3\. Prepare the files

#3-prepare-the-files

cp Caddyfile.example Caddyfile   # then set your hostname, email, and remote_ip CIDR
mkdir -p aiter-cache crash-dumps
chmod +x vllm-entrypoint.sh
sha256sum -c SHA256SUMS        # verify the overlays before first start

4\. Start

#4-start

docker compose config -q
docker compose up -d
docker compose logs -f inference

A healthy start takes ~5 minutes and must show all of:

Model loading took 156.67 GiB
DSpark draft model loaded: 96 params
GPU KV cache size: 1,927,444 tokens
Maximum concurrency for 262,144 tokens per request: 7.35x
Created mmap file /dev/shm/vllm_offload_...mmap (103.08 GB)
Capturing CUDA graphs (FULL)
Application startup complete

After graph capture, run rocm-smi --showmeminfo vram. The warmed high-water mark is ~204.5 GB of 205.8 GB. If only a few hundred MB remain, the server may start but fail on the first request.

5\. Smoke-test

#5-smoke-test

HOST='your-host.example.com'
curl -fsS "https://$HOST/v1/models"
curl -sS "https://$HOST/v1/completions" \
  -H 'Content-Type: application/json' \
  -d "{\"model\": \"deepseek-ai/DeepSeek-V4-Flash-0731\",
       \"prompt\": \"Calculate 17 * 23. Answer with the number only.\",
       \"temperature\": 0, \"max_tokens\": 32}"

The patches

#the-patches

Each patches/*.py file is a full-file overlay mounted read-only over its counterpart in the container; compose.yaml contains the target paths. The corresponding diffs/*.patch records the change from its upstream base. The base image remains digest-pinned, so upgrades require changing the image reference and revalidating the stack.

| Overlay | Mounted over | Fixes | Needed when | | --- | --- | --- | --- | | gpt_oss_triton_kernels_moe.pack128-fused-silu-fast-routing.py | vllm/.../fused_moe/experts/gpt_oss_triton_kernels_moe.py | MXFP4 bitmatrix padding lanes + fused-SiLU grouped experts + fast DeepSeek routing | Required for the MXFP4 Triton path; the mask fix is not yet upstream | | mxfp4.fused-silu.py | vllm/.../fused_moe/oracle/mxfp4.py | Gate/up interleave layout for the fused-SiLU kernel | Required with the fused-SiLU overlay; skip both if you keep the standard SiLU path | | triton-kernels-matmul-ogs-opt-flags.dsv4-mi300x.py | vllm/third_party/triton_kernels/matmul_ogs_details/opt_flags.py | gfx942 MXFP4 OGS tile geometry (up to 1,536 routed rows) | Performance on gfx942; the stock geometry slows sharply above 768 routed rows | | fused_compress_quant_cache.fnuz-shuffle.py | `vllm/models/deepseek_v4/common/ops/fused_compress_quant_cache.py

...[truncated by AAIF intake]...