Bringing PyTorch Monarch to AMD GPUs: Single-Controller Distributed Training on ROCm

Bringing PyTorch Monarch to AMD GPUs: Single-Controller Distributed Training on ROCm

• ID: 8aed8c89
• 原文链接: https://pytorch.org/blog/bringing-pytorch-monarch-to-amd-gpus-single-controller-distributed-training-on-rocm/
• 作者: PyTorch Team
• 日期:
• 分类: infra
• 标签: pytorch, monarch, rocm, distributed-training, fault-tolerance
• 质量评分: 4/5
• 抓取时间: 2026-07-26T23:35:50+08:00
• Obsidian 证据: OpenClaw定时任务/ClawFeed24小时高价值一览/2026-07-26-ClawFeed24小时高价值一览.md

一句话

大模型训练基础设施开始把 GPU 和节点故障当成常态来设计恢复路径

中文导读

PyTorch 团队把 Monarch 的单控制器、actor runtime、process mesh 和 supervision tree 带到 ROCm，让 AMD GPU 集群可以跑弹性分布式训练。文章给出 TorchTitan、TorchFT、Monarch 组合后的无全局重启恢复流程，包括本地重启、Lighthouse 选 donor、peer checkpoint transfer 和 quorum 重建，并展示 128/256 GPU 规模下的故障恢复实验。

English Summary

The PyTorch post describes porting Monarch’s single-controller runtime and distributed communication stack to ROCm for AMD Instinct GPUs. It focuses on elastic, fault-tolerant distributed training using actors, process meshes, supervision trees, TorchTitan and TorchFT, including local restart, donor selection, peer checkpoint transfer and quorum recovery.

Obsidian 证据摘录

d
- 验证状态：已落盘且非空

## 今日精选
1. Open-weight AI is having its Kubernetes moment. Let's not ruin it. — 把开源云原生的历史经验套到 open-weight 模型生态，判断有明确边界，也有政策含义。
2. Android May Soon Restrict On-Device ADB, Affecting Shizuku, libadb and Developers — 不是单纯抱怨限制，而是把 loopback ADB、Wireless Debugging、TCP/IP ADB 的风险路径拆清楚。
3. Bringing PyTorch Monarch to AMD GPUs: Single-Controller Distributed Training on ROCm — 有具体工程细节和规模验证，适合跟踪大模型训练基础设施的人读。

## 可发布正文如下

本期从 ClawFeed 候选里只保留真正读过正文、评分超过 7 分的内容，偏 AI agent、前沿判断和不太枯燥的深度文章。

- 标题：Open-weight AI is having its Kubernetes moment. Let's not ruin it.
  评分：8.8/10
  推荐语：这篇值得读，因为它没有把 open-weight 模型简单写成“开源会赢”，而是用 Kubernetes 经验解释生态为什么会围绕一个可运行、可改造、可部署的基座聚集。文中把边界也交代清楚：open-weight 不等于完整开源，模型微调也不像 Kubernetes 代码贡献那样能回流到统一上游。
  摘要：作者认为，Qwen、GLM-5.2、Kimi K3 这类模型让 open-weight AI 接近云原生当年的平台化节点，围绕推理服务、agent runtime、评测、沙箱、可观测性和垂直微调会形成复合创新。文章反对美国用宽泛禁令切断中国 open-weight 模型，主张用前沿级美国开放模型、政府采购、标准测试和配套基础设施参与竞争。
  链接：https://tobi.knaup.me/2026-07-25-open-weight-ai-is-having-its-kubernetes-moment/

- 标题：Android May Soon Restrict On-Device ADB, Affecting Shizuku, libadb and Developers
  评分：8.2/10
  推荐语：这篇对 Android 开发者有用，价值在于把“限制 on-device ADB”背后的安全动机和实际使用路径分开讨论。正文明确列出恶意 App 在普通用户、Wireless ADB、TCP/IP ADB 三种场景下必须跨过的人为授权步骤，也提出了比永久封禁 loopback 更温和的持久开关方案。
  摘要：文章基于 Google IssueTracker 中 ADB 维护者关于只绑定 `wlan0` 的讨论，指出这会影响 Shizuku、libadb-android、Termux、本机调试、VPN / Ethernet ADB 等工作流。作者认可减少 ADBD 暴露面的目标，但认

原文 / 抓取内容

Bringing PyTorch Monarch to AMD GPUs: Single-Controller Distributed Training on ROCm

原文链接: https://pytorch.org/blog/bringing-pytorch-monarch-to-amd-gpus-single-controller-distributed-training-on-rocm/

Featured projects

• [

Training state-of-the-art large language models (LLMs) with billions of parameters requires distributed training across hundreds or thousands of GPUs. At this scale, hardware failures are not exceptional events—they are expected. A single GPU memory error, network partition, or node crash can bring down an entire training run that has been progressing for days or weeks. While our previous work demonstrated near-linear scaling of FP8 training at scale (achieving 96.16% scaling efficiency on a 1024-GPU MI325 cluster with DeepSeekV3-671B), the key challenge remains: reliability at scale.

To address these challenges, we have brought PyTorch Monarch to AMD Instinct GPUs with ROCm, expanding the single-controller model beyond CUDA environments and bringing this emerging runtime to a broader hardware ecosystem.

In this blog, we will explore the architecture of PyTorch Monarch, walk through the engineering effort required to port Monarch’s GPU runtime and distributed communication stack to ROCm, and demonstrate how the system dynamically recovers from node failures without halting the entire training job. By the end, you will understand how Monarch enables elastic, fault-tolerant distributed training on AMD GPUs and why this represents a significant step toward stable, large-scale AI infrastructure.

The Challenge: Reliability at Scale

Traditional fault-tolerance strategies rely heavily on periodic checkpointing: saving the full model state to persistent storage at regular intervals. When a failure occurs, the entire job restarts from the last checkpoint. While conceptually simple, this approach has significant drawbacks.

<table style="height: 322px;" width="986"><tbody><tr><td><b>Challenge</b></td><td><b>Impact</b></td></tr><tr><td><b>Checkpoint overhead</b></td><td><span style="font-weight: 400;">Writing hundreds of gigabytes of model state to storage consumes time and I/O bandwidth.</span></td></tr><tr><td><b>Wasted computation</b></td><td><span style="font-weight: 400;">All progress since the last checkpoint is lost upon failure.</span></td></tr><tr><td><b>Cluster idle time</b></td><td><span style="font-weight: 400;">The entire cluster sits idle while the failed node is replaced and the job restarts.</span></td></tr><tr><td><b>Scalability limits</b></td><td><span style="font-weight: 400;">As cluster size grows, the probability of failure during any checkpoint interval increases.</span></td></tr></tbody></table>

For truly large-scale training, scaling is not enough—training must also recover from failures. We need a more dynamic approach, one that allows healthy nodes to continue training while failed nodes recover and rejoin, minimizing wasted computation and maximizing GPU utilization. This is where PyTorch Monarch comes in.

What is PyTorch Monarch?

PyTorch Monarch introduces a new distributed programming paradigm that enables developers to orchestrate entire GPU clusters from a single Python program. With its actor-based runtime, process mesh abstraction, and asynchronous execution model, Monarch simplifies large-scale distributed training and enables complex workflows that combine training, evaluation, and reinforcement learning within one unified script.

The architecture operates at multiple distinct levels:

1. Python API: A single-program interface where developers write simple Python code to get distributed GPU execution. 2. Monarch Runtime: Manages actors and meshes, supervision trees, and tensor sharding. 3. Rust Runtime (Tokio): Ensures high performance and memory safety. 4. Infrastructure: Integrates with RDMA, RCCL/NCCL, SLURM, Kubernetes, and SkyPilot.

_Figure 1: PyTorch Monarch architecture decoupling Python API from the Rust runtime and infrastructure._

By decoupling the parallelism strategy used within each training replica from the fault-tolerance mechanism used across replicas, Monarch provides a cleaner fault-tolerance model. Failures are isolated (actors have private state, crashes do not propagate), hierarchical (handled at the lowest possible level), and recovery is fast (seconds for local restart, minutes only if escalated).

_Figure 2: Monarch’s hierarchical fault-handling model and supervision tree._

Porting Monarch to ROCm: Ecosystem Integration

Bringing Monarch to AMD GPUs required significant engineering effort to port the GPU runtime and distributed communication stack to ROCm.

We successfully implemented three main porting paths:

1. Collective Communications: We used hipify_torch to convert the C++ bridge code from CUDA to HIP and linked against RCCL, which mirrors NCCL’s API. 2. GPU Memory Management: We extended the build system to auto-detect the platform and route CUDA driver API calls through their HIP equivalents. 3. RDMA Integration: Configuring GPU_PLATFORM=rocm keeps the libibverbs\-based RDMA path intact while swapping the GPU-side bindings from CUDA to HIP for GPU-direct transfers.

_Figure 3: Porting Monarch from CUDA to ROCm via hipify\_torch and auto-detection._

Moreover, two cross-cutting issues shaped the port and deserve a closer look:

1. No static link for the HIP runtime: NVIDIA ships libcudart_static.a, so the CUDA path links cudart_static directly. ROCm ships no static equivalent for libamdhip64, so the ROCm build links amdhip64 dynamically. Both platforms additionally dlopen the GPU driver API functions, including hipMemCreate, cuMemCreate, and related calls, keeping the runtime contract identical on either side. 2. Rust compatibility shim instead of forking the bindings: Once hipify_torch rewrites the C/C++ headers, bindgen emits HIP-named types such as hipError_t, hipDeviceptr_t, and hipStream_t. Rather than add branches at every Rust call site, we added a rocm_compat module in nccl-sys and rdmaxcel-sys that re-exports HIP symbols under their CUDA names, for example pub type cudaError_t = hipError_t and pub use hipSetDevice as cudaSetDevice. The rest of the Rust code stays platform-agnostic.

These efforts culminated in the introduction of HIP type aliases in Rust, with all 1,171 tests passing, ensuring full support for ROCm 7.0+. We have upstreamed these contributions to the open-source community (see PR #2393 and PR #2891).

Today, Monarch on ROCm provides full ecosystem support, including the Actor runtime, RDMA, Supervision, and Tensor sharding. It runs seamlessly on SLURM (HPC), Kubernetes (Cloud native), and SkyPilot (Multi-cloud), enabling downstream engines like TorchTitan (Training engine) and TorchFT (Fault tolerance) for production workloads.

Case Study: Fault-Tolerant Training at Scale

To demonstrate the power of Monarch on AMD GPUs, we integrated it with TorchTitan and TorchFT to build a resilient, checkpoint-less distributed training architecture.

Architecture Overview

The architecture consists of three layers:

• Monarch: Acts as the orchestrator, managing process and cluster orchestration. It spawns ReplicaActors and a Lighthouse service, organizing GPUs into Process Meshes.
• TorchFT: Handles fault tolerance at the step level. It contacts the Lighthouse for quorum coordination, performs Quorum AllReduce, and skips failed nodes.
• TorchTitan: Serves as the training engine, executing the Forward (FSDP), Backward, and Optimizer steps, while managing checkpoints and metrics.

__

_Figure 4: The resilient training stack on AMD GPUs integrating Monarch, TorchFT, and TorchTitan._

In this setup, Monarch provides a supervision tree for fine-grained fault detection and isolation. When a failure is injected into the training actors, it is detected by the Lighthouse and handled by TorchFT. The healthy replicas continue training independently despite peer failures, without requiring a global interruption.

Dynamic Fault Recovery Workflow

Let us walk through a concrete scenario with four replica groups to understand the recovery workflow.

1. Normal Training: The OrchestrationManager spawns 4 ReplicaActors (Monarch Supervisors) and a Lighthouse. Each ReplicaActor spawns a Replica with 8 GPU processes running TorchTitan trainers. All 4 replicas are ready (quorum_id=1) and DiLoCo gradient synchronization occurs every 20 steps. 2. Failure Detection: A GPU process in Replica 0 crashes. The Monarch supervisor captures the report_training_error (with full traceback) before the process dies. Replicas 1, 2, and 3 are marked as unaffected and continue training. 3. Local Restart: ReplicaActor 0 initiates an in-place restart (_stop_and_restart()), stopping the old process mesh and spawning a new one. Meanwhile, the other 3 replicas continue syncing (quorum_id=2) 4. Peer Checkpoint Transfer: The Lighthouse selects Replica 1 as the donor. A peer checkpoint transfer (model, optimizer, scheduler, and trainer state) is initiated from Replica 1 to the recovering Replica 0. All replicas pause briefly at the quorum boundary while the new quorum forms. 5. Resumed Training: Once Replica 0 is synced, the new quorum (quorum_id=3) is established with all 4 replicas, and DiLoCo synchronization resumes.

_Figure 5: Dynamic fault recovery workflow demonstrating peer checkpoint transfer without global checkpoint reload._

The entire recovery process completes without any manual intervention, without full checkpoint restarts, and with minimal disruption to the overall training throughput.

Performance Characteristics

We validated this approach on both SLURM and Kubernetes environments using AMD Instinct MI300-class clusters.

SLURM 16-Node MI300 Cluster (128 GPUs)

We trained a Llama 3 8B model on a 16-node SLURM cluster (totaling 128 MI300 GPUs), injecting RCCL failures every 180 seconds with a quorum sync every 20 steps. The results were remarkable:

• The number of active workers fluctuated dynamically (between 8 and 16) due to the injected failures.
• Training continued seamlessly despite frequent failures—there was no full restart.
• The loss curve showed steady convergence, closely matching the baseline run without failure injection.

_Figure 6: Training continues despite frequent failures on a 16-node SLURM MI300 cluster. Note that not a single replica was down for 30+ minutes; recovery was fast, and different replicas were killed and recovered dynamically._

Kubernetes 32-Node MI355 Cluster (256 GPUs)

We scaled the experiment to a 32-node Kubernetes cluster (totaling 256 MI355 GPUs). The number of participants remained highly stable (fluctuating slightly between 30 and 32 during recovery events), and the global average loss decreased smoothly from 12 to approximately 4. This demonstrates that the Monarch fault-tolerance model works reliably on both SLURM and Kubernetes at scale.

_Figure 7: Stable recovery and smooth loss convergence on a 32-node Kubernetes MI355 cluster._

Summary and Future Directions

Training large AI models at scale requires more than raw computational power—it demands resilient infrastructure that can gracefully handle the inevitable hardware failures. By bringing PyTorch Monarch to AMD Instinct GPUs with ROCm, we have demonstrated a practical approach to fault-tolerant distributed training that minimizes wasted computation and maximizes GPU utilization.

This integration represents several significant achievements for large-scale training on AMD GPUs:

• First large-scale validation on AMD hardware: We successfully deployed Monarch with TorchTitan and TorchFT on AMD GPUs, demonstrating that the ROCm software stack fully supports advanced fault-tolerance mechanisms.
• Cleaner fault-tolerance model: Monarch provides a robust supervision tree and process mesh abstraction, isolating failures and enabling rapid local recovery.
• Ecosystem readiness: The approach works seamlessly across SLURM and Kubernetes, making it ready for production workloads.

The key architectural insight is the use of Monarch’s actor-based runtime and supervision trees to isolate failures, combined with TorchFT’s quorum-based synchronization to allow healthy nodes to continue training. For teams running large-scale training workloads on AMD GPUs, this integration offers a path toward more stable, efficient, and cost-effective model development.

Looking ahead, our next steps include:

• Extending NIC support and improving runtime performance.
• Expanding Monarch to support more pre-training and reinforcement learning (RL) frameworks on ROCm.
• Further optimizing fault tolerance performance, specifically reducing rejoin reload latency and overlapping recovery with compute.
• Continuing our open-source collaboration with the PyTorch community.

Additional Resources

• PyTorch Monarch GitHub Repository
• TorchTitan Documentation
• TorchFT GitHub Repository
• Resilient Large-Scale Training: Integrating TorchFT with TorchTitan on AMD GPUs

Disclaimers

Third-party content is licensed to you directly by the third party that owns the content and is not licensed to you by AMD. ALL LINKED THIRD-PARTY CONTENT IS PROVIDED “AS IS” WITHOUT A WARRANTY OF ANY KIND. USE OF SUCH THIRD-PARTY CONTENT IS DONE AT YOUR SOLE DISCRETION AND UNDER NO CIRCUMSTANCES WILL AMD BE LIABLE TO YOU FOR ANY THIRD-PARTY CONTENT. YOU ASSUME ALL RISK AND ARE SOLELY RESPONSIBLE FOR ANY DAMAGES THAT MAY ARISE FROM YOUR USE OF THIRD-PARTY CONTENT.

_© 2026 Advanced Micro Devices, Inc._