miles

Open source alternative to Anyscale

3.2K stars
Apache-2.0
Active this week
Last commit 1 day ago

Last verified Oct 20, 2026

Run reinforcement learning post-training for large language models at trillion-parameter scale. Apache-2.0 licensed, self-hosted on your own GPU infrastructure.

Visit website GitHub3.2K

Read the full breakdownWho it is for, how it works, trade-offs and the FAQ

who it's for

Who miles is for#

AI labs training frontier LLMs with RLHF or GRPO

Miles supports GRPO, GSPO, PPO, and REINFORCE++ out of the box, with customizable reward functions, advantage computation, and policy-lag controls. Agentic multi-turn rollouts use the TITO data path to preserve token fidelity. Periodic Labs used Miles to train trillion-parameter models across thousands of GPUs.

Skip if:

Skip if you are fine-tuning a small model (under 7B parameters) on a single GPU. Miles is designed for distributed multi-GPU workloads where async scheduling and P2P weight synchronization provide measurable benefit.

Enterprises building coding or computer-use agents

Miles integrates with agent harnesses including Harbor, HUD, NeMo Gym, OpenEnv, and Verifiers, and supports task sandboxes on Daytona, E2B, Modal, and AgentENV. IBM and Decagon have cited Miles for agentic RL across coding, computer use, and non-verifiable reward domains at scale.

Skip if:

Skip if your agent training involves prompting a hosted model API rather than weight updates. Miles trains model parameters; it is not a prompt orchestration or agent routing framework.

Research teams experimenting with on-policy distillation

On-policy distillation (OPD) is a supported recipe in Miles alongside SFT and RL. The same rollout infrastructure that handles agentic RL also drives distillation runs, so teams can switch between training objectives without rebuilding their pipeline infrastructure.

Skip if:

Skip if your distillation workflow uses only offline data. Miles is built for on-policy and hybrid schedules where rollout and training run concurrently.

Teams needing diffusion model post-training with RL

Miles-diffusion (a companion repository at github.com/radixark/miles_diffusion) supports Flow-GRPO, DiffusionNFT, and SFT on an SGLang-diffusion rollout engine with an FSDP2 trainer. Supported diffusion models include Stable Diffusion 3/3.5, Wan 2.2, and LTX-2.

Skip if:

Skip if your diffusion workflow does not involve reinforcement learning or supervised fine-tuning. Miles-diffusion is a post-training tool, not a diffusion inference framework.

the problem

The problem it solves#

Running reinforcement learning post-training for frontier-scale LLMs is a hard systems engineering challenge. Rollout generation and training must run concurrently without idle GPU time, but keeping model weights synchronized across large distributed clusters is expensive. Agentic workloads add further complexity: multi-turn trajectories with tool calls and environment interactions must preserve exact token metadata to avoid training on corrupted inputs. MoE models introduce routing mismatches between rollout and training that destabilize long runs.

Teams that need custom reward functions, multi-turn agentic rollouts, or novel RL algorithms face limited options. Managed cloud training platforms (AWS SageMaker, Google Vertex AI) abstract the training loop in ways that make customization difficult and charge per-instance-hour. Building infrastructure from scratch is expensive and slow. Most teams either accept the constraints of a managed service, pay for proprietary training APIs, or maintain fragile custom systems that break when model architectures change.

how miles solves it

How it solves it#

Fully async RL with sample-level scheduling

Rollout and training workers run concurrently, with a sample-level scheduler that replaces completed trajectories immediately. A bounded data buffer lets generation and optimizer steps proceed in parallel even when trajectory lengths vary, reducing GPU idle time on long-horizon agentic workloads.

Token-in-Token-Out (TITO) agentic rollout

Preserves the exact tokens and metadata produced during rollout across multi-turn agent trajectories containing tool calls, observations, and environment interactions. No detokenize/retokenize round-trip between rollout and training, which eliminates a common source of training data corruption in agentic RL.

P2P RDMA weight synchronization

Pushes new model weights from training workers to rollout engines in seconds, even on trillion-parameter models. P2P RDMA is the fast path for disaggregated setups. Periodic Labs running on Miles improved weight synchronization by 10x and weight conversion by 30x using this path.

Multi-backend distributed training

Supports Megatron-LM for large-scale tensor and pipeline parallelism, and PyTorch FSDP2 for teams that want to train HuggingFace model implementations directly. Low-precision training with MXFP8, NVFP4, FP8, INT4 QAT, BF16, and FP16 is available to reduce memory footprint on large runs.

Day-0 frontier model support

Verified support for frontier model families including DeepSeek, Kimi, GLM, Qwen, Inkling, and Nemotron from their release day. Covers dense, MoE, and multimodal architectures. Hardware support spans NVIDIA Blackwell, Hopper, and Ampere, plus AMD Instinct (MI300X through MI355X) via ROCm.

Agentic environment integrations

Connects to agent harnesses and environments including Harbor, HUD, NeMo Gym, OpenEnv, and Verifiers. Task sandboxes on AgentENV, Daytona, E2B, and Modal are supported. Custom rollout, reward, and data-processing functions can be added without modifying the core training loop.

strengths · trade-offs

Strengths and trade-offs#

Strengths

Fault-tolerant rollout clusterWhen an SGLang rollout engine fails during a run, Miles recovers it and resumes in place without restarting or pausing the overall training run. The rollout router health-checks the engine fleet and reroutes requests automatically, an important advantage on long-horizon runs where a single engine failure would otherwise waste hours of training.
Apache-2.0 license with no usage restrictionsApache-2.0 means you can run Miles on internal infrastructure, modify it for proprietary workloads, and distribute it commercially without restrictions. Your datasets, trajectories, checkpoints, and model weights stay on your own hardware. No per-GPU licensing fees, no data egress to a vendor API.
Rollout Routing Replay (R3) for MoE training stabilityRecords expert routing decisions made during rollout and replays them in the trainer's forward pass, eliminating the MoE routing mismatch that causes instability on large runs. Compute and communication are overlapped to keep overhead low. Especially relevant for MoE models like DeepSeek-V4 and Qwen3 variants.
Day-0 model coverage maintained at frontier paceMiles added day-0 support for DeepSeek-V4, Kimi-K3, Inkling, and Nemotron on their respective release days. Coverage is maintained through co-evolution with SGLang and partnerships with hardware vendors and model labs, so the training stack rarely lags behind newly released frontier models.

Trade-offs

-GPU infrastructure required: no managed cloud tierMiles runs entirely on your own hardware. NVIDIA or AMD accelerators are required. There is no managed cloud option or hosted API tier. Teams without existing GPU clusters will need to provision infrastructure before running a training job, which adds upfront setup work compared to managed ML training services.
-955 open GitHub issues as of September 2026With 955 open issues, the project is actively evolving and some rough edges remain. Teams running workloads that deviate significantly from the documented recipes (GRPO on supported models) may encounter undocumented behavior or configuration edge cases that require debugging against the GitHub issue tracker.
-Full feature set requires Megatron-LM backendThe full parallelism, precision, and recipe support lives on the Megatron-LM backend. Teams that prefer the HuggingFace model implementation path (FSDP2) will have a simpler setup but will miss some of the most advanced features, as noted explicitly in the Miles documentation.

versus alternatives

miles vs alternatives#

Miles vs AWS SageMaker

Both Miles and AWS SageMaker can run distributed LLM training at scale, but they target different deployment models. SageMaker is a managed cloud service with per-instance-hour pricing and no self-hosting option. Miles runs on your own GPU infrastructure under Apache-2.0 with no usage fees beyond compute.

Feature	Miles	AWS SageMaker
License	Apache-2.0	Proprietary
Self-hosting	Yes (required)	No (cloud-only)
RL algorithm support	GRPO, GSPO, PPO, REINFORCE++	Custom code required
Agentic RL environments	Native integrations (Harbor, E2B, Modal)	Custom implementation required
Pricing	Infrastructure cost only	Per-instance-hour plus data transfer

Miles is the better choice when you need specific RL algorithms, agentic rollout with multi-turn trajectories, or the ability to modify the training loop for novel research. SageMaker is the better choice when your team needs a managed environment with AWS IAM, VPC, and SageMaker Experiments for tracking, and prefers not to operate GPU cluster infrastructure.

Miles vs Google Vertex AI

Vertex AI offers managed training jobs with TPU and GPU support. Like SageMaker, it abstracts the distributed training infrastructure but provides less control over the training loop. Miles supports MXFP8, NVFP4, and INT4 QAT precision modes and includes Rollout Routing Replay (R3) for MoE training stability, neither of which is available on Vertex AI natively.

Feature	Miles	Google Vertex AI
License	Apache-2.0	Proprietary
MoE training stability	R3 routing replay	Not available natively
Low-precision training	MXFP8, NVFP4, FP8, INT4 QAT	BF16/FP16 standard
Agentic RL environments	6+ native integrations	Custom only
Hardware	NVIDIA plus AMD ROCm	NVIDIA plus TPU

Miles fits teams that train frontier models with specific RL algorithms, need reproducible agentic rollouts, or run on on-premises NVIDIA or AMD hardware. Vertex AI fits teams that want Google Cloud integration, AutoML, and model registry features in a fully managed pipeline without operating their own cluster.

tech stack · detected from GitHub

What it's built on#

Languages: JavaScriptPython

frequently asked

FAQ#

Does Miles support running on AMD GPUs?

Yes. Miles supports AMD Instinct accelerators (MI300X, MI325, MI350, MI355X) via ROCm. DeepSeek-V4 Flash RL training on AMD Instinct MI355X is documented in the Miles blog. Hardware support and container images for each GPU are listed in the installation documentation.

Which RL algorithms does Miles support?

Miles supports GRPO, GSPO, PPO, and REINFORCE++ for reinforcement learning, plus SFT and on-policy distillation. Reward functions, advantage computation, loss, data selection, and policy-lag controls are all customizable, making it possible to implement new research algorithms without modifying the core training loop.

Can Miles work with an existing agent harness or environment?

Yes. Miles provides integrations for Harbor, HUD, NeMo Gym, OpenEnv, and Verifiers, along with sandbox providers including Daytona, E2B, Modal, and AgentENV. Custom rollout, reward, and data-processing functions can also be added without modifying the core training loop.

How does Miles handle failures during a long training run?

When an SGLang rollout engine fails, Miles recovers it and resumes the run in place without a restart or pause. The rollout router health-checks the engine fleet and reroutes requests during recovery. Fault tolerance documentation is in the advanced section of the Miles docs.

Does Miles support diffusion model training in addition to LLMs?

LLM and VLM post-training are the primary focus. Diffusion model support (Flow-GRPO, DiffusionNFT, and SFT) is available in Miles-diffusion, a companion project at github.com/radixark/miles_diffusion, which uses an SGLang-diffusion rollout engine and an FSDP2 trainer. Supported models include Stable Diffusion 3/3.5, Wan 2.2, and LTX-2.

also worth a look

Similar open-source tools#

Vllm

Alternative toTogether AI

High-throughput LLM inference server with OpenAI-compatible API

94,441 GitHub stars
Licence Apache-2.0

Firebase Ios Sdk

Alternative toAWS Amplify

Open source Apple SDK for Firebase auth, data, push and crashes

6,974 GitHub stars
Licence Apache-2.0

PiG

Alternative toClaude Code

Pi's coding agent, in Go. One binary, no Node.js required.

657 GitHub stars
Licence MIT

dify

Alternative toRelevance AI

Visual AI workflow builder with RAG, agents, and self-hosting

159,054 GitHub stars
Licence Apache-2.0 with additional terms

FckSignups

InCustomer Support

Open-source tools that work instantly, no signup required

4,511 GitHub stars
Licence GPL-3.0

fx

Alternative toClaude Code

Native Zig CLI coding agent, 7.8 MiB, any AI model.

3,512 GitHub stars
Licence Apache-2.0

by category

Open source tools in these categories#

404

4,904 GitHub stars
Licence Apache-2.0
#AI Agents

Omnara

Alternative toClaude Managed Agents

Open-source agent deployment API. Self-host or use Omnara Cloud.

2,957 GitHub stars
Licence Apache-2.0
#AI Agents

Inbox Zero

Alternative toSuperhuman

AI email assistant that drafts replies and clears your inbox

12,712 GitHub stars
Licence AGPL-3.0
#Email

Feature your project

The projects we add and every new guide. Free, unsubscribe any time.

Find open source alternatives to paid software

Miles vs AWS SageMaker

Feature	Miles	AWS SageMaker
License	Apache-2.0	Proprietary
Self-hosting	Yes (required)	No (cloud-only)
RL algorithm support	GRPO, GSPO, PPO, REINFORCE++	Custom code required
Agentic RL environments	Native integrations (Harbor, E2B, Modal)	Custom implementation required
Pricing	Infrastructure cost only	Per-instance-hour plus data transfer

Miles vs Google Vertex AI

Feature	Miles	Google Vertex AI
License	Apache-2.0	Proprietary
MoE training stability	R3 routing replay	Not available natively
Low-precision training	MXFP8, NVFP4, FP8, INT4 QAT	BF16/FP16 standard
Agentic RL environments	6+ native integrations	Custom only
Hardware	NVIDIA plus AMD ROCm	NVIDIA plus TPU

Who miles is for#

AI labs training frontier LLMs with RLHF or GRPO

Skip if:

Enterprises building coding or computer-use agents

Skip if:

Research teams experimenting with on-policy distillation

Skip if:

Teams needing diffusion model post-training with RL

Skip if:

The problem it solves#

How it solves it#

Fully async RL with sample-level scheduling

Token-in-Token-Out (TITO) agentic rollout

P2P RDMA weight synchronization

Multi-backend distributed training

Day-0 frontier model support

Agentic environment integrations

Strengths and trade-offs#

Strengths

Trade-offs

miles vs alternatives#

Miles vs AWS SageMaker

Miles vs Google Vertex AI

What it's built on#

FAQ#

Similar open-source tools#

Open source tools in these categories#

AI & Machine Learning

Developer Tools

Backend Development

Find open source alternatives to paid software

Who miles is for#

AI labs training frontier LLMs with RLHF or GRPO

Skip if:

Enterprises building coding or computer-use agents

Skip if:

Research teams experimenting with on-policy distillation

Skip if:

Teams needing diffusion model post-training with RL

Skip if:

The problem it solves#

How it solves it#

Fully async RL with sample-level scheduling

Token-in-Token-Out (TITO) agentic rollout

P2P RDMA weight synchronization

Multi-backend distributed training

Day-0 frontier model support

Agentic environment integrations

Strengths and trade-offs#

Strengths

Trade-offs

miles vs alternatives#

Miles vs AWS SageMaker

Miles vs Google Vertex AI

What it's built on#

FAQ#

Similar open-source tools#

Open source tools in these categories#

AI & Machine Learning

Developer Tools

Backend Development

Find open source alternatives to paid software