Run continuous reinforcement learning post-training for trillion-parameter models
Miles reduces the idle time where inference and training wait for each other during the model's knowledge absorption process. By separating the inference engine from the training engine to run them simultaneously, it enables continuous post-training for large-scale models. It is an enterprise reinforcement learning framework that forked from slime and has evolved alongside it.
v0.1 release and latest model support
This version targets production-level post-training by enhancing precision and stability features. It provides day-0 support for the latest frontier models, including DeepSeek-V4.1, Qwen3.8, and Kimi K3, in conjunction with SGLang. It also supports 8-bit and 4-bit RL training tailored for the Blackwell architecture, and DeepSeek-V4 Flash RL training is possible on AMD Instinct MI355X.
Asynchronous pipeline and fast weight transfer
Miles operates rollout (inference) and training workers in a fully asynchronous manner. This reduces pipeline wait times and allows for flexible configuration of on-policy and off-policy schedules. Even for models with trillions of parameters, new weights are delivered to the engine in seconds, preventing interruptions in the training loop. In distributed environments, P2P RDMA is used to increase weight transfer speed. It supports LoRA and multi-LoRA, enabling frontier-scale model training with a small number of GPUs, and allows trained adapters to be loaded directly into SGLang for inference.
Accuracy maintenance and fault recovery
It supports the Token In Token Out (TITO) method to maintain accuracy without re-converting tokens between inference and training. To eliminate instability caused by routing mismatches in MoE models, it provides the R3 (Rollout Routing Replay) feature, which reuses expert routing recorded during rollout in the learner's forward pass. Even if the SGLang engine stops, Miles recovers that engine and resumes execution immediately without halting the entire run.
Hardware and agent environment support
It supports a wide range of hardware, from NVIDIA's GB300 to A100, and from AMD's MI355X to MI300X. It can be deployed with the same run script via Ray or Kubernetes, offering high flexibility in infrastructure choice. It provides various agent environment connectors, such as Harbor, HUD, and NeMo Gym, for training coding and computer-use agents. Additionally, it supports diffusion model training, including Flow-GRPO and DiffusionNFT, through Miles-diffusion.
Pre-adoption checks
Miles is written in Python and operates in integration with SGLang, Megatron-LM, and torch_memory_saver. It also supports FSDP2 and torchtitan backends. Before adoption, you should verify that your GPU model is included in the support list and determine whether to use Ray or Kubernetes as the orchestrator. Specifically, if using low-precision training (MXFP8, NVFP4), it is advisable to check if you are following recipes for numerical stability. The README does not specify a particular license type, so license terms should be verified separately before use in an enterprise environment.
