Build and document large language model experiments with an open research platform
Marin provides an environment for transparently recording and reproducing every stage of large language model development, from data cleaning to model training and evaluation. It is not just code but a community and software platform that shares the knowledge of the research process and experimental results. This allows readers to directly construct complex model training pipelines and verify experimental outcomes.
Open development that records failed experiments
Marin records every step from raw data to the final model, keeping failed experiments as part of that record. Its core value lies in publishing not only model artifacts and infrastructure but also the process knowledge required to build the models. Currently, Marin focuses on the pre-training and post-training of large Mixture-of-Experts models to solve important tasks for scientists and researchers.
Delphi: Scaling laws and reproducible pipelines
Marin offers Delphi, an open scaling suite inspired by Pythia. Delphi extends LLM recipes in the range of 3e18 to 1e23 FLOPs, consisting of scaling recipes that map compute budgets to model configurations, a scaling suite trained on Google TPU Research Cloud, and scaling laws that predict large models using small Delphi models. Checkpoints for all runs are available on Hugging Face, and a deterministically reproducible training mixture pipeline is also published on the open Nemotron-CC, StarCoderData, and ProofPile 2 datasets. The repository includes recipe code in the CompletedAdamHParams class, development methodology in the add_scaling_heuristic agent skill, and plot preparation data for Delphi charts.
Execution structure that defines experiments like a Makefile
For example, a script to train a small model on the TinyStories dataset defines a step to tokenize the dataset and a step to train the model using that result. The training step depends on the tokenization step, so it runs only after tokenization is complete. This structure applies flexibly from small model training to scaling up to larger dataset mixtures or large GPU/TPU pods. Experiments are run using StepRunner along with related libraries such as fray.cluster and levanter.optim.
Other training achievements and model examples
Marin has also been used to build audio-text, DNA, and protein models. The marin/experiments library encourages such work. Previously, an 8B parameter model was trained using Marin, which showed better performance than Llama 3.1 8B in the base model benchmark suite. A Marin 32B model has also been trained. The Open Athena blog provides additional integrated learning content, including task scheduling across heterogeneous clusters (Iris), improved pre-training efficiency, scaling laws that extrapolate large models from small ones up to 300 times, and MoE expert load balancing validated at the 32B-A5B scale.
Pre-use checks and contribution methods
Marin is written in Python, and documentation can be found on ReadTheDocs or in the docs/ folder. To get started, it is recommended to install Marin, follow an example of training a small language model, and look at how to run the DCLM 1B/1x experiment. You can also join the Marin Discord to communicate with the community. Contributors can check the project workflow in CONTRIBUTING.md and use loadable agent skills in .agents/skills/ or .claude/skills/. For example, .agents/skills/add-dataset/ contains a step-by-step guide for adding a new dataset. Marin's core collaborators are from Stanford CRFM and Open Athena, and the research is supported by partners such as the Jen-Hsun and Lori Huang Foundation.
