grail-v0: How We Built a Fully Open, Incentivized, Decentralized Reinforcement Learning System
How we designed the architecture, verification pipeline, incentives, algorithms, and early results behind one of the first open decentralized RL networks with real-time transparency
Reinforcement Learning (RL) is now central to post-training large language models, but running it at scale remains expensive and only partially open. With grail-v0, the first viable version of grail project, we introduce a decentralized and incentivized RL system designed around a core principle of building in the open. The entire training loop is public and verifiable, the codebase is fully open-source, and all training metrics stream live through our W&B project, giving anyone real-time visibility into how the model learns. To the best of our knowledge, grail is one of the first decentralized RL systems to pair an incentivized network with full real-time transparency.
This post walks through the architecture, explains how we verify rollouts and structure incentives to make decentralized training practical, and shares early results from training a 1.5B parameter model on a live decentralized network. By decentralizing the most expensive stage of RL and exposing the entire process in real time, grail-v0 demonstrates a path toward scalable, transparent, and eventually more cost-efficient post-training. We end by describing the project’s roadmap and the key developments that will define the grail’s next milestone.1
Before we describe the architecture, we briefly introduce the underlying network. grail-v0 runs on a decentralized infrastructure that provides a mainchain for tracking participants and their performance, along with application-specific subnets that define the logic for each task. grail is one such subnet. The infrastructure is built on Bittensor, but no prior knowledge is needed; here it simply supplies identities, incentives, and a lightweight coordination layer for decentralized training.
The Grail Architecture
We begin with the architecture that enables decentralized, incentivized RL. As shown in Figure 1, grail-v0 consists of three types of nodes: miners, validators, and a trainer. Miners generate inference rollouts. Validators verify those rollouts and assign weights that drive incentives. The trainer consumes the verified rollouts to update the model. All coordination happens through Cloudflare R2, which serves as the shared storage layer for checkpoints and rollout data.
This structure reflects a key design choice in grail-v0: keeping the trainer centralized while decentralizing rollout generation. Recent RL systems identify rollout generation as the primary bottleneck in large-scale training (Liu et al., 2025; Piché et al., 2025; Wang et al., 2024). Prime Intellect 2 reports a training-to-inference FLOP ratio of roughly 1:4, indicating that inference consumes the majority of compute in practice (Garg et al., 2025). Since inference is the clear bottleneck, we decentralize this component and keep the trainer simple and efficient.
As shown in Figure 2, grail-v0 operates on a window-based training cycle, with each window lasting 30 blocks, or about 6 minutes. The diagram illustrates the flow within a single window. During a window, the trainer downloads verified rollouts from trusted miners through R2, filters them to select the highest-quality groups, applies training updates, and then publishes a new checkpoint at the window boundary. In the next window, miners download this checkpoint, generate new rollouts with proofs, and upload them to their respective buckets. Validators then process these rollouts in the following window, verifying that each sample was produced from the correct checkpoint and assigning miner weights based on throughput and reliability. These weights determine which miners are considered trusted on-chain.
grail-v0’s pipeline introduces a fixed, one-window lag between rollout generation, validation, and training. Using window W as the reference:
Miners generate rollouts in window W using the checkpoint from window W–1
Validators verify those rollouts in window W+1
The trainer updates the model in window W+1 using rollouts produced in Window W
This creates a predictable delay of exactly one window across the system.
On the trainer side, all rollouts arriving at the beginning of window W were generated from the checkpoint published at the end of window W–2. Training is therefore off-policy by roughly one window, but accepting this delay keeps the trainer updating continuously. Prior works show that bounded delays of this kind are effective at scale, including the rollout–learner lag in PRIME-RL and the pipelined actor–learner structure in PipelineRL (Chowdhury et al., 2025; Garg et al., 2025).
For the validator, the same delay means that rollouts generated in window W are only validated in window W+1. While validation is happening on those rollouts, the trainer is already using them for learning, and miners are producing the next batch. This pipelining ensures that miners, validators, and the trainer all remain active with no synchronization stalls, preserving both throughput and full asynchrony.
However, delayed validation raises the question of whether miners could send incorrect rollouts that influence one window of training before detection. In practice, miners are strongly incentivized to remain honest: incorrect rollouts are detected in the next window and penalized immediately, and miners with a consistent history of correctness receive higher weights. Even if a trusted miner attempted to cheat, the impact would be limited to a single training window, which did not destabilize learning in our experiments, especially with the guardrails built into the trainer. This incentive structure keeps latency low while maintaining robust validation and stable training under the one-window delay.
This design gives us two main advantages:
Incentivized scaling and optimization. Miners can deploy inference on any hardware setup and freely optimize the miner code. In deployment, we have already seen miners push far beyond the reference implementation: a single A100 running the default miner produces about 128 rollouts per window, while the top miner generates more than 7,000 rollouts per window. This strongly suggests multi-GPU or multi-node inference and custom throughput optimizations, which the incentive structure actively rewards.
Fast and lightweight verification. The validator is designed to check rollouts far faster than miners can generate them, even under varied hardware and miner behavior. This keeps verification from becoming a bottleneck and allows the pipeline to remain fluid as miner throughput scales.
With the overall architecture in place, we now move to the core components of grail-v0. We begin with the verification pipeline and the grail proof, which ensures that rollouts are generated from the correct model checkpoint. We then describe the modifications required to stabilize GRPO in a decentralized environment, and conclude with how miner incentives reduce UID pressure and encourage high-throughput rollout generation.
Verification Pipeline
The validator’s job is to make sure every rollout submitted by a miner is both model-consistent and useful for training. In practice, this means verifying that miners are not gaming the incentive system to inflate their rewards and not attempting to poison the training pipeline with manipulated or invalid data.
To do this, the validator performs a sequence of lightweight checks on each sampled rollout. These checks fall into three categories:
1. Structural checks
These confirm that the rollout is well-formed and follows the expected structure:
Token IDs sent by the miner match the expected tokenizer.
The data sent by the miner has the correct typing and schema.
2. Environment-consistency checks
These ensure the rollout corresponds to the correct task and that miners have not altered the environment inputs or outputs/observations:
The reward and success metrics match what the validator computes.
The environment prompt matches the assigned prompt for that miner.
3. Model-consistency checks
These checks verify that the rollout was generated by the correct model rather than a smaller, modified, or manipulated one:
Hidden-state commitments must pass the grail proof.
Reported log probabilities for each chosen token must match those of the correct checkpoint.
The distribution of the probabilities for the chosen tokens must match the profile of the expected model.
Each rollout must either reach the maximum allowed length or end with a valid termination token whose probability is consistent with the model.
At first glance, these checks appear computationally expensive. In practice, verification is lightweight because the validator only needs to sample a small subset of miners, rollouts, and occasionally even token positions within those rollouts, which keeps the overall cost low.
This blog focuses specifically on the design of the two most important checks: the grail proof and the log-probability distribution check.
Grail Proof
Verifying that miners actually used the correct model is a central challenge in decentralized RL. Hidden states are large, noisy, and hardware-dependent, and naive equality checks fail under the small numeric drift that naturally arises across different GPUs, drivers, frameworks, and especially inference-optimized libraries like vLLM and Sglang.
We want a method that is lightweight enough to run across thousands of rollouts per window, yet strong enough to prevent miners from using smaller models or modifying parts of the sequence we do not directly inspect. Recent work, such as TopLoc (Ong et al., 2025), which was deployed in a decentralized RL setting in Prime Intellect 2 (Garg et al., 2025), uses top-k activations and polynomial encodings to robustly verify hidden states. TopLoc’s proof size is roughly 256 bytes for every 32 new tokens, or about 8 bytes per token.
The grail proof follows a similar philosophy but pushes the design toward minimal overhead. For each challenged token, we construct a 4-byte sketch that acts as a fingerprint of the hidden state at that position, which is half the size of the 8-byte sketches used in TopLoc. The proof is cheap to compute, compact to store, and extremely difficult to forge.
What the Grail Proof Does
At a high level, the validator challenges a small set of token positions and checks whether the miner’s hidden-state sketches match the hidden states produced by the validator’s copy of the model. Instead of transmitting full hidden states, which are large and implementation-dependent, the miner sends compact sketches that capture the essential structure of the top activations. These sketches behave like cryptographic fingerprints: efficient to generate, minimal to transmit, and highly resistant to manipulation.
The grail proof is built in three steps:
Step 1: Selecting the Most Informative Dimensions
Hidden states contain thousands of dimensions, but most of the signal lies in the largest coordinates. Following the insight from TopLoc (Ong et al., 2025), we select the top k = 32 dimensions by absolute value. These dimensions are the most stable across GPU kernels and model formats, and they provide enough resolution for verification without incurring high bandwidth costs.
Step 2: Logarithmic Quantization
The selected activations are then quantized into discrete buckets using a sign-preserving logarithmic mapping:
We use B = 16 buckets per sign, and hᵢ denotes the hidden-state value at dimension i. Logarithmic bucketing is important because LLM hidden-state magnitudes exhibit heavy tails, with rare outliers that can be several orders of magnitude larger than typical activations. A linear quantizer allocates too much resolution near large values and too little near zero, making it sensitive to small floating-point drift. The log mapping, by contrast, compresses large values, expands small ones, and naturally absorbs GPU- and implementation-level noise while still preserving the structural differences in the hidden state that matter for verification.
Step 3: Random Linear Sketching
From these bucketed values, we produce a single compact sketch:
where the coefficients rᵢ lie in [-127, 127] and are derived from shared verifiable randomness. The miner sends one 32-bit sketch per challenged token; the validator recomputes the sketch and accepts only if the modular distance falls within an adaptive tolerance τ(p) tied to the token position p:
where p is the token position and τᵦ = 50. Early tokens, which anchor the sequence, use the base tolerance, while later tokens receive slightly more slack to account for the natural accumulation of numerical drift over time. This keeps verification precise for the parts of the sequence that matter most while remaining robust across hardware and software variations.
Security Against Forgery
For the grail proof to be meaningful, forging a valid sketch must be effectively impossible without running the real model. In practice, this is exactly what we observe. Each sketch is computed from 32 top-k activations, where bucketed values lie in the range [−15, +15] and the sketch coefficients lie in [−127, +127]. This bounds every sketch to a finite integer range of 121,921 possible values (from −60,960 to +60,960). The validator accepts a sketch only if it falls within a small tolerance window around the recomputed value. Using the upper bound of ±100, only 201 values out of the entire 121,921-value space are accepted for each challenged position, giving an adversary at most a 0.16 percent chance of guessing correctly:
Because the validator challenges 16 positions independently, a miner would have to guess all 16 sketches correctly to forge a full proof:
This corresponds to roughly 148 bits of effective security, comfortably above the 128-bit standard commonly used in modern cryptographic systems. Even under looser tolerances for later tokens, the success probability remains astronomically small.
Why Hidden-State Proofs Alone Are Not Enough
Although the grail proof ensures that the miner used the correct model for generating the hidden-states, our early deployment revealed two attack vectors that manipulate parts of the rollout not covered by the sketch. The first is the prefill–decode split, where miners run the decode phase with a smaller model and then switch to the correct model only for the prefill step. The second is completion prefix micro-manipulation, where miners slightly change the first few tokens of the completion to embed shortcuts or leak answers. In practice, this causes the completion length to collapse, because the model no longer needs to reason through the problem once the answer has been injected into the prefix.
Both attacks succeed because hidden-state proofs only verify that “these hidden states came from the right model,” not that “these tokens were actually sampled from the right model.” What the trainer ultimately consumes are the tokens, not the hidden states. This gap means that hidden-state verification alone cannot defend against decode-time spoofing and motivates the need for a mechanism that evaluates the miner’s token-level behavior directly.
This motivates the introduction of token-distribution verification, which inspects the probability trace of the entire rollout to detect when miners switch models, manipulate prefixes, or otherwise deviate from the true sampling distribution of the expected model.
Token-Distribution Verification
Every model has a distinctive pattern in how it assigns probabilities to the tokens it generates. When a miner switches to a smaller model or tampers with the prefix, that pattern changes in measurable and consistent ways. Token-distribution verification builds on this observation.
For each decoding step, the validator computes the probability that the expected model assigns to the token reported by the miner. Across the entire rollout, this produces a sequence of probabilities that captures the miner’s sampling behavior. We experimented with many different statistics over this sequence, including skewness, kurtosis, and bimodality coefficient, but most of them were either too noisy or too sensitive to prompt or model-level variations. The two metrics that consistently distinguished honest rollouts from manipulated ones were the 10th percentile of the probability trace and, in more difficult cases, the median.
One reason the 10th percentile is so effective comes from a recurring pattern we observed across a handful of open-source models. Large model families often produce a bimodal probability distribution over chosen tokens, with one cluster at reasonable probabilities and another near zero, a behavior also noted in Prime Intellect 2 (Garg et al., 2025). Smaller models, on the other hand, tend to produce a unimodal distribution dominated by values close to zero. In both cases, the lower tail becomes highly pronounced, which makes the 10th percentile a reliable indicator of manipulation.
To detect both kinds of manipulation, we evaluate the probability sequence in two separate windows. Global model swaps affect the overall probability trace, so a full-window analysis captures that behavior. Prefix edits mostly disturb the earliest probabilities, so an initial-window analysis focuses on the beginning of the sequence, where miners tend to hide these targeted manipulations.2
Failures in this check are handled differently from failures in the grail proof. Hidden-state proof violations are extremely reliable in our experiments, so we treat them as hard failures. When a miner fails the grail proof, it is immediately banned for about two hours, because false positives for this check are essentially nonexistent. The token-distribution verification, however, is more sensitive, and it can produce occasional false positives due to natural variation in decoding or prompt structure. Because of this, we treat each drop in the 10th percentile as a soft anomaly rather than an automatic failure. A miner is only penalized when a significant fraction of its sampled rollouts fail this check within a window. Once this threshold is crossed, the miner is considered untrustworthy and is banned in the same way as a hard failure.
Training Pipeline
Our training pipeline builds on GRPO, originally introduced in DeepSeekMath (Shao et al., 2024). We implemented many GRPO variants in the open-source codebase, including multiple normalization schemes, IS strategies, clipping methods, and auxiliary regularizers. While these options remain available, the versions that consistently performed well in a decentralized, window-lagged setting are the ones described below. These choices were selected to improve overall training stability and efficiency, and to remain robust under delayed rollouts (i.e., off-policy rollouts) and the heterogeneous behavior of decentralized miners.
Algorithmic Improvements over Base GRPO
DAPO-style normalization. In GRPO, the loss is normalized by completion length, which down-weights long chains of thought and introduces length-dependent gradient dilution. Following DAPO (Yu et al., 2025), we replace this with the token-level loss used in their formulation while keeping GRPO’s group-relative advantages. This removes length-induced variance and yields much more stable updates, which is especially important when rollouts are at least one window old and slightly off-policy.
Sequence-level importance sampling (GSPO-style). Since, at this stage, the reward is defined over the entire completion, we follow GSPO (Zheng et al., 2025) and use the GSPO length-normalized sequence-level importance ratio for each response instead of GRPO’s token-level ratios. GSPO shows that sequence-level weighting aligns the off-policy correction with sequence-level rewards and avoids the high-variance training noise introduced by token-wise ratios, especially on long answers. In grail-v0, where miners generate rollouts asynchronously with varying policy lag, this GSPO-style sequence-level importance sampling reduces variance and yields much smoother and more stable updates than token-level weights.
Asymmetric GRPO clipping. We use separate upper and lower clipping bounds, following the Clip-Higher strategy in DAPO (Yu et al., 2025) and analyses of GRPO clipping dynamics (Mroueh, 2025). DAPO demonstrates that relaxing only the upper clip mitigates entropy collapse and improves reasoning performance by allowing low-probability but useful exploratory tokens to receive stronger positive updates. We adopt a similar asymmetric scheme: a slightly larger upper clip to reinforce clearly good behaviors and a standard lower clip to prevent large probability drops. This does not directly correct off-policiness, but it improves robustness to stale or low-quality rollouts and accelerates learning under varying miner quality.
Light entropy regularization instead of reference-KL. Several recent studies report that KL penalties toward the base model provide little benefit in reasoning-focused RL and can even hinder exploration or degrade performance (Hu et al., 2025; Lambert et al., 2025). In contrast, entropy-based methods have been shown to prevent entropy collapse and consistently improve reasoning accuracy by promoting controlled exploration (Cheng et al., 2025; Wang et al., 2025; Cui et al., 2025). Following these insights, we remove the reference-KL term in grail-v0 and apply a small entropy bonus instead, which keeps the policy sufficiently exploratory without opposing the reward signal. This yielded the most stable training behavior across our 1.5B and 7B runs.
High-Quality Rollout Filtering and Group Ranking
At this stage of the pipeline, each window generates three to four times more rollouts than the trainer can consume, which allows us to be highly selective. We therefore apply a set of filtering and ranking steps to retain only the rollouts that provide the strongest learning signal.
Zero-variance filtering. Groups where all rollouts have identical rewards are discarded. These groups provide no learning signal and often arise from trivial completions or repeated formatting patterns. Removing them reduces noise and improves update stability.
GFPO-style group ranking. After removing zero-variance groups, we score the remaining groups using an efficiency metric inspired by GFPO (Li et al., 2025). This score balances reward density (reward per token) with intra-group advantage variance, capturing both how informative and how diverse a group’s rollouts are. Groups with higher useful variance and more reward-dense completions rank higher. We then keep only the top fraction for training. This pruning step improves sample efficiency and leads to more stable gradients.
Incentive Mechanism
Our incentivization strategy is built around two core goals:
Encourage miners to optimize their inference pipelines so they can generate as many rollouts as possible within each window of time.
Prevent UID pressure, ensuring miners improve a single miner instead of creating many weaker miners to game the scoring system.
Each miner’s score is tied to the number of valid inference rollouts they submit per window. A linear allocation of score based on rollout count does not work well: it does not push miners to compete aggressively on performance, and it incentivizes identity splitting, since multiple low-output UIDs can collectively achieve more weight than a single optimized miner.
To solve both problems at once, we use superlinear scoring, where weights are proportional to a power-law function of each miner’s normalized rollout count. This makes high-performance miners disproportionately more valuable, and it penalizes splitting work across multiple identities.
Let 𝑐ᵢ be the number of rollouts produced by miner i. We first normalize each miner’s contribution:
This produces a fractional share between 0 and 1 for every miner. We then apply a superlinear exponent α = 4:
This superlinear reward curve pushes miners to maximize throughput while making identity splitting strictly unprofitable, thereby aligning performance incentives with Sybil resistance in a single mechanism.
Experiments
All results in this section come from a single training run on the live chain, so every comparison across datasets, baselines, and evaluation modes reflects one consistent experimental setup. We train Qwen2.5-1.5B-Instruct for 100 windows on the EleutherAI MATH dataset, which corresponds to roughly 320 trainer update steps once evaluation-only windows are accounted for. Hyperparameters follow GRPO-style best practices drawn from Devvrit Khatri et al. (2025) together with our empirical calibration. These hyperparameters are presented in Table 1.
Dataset Choice and Splitting Strategy. To ensure strict evaluation integrity, all training and testing are aligned with the EleutherAI evaluation ecosystem. The harness evaluates on the official test split of this dataset. To avoid any leakage, we exclusively use the train split for both training and validation. The train split contains 7500 examples across subjects. From this set, we extract a stratified 500-example validation split that is fixed for the entire run. The remaining 7000 problems form the training set.
Evaluation Protocol. All test-set evaluations on GSM8K, MATH500, and AMC 2023 are performed using the EleutherAI evaluation harness for standardized and reproducible measurement. Decoding is configured to be deterministic (temperature = 0, no sampling), which is common practice for mathematical reasoning tasks. Error bars in all bar plots correspond to the standard error values reported by the harness.
Training and Validation Curves Detail. Training curves are computed using 16 sampled completions across approximately 440 prompts per window. Validation curves are computed every 20 windows using 5 completions per prompt on the full 500-example held-out set.
This setup allows us to answer three core questions.
1. Does Training Actually Work?
Figure 3 shows the pass@1 and pass@5 learning curves for both training and validation over 320 update steps. The model improves from approximately 3 percent to 41 percent on pass@1 and from 10 percent to 63 percent on pass@5, representing about 13-fold and 6-fold gains. Validation curves rise in close parallel with training curves, sharing the same rapid early improvement and similar saturation points, and show no clear signs of overfitting. This behavior indicates stable, well-behaved learning dynamics, while the next subsection confirms that these gains translate into improved performance on external benchmarks.
2. How Much Does the base Model Improve?
After establishing stable learning dynamics, we evaluate whether these gains translate into improved performance on external reasoning benchmarks. We compare the initial model with the trained checkpoint on GSM8K, MATH, and AMC 2023 using a consistent 0-shot setup, which isolates the effect of reinforcement learning by removing any influence from prompting strategies. This provides a direct measurement of the model’s acquired reasoning ability.
As shown in Figure 4, the trained model substantially outperforms both the 0-shot and 4-shot versions of the base model across all tasks. GSM8K accuracy increases from 57.9 percent (4-shot) to 72.2 percent. On MATH, performance rises from 12.7 percent to 47.6 percent, nearly a four-fold improvement. Even on AMC 2023, where the base model performs poorly, accuracy improves from 7.5 percent to 25 percent. These results demonstrate that our decentralized RL setup yields significant and transferable reasoning gains that far exceed what prompting techniques alone can produce.
3. Can Decentralized Off-Policy GRPO Match an On-Policy Baseline?
A central question is whether decentralized, partially off-policy GRPO training can reproduce the learning dynamics of a mature on-policy GRPO pipeline. To test this, we run a matched baseline using TRL configured to mimic GRPO-style updates, with similar group size, batch size, and overall hyperparameters. The key difference is architectural: TRL collects fresh rollouts at every update step and therefore behaves on-policy, while grail-v0 trains from decentralized miner rollouts that can be up to eight steps old. Aside from this off-policy lag and the differences required by decentralization, both experiments share the same dataset and evaluation protocol.
Figure 5 shows that the two systems produce closely aligned learning curves on both the training and validation sets. Accuracy improves at similar rates, saturates at comparable levels, and exhibits nearly parallel behavior for both pass@1 and pass@5. Although this is a single run and cannot establish statistical significance, the consistency of the trajectories suggests that grail-v0’s decentralized, delayed-off-policy GRPO behaves similarly to a centralized on-policy GRPO pipeline. The main takeaway is that decentralized post-training can achieve training dynamics on par with established on-policy frameworks.
Figure 6 shows the final test-set performance of grail-v0 and the TRL baseline across GSM8K, MATH500, and AMC 2023. The results are closely aligned across all three benchmarks, with grail-v0 matching or slightly exceeding the performance of the on-policy TRL run. Although this is a single experiment and should not be interpreted as statistically conclusive, the similarity in final accuracy supports the central observation from the training curves: decentralized, partially off-policy GRPO can achieve end-to-end model quality comparable to a conventional on-policy pipeline.3
The Remaining Challenges
With grail-v0 now training a real model on a live network, our next milestone is to make the system faster, more scalable, and more robust. This requires progress on four fronts: reducing communication overhead, decoupling compute from communication, strengthening verification to guard against more subtle attacks, and refining incentives to promote higher-quality rollouts without destabilizing training. Addressing these challenges will let us shrink window sizes, support larger models, and sustain reliable learning as participation increases.
1. Decoupling Computation from Communication
Figure 2 shows that in grail-v0, the trainer, miners, and validators alternate between computing and communicating within each 30-block window. Miners stop generating rollouts while downloading the next checkpoint, the trainer waits idly for new rollouts before continuing updates, and uploads and downloads are handled synchronously inside the main process. Because these operations occur sequentially, a large fraction of every window is spent on communication rather than actual computation.
To improve throughput, our goal is to restructure each role so that computation never pauses for communication. The trainer should continue applying updates while separate asynchronous workers fetch rollouts and upload checkpoints in the background. Similarly, miners should generate rollouts continuously while a separate thread or process retrieves new checkpoints as soon as they become available. This decoupled, pipelined design, similar in spirit to PipelineRL (Piché et al., 2025), ensures that rollout generation, verification, and trainer updates overlap rather than block one another. By keeping both miners and the trainer compute-bound instead of communication-bound, we can reduce idle time, shrink effective window duration, and significantly accelerate the overall training loop.
We have already started this transition. On the Bittensor testnet, we deployed a fully asynchronous trainer for Qwen2.5-7B-Instruct and successfully trained it, demonstrating that grail can scale beyond 1.5B parameters once synchronization constraints are removed. You can find the results in this Wandb dashboard!
2. Reducing Communication Bottlenecks to Improve Training Speed
Communication between miners and the trainer currently happens over TCP/IP using Cloudflare R2. This adds latency and limits bandwidth, especially as model sizes grow. Since miners must upload rollouts and download checkpoints through this channel, communication becomes a bottleneck that directly slows training.
Reducing communication overhead is essential for two reasons. First, it directly speeds up training because miners can produce rollouts and the trainer can consume them more quickly. Second, reducing communication makes it possible to shrink the window size, which in turn reduces the off-policy delay and increases the frequency of updates. Together, these improvements contribute to the overall goal of making grail fast enough to train significantly larger models in future milestones.
To address this challenge, we are exploring two paths. On the algorithmic side, we aim to improve tolerance to off-policy rollouts so that the trainer can update safely even when rollouts arrive less frequently. On the systems side, we reduce communication volume by replacing large data transfers with compact representations, allowing miners and the trainer to exchange only the essential information needed for updates.
3. Strengthening the Security and Robustness of the Verification Proofs
The current grail proof and the token-distribution verification have worked well during deployment, but we still see open questions around their long-term robustness. Certain attacks, such as speculative decoding, are not yet formally eliminated. Improving the security of these proofs is essential for scaling grail to higher participation and higher economic incentives.
Our goal is to make the proof both more efficient and more stable in fully asynchronous settings. Methods like LOGIC, which apply statistical tests to detect subtle distribution shifts, offer promising ideas for making verification faster and more reliable. As we improve communication efficiency and shrink window sizes, verification also needs to remain reliable under faster and more continuous interaction.
4. Incentivizing Higher-Quality Rollouts Without Excess Off-Policy Drift
As grail scales, we want miners to produce not only more rollouts but also higher-quality rollouts. This helps the trainer learn faster and makes the entire system more efficient. However, pushing miners to maximize throughput can create incentives for overly aggressive sampling strategies or off-policy drift, which can harm training stability.
To address this, we plan to update the distribution proof so it can tolerate a controlled amount of off-policy variation while penalizing extreme deviations. This strikes a balance between encouraging creativity and efficiency on the miner side and keeping the rollouts within a range that is safe and useful for training. Recent distribution-based verification techniques, including those inspired by LOGIC, provide promising tools for achieving this.
Conclusions
grail-v0 marks a clear milestone for us. It shows that decentralized RL can run end-to-end on a live network within our own infrastructure, and it does so with full transparency across a pipeline complex enough to train a decent LLM model. Every rollout, proof, and update is visible in real time, and the system achieves stable GRPO learning with meaningful gains on GSM8K, MATH, and AMC.
This version also confirms that the architecture holds together in practice. Hidden-state sketches, distribution checks, delayed off-policy GRPO, and a superlinear incentive structure interact coherently rather than as isolated ideas. The pipeline not only functions but does so in a way that others can audit, reproduce, and learn from.
The next steps follow directly from this foundation. We need to reduce communication overhead, move toward a fully asynchronous pipeline, strengthen verification, and extend the system to larger models and more complex tasks. grail-v0 is the first working version, and it shows that open decentralized RL can be both practical and transparent as the network grows.
Author: Erfan Miahi
Codebase: https://github.com/one-covenant/grail
Real-Time Training & Validation Logs: https://wandb.ai/tplr/grail/
References
Agarwal, R., Machado, M. C., Castro, P. S., & Bellemare, M. G.
Deep Reinforcement Learning at the Edge of the Statistical Precipice.
NeurIPS, 2021.
Cheng, Z., et al.
Reasoning with Exploration: An Entropy Perspective for LLMs.
Technical Report, 2025.
Chowdhury, S., et al.
PRIME-RL: Asynchronous Reinforcement Learning for Large Language Models.
Technical Report, 2025.
Colas, C., Sigaud, O., & Oudeyer, P.-Y.
How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments.
arXiv:1806.08295, 2018.
Colas, C., Sigaud, O., & Oudeyer, P.-Y.
A Hitchhiker’s Guide to Statistical Comparisons of Reinforcement Learning Algorithms.
arXiv:1904.06979, 2019.
Cui, Y., et al.
The Entropy Mechanism of Reinforcement Learning for Reasoning LMs.
Technical Report, 2025.
DeepSeek-AI
DeepSeekMath: Mathematical Reasoning with GRPO.
Technical Report, 2024.
Engstrom, L., Ilyas, A., Santurkar, S., Tsipras, D., Tran, B., & Madry, A.
Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO.
ICLR, 2020.
Garg, A., et al.
Prime Intellect 2 Supplementary: Token Probability Pathologies in LLM Decoding.
Technical Report, 2025.
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D.
Deep Reinforcement Learning That Matters.
AAAI, 2018.
Hochlehnert, A., Chevalier, Q., Cherti, M., Kossen, J., Winther, O., & Risi, S.
A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility.
Technical Report, 2025.
Hu, X., et al.
OpenReasoner-Zero: Training Reasoning LLMs Without KL Penalties.
Technical Report, 2025.
Islam, R., Henderson, P., Gomrokchi, M., & Precup, D.
Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control.
McGill Technical Report, 2017.
Khatri, D., et al.
The Art of Scaling Reinforcement Learning Compute for LLMs.
arXiv:2510.13786, 2025.
Kossen, J., et al.
LOGIC: Log-Probability Consistency Checking for LLM Verification.
Inference.ai Blog, 2025.
Lambert, N., et al.
A Modern Overview of Reinforcement Learning from Human Feedback.
Survey / Technical Report, 2025.
Li, R., et al.
GFPO: Grouped Feedback Policy Optimization.
Technical Report, 2025.
Liu, X., Chen, Z., Zhang, Y., et al.
AReaL: Asynchronous Reinforcement Learning for Reasoning LLMs at Scale.
Technical Report, 2025.
Mroueh, Y.
GRPO Dynamics and Success Amplification.
Technical Report / Blog, 2025.
Ong, J., et al.
TopLoc: Top-k Localized Polynomial Commitment Proofs for LLM Hidden-State Verification.
Technical Report, 2025.
Piché, D., et al.
PipelineRL: Fully Pipelined Actor-Learner Architectures for Reinforcement Learning at Scale.
Technical Report, 2025.
Shao, J., et al.
DeepSeekMath: Improving Mathematical Reasoning via Group Relative Policy Optimization.
Technical Report, 2024.
Stiennon, N., et al.
Learning to Summarize with Human Feedback.
NeurIPS, 2020.
Wang, Y., et al.
OpenRLHF: An Open-Source Framework for Reinforcement Learning from Human Feedback.
arXiv:2405.11143, 2024.
Wang, Z., et al.
AEPO: Arbitrary Entropy Policy Optimization for Reasoning LLMs.
Technical Report, 2025.
Yu, Q., et al.
DAPO: Decoupled Clip and Dynamic Sampling Policy Optimization for LLM Reinforcement Learning.
Technical Report, 2025.
Zhang, Y., et al.
GSPO: Group-wise Sequence Policy Optimization for Efficient LLM RL.
Technical Report, 2025.
This article focuses on the key design choices rather than every implementation detail. For the full picture, you can explore the open-source codebase.
We plan to evolve this verification over time by integrating more rigorous statistical tests to reduce false positives and strengthen proof robustness. For example, methods inspired by LOGIC, which uses token-level log-probability distributions and statistical verification, could be adapted for our decentralized RL setting.
To obtain statistically meaningful comparisons, more than a single training seed is needed because reinforcement learning variance is well-documented across many studies (Islam et al., 2017; Henderson et al., 2018; Colas et al., 2018, 2019; Engstrom et al., 2020; Agarwal et al., 2021; Hochlehnert et al., 2025). With this limitation in mind, the results should be interpreted carefully. Despite that, our preliminary experiments show that the two methods appear comparable on a single seed, and our five internal runs with the same hyperparameters converged to nearly identical asymptotic performance.
 | A guest post by
|
This is the first time we do something like this. Amazing job.
Very cool. Thanks for sharing, Erfan!