Introducing PULSELoCo: More Than 100x Lower Trainer Communication for Distributed RL
PULSELoCo matches DiLoCo while cutting trainer-to-trainer communication by more than 100x versus DDP in our largest setting.
Distributed RL post-training has two expensive communication channels. The first is trainer-to-inference weight synchronization: trainers refresh rollout workers, the inference workers that generate trajectories, with new policy weights. Training workers also need to synchronize with each other, either by exchanging ordinary gradients every step or by periodically merging local weight updates.
Earlier this year, when we introduced PULSE, the core idea was simple: rollout workers run inference with BF16 (bfloat16) weights. Many FP32 trainer weight updates round back to the same BF16 value. From the rollout worker's point of view, nothing about the model computation or decoding procedure changes. Sending those updates immediately is wasted bandwidth. PULSE communicates a parameter only when it changes the BF16 value actually used for computation. We call this compute-visibility.
Figure 1 shows the BF16 rule at the level of one weight: an FP32 update can exist on the trainer’s FP32 master weights, but until it crosses a BF16 rounding boundary, the rollout worker still sees the same value.
The first algorithm we built around this observation was PULSESync, the trainer-to-inference weight-synchronization version of PULSE. It sends sparse BF16 patches instead of full checkpoints from trainers to inference workers. The receiver reconstructs the BF16 weights bit-for-bit, so for the rollout worker, this is a free optimization: same weights, same forward pass, a transfer more than 100x smaller.
The same idea later showed up in the infrastructure behind Cursor’s Composer 2 training run. In Fireworks’ post, “Frontier RL is Cheaper Than You Think”, they describe 1 TB policy updates sent as roughly 2% deltas with exact reconstruction, and cite PULSE as theoretical support for the same high-sparsity effect. Their work shows that PULSESync-style weight synchronization is practical even in trillion-parameter regimes.
PULSESync handles the trainer-to-inference weight-sync channel: moving fresh policy weights from trainers to the inference fleet. The second channel is trainer-to-trainer update synchronization, where training workers merge their local updates back into a shared global policy.
Leveraging this same sparsity structure for synchronizing training workers is more complex. Specifically, trainers typically synchronize their gradients, which we have shown in our work to be explicitly dense. A natural idea is then to let training workers perform a local update step with Adam and compute and aggregate the update, which will be sparse. This leads us naturally to the DiLoCo-style approach! In DiLoCo-style training, what workers synchronize is called a pseudo-gradient: the update from the shared starting checkpoint to a worker’s local endpoint. The natural question was whether the same compute-visibility rule could make that update sparse as well.
PULSELoCo is our answer.
In our largest setting, Qwen2.5-7B with an 8-step local-update window (H = 8), PULSELoCo matches DiLoCo’s learning behavior while making trainer-to-trainer communication much smaller. Compared with DiLoCo, the local-update baseline, PULSELoCo sends more than 17x less data. Compared with DDP (distributed data parallel), where workers synchronize dense gradients every optimizer step, it reduces communication by about 138x.
Figure 2 shows what that reduction means in practice. At a 90% GPU-utilization target, the required trainer-to-trainer bandwidth drops from about 44 Gbit/s with DiLoCo to about 2.6 Gbit/s with PULSELoCo.
The other result is just as important: to our knowledge, this is the first empirical demonstration that DiLoCo-style local-update RL post-training works for LLMs. Prior DiLoCo work focused mostly on pre-training. RL is a different regime: during each local-update window, rollout workers serve the shared global checkpoint while trainers take private local steps. As H grows, those rollouts become increasingly off-policy relative to the local trainer weights. So before PULSELoCo can be useful, the DiLoCo-style RL baseline has to work at all.
PULSELoco
In DDP, every worker synchronizes the full FP32 gradient every optimizer step. For example, in the 7B setting, that is roughly 30.5 GB of FP32 gradient data per worker.
DiLoCo reduces how often workers communicate. Each worker starts from the same checkpoint, runs H local optimizer steps, and synchronizes its pseudo-gradient once per round.
A stable DiLoCo-style baseline helps in two ways. It gives us local-update training for RL post-training, and it cuts synchronization frequency relative to DDP by a factor of H. But DiLoCo still sends a full FP32 pseudo-gradient at every synchronization round. So the question becomes: can we keep the local-update structure, but exploit the RL sparse update structure to make the pseudo-gradient itself sparse?
Raw gradients are dense in RL post-training, so sparsifying raw gradients is not the right objective. Compute visibility applies to updates, not gradients. PULSELoCo is how we apply PULSE to the trainer-to-trainer channel: it applies the BF16 visibility rule to DiLoCo-style pseudo-gradients, with FP32 error feedback for the entries that are not sent yet.
At a high level, PULSELoCo keeps DiLoCo's local training loop and outer optimizer intact. After local training, each trainer computes the same FP32 pseudo-gradient that DiLoCo would synchronize. PULSELoCo then adds error feedback and sparsifies the synchronization payload. The full pseudocode is in the paper, but the loop is simple:
Start from the same global checkpoint.
Run H local Adam steps.
Form the DiLoCo pseudo-gradient: the update from the shared checkpoint to the trainer's local endpoint.
Add the previous round's unsent FP32 entries back into the current pseudo-gradient, so small updates can accumulate across rounds.
Send only the compute-visible entries of this error-feedback-corrected FP32 pseudo-gradient.
Keep the unsent entries in FP32 error feedback until they become large enough to change the BF16 value used by the next forward pass.
Average the sparse FP32 pseudo-gradients, then apply the typical DiLoCo outer optimizer.
That is the key point: PULSELoCo changes the communication payload, not the local optimizer or the outer optimizer. DiLoCo synchronizes the full FP32 pseudo-gradient; PULSELoCo synchronizes the compute-visible part and carries the rest forward in error feedback.
What We Tested
We evaluated DDP, DiLoCo, and PULSELoCo in the same modified TRL GRPO (Group Relative Policy Optimization) training loop. The task is the MATH benchmark. The models are Qwen2.5-1.5B, Qwen2.5-3B, Qwen2.5-7B, and Llama-3.2-3B. All runs use four workers. For DiLoCo and PULSELoCo, we use H = 8 local steps for Qwen and H = 4 for Llama.
The learning curves check both requirements: DiLoCo-style local-update training remains viable in RL post-training, and PULSELoCo recovers DiLoCo-like learning curves while making the trainer-to-trainer data much smaller.
Across the main settings, each PULSELoCo worker sends only 3.6-5.2% of FP32 pseudo-gradient values per synchronization round. In other words, the pseudo-gradient communication is 94.8-96.4% sparse after error feedback.
Figure 4 shows the measured sparsity across the evaluated models.
On Qwen2.5-7B, DiLoCo sends a full 30.5 GB FP32 pseudo-gradient per worker per synchronization round. PULSELoCo sends a 1.77 GB encoded sparse update. That is more than 17x smaller than DiLoCo.
Compared with DDP, the savings are larger because PULSELoCo also synchronizes less often. With H = 8, the Qwen2.5-7B comparison is roughly:
DDP over the same eight optimizer steps: about 244 GB per worker using the same per-worker byte count.
PULSELoCo encoded sparse update: 1.77 GB per worker per synchronization round.
Reduction: about 138x.
These are calculated communication sizes, not end-to-end networking measurements. The important point is the two-factor saving: communicate every H steps, then make each synchronized update sparse.
Why This Matters
The point is not just a better compression ratio. Distributed RL has two bandwidth bottlenecks: keeping rollout workers current and keeping trainers synchronized. PULSESync addresses the first with lossless sparse BF16 patches, cutting weight-sync traffic by more than 100x. PULSELoCo addresses the second with sparse DiLoCo-style pseudo-gradient synchronization, making trainer communication more than 17x smaller than DiLoCo and about 138x smaller than DDP in our largest setting, while matching DiLoCo’s learning behavior.
Together, they make geo-distributed RL post-training over commodity links finally practical. The system no longer has to assume every trainer and rollout worker sits inside a single cluster. That opens a path for open-source groups and labs to use distributed computing without datacenter-grade interconnect being the bottleneck.
Read The Paper
This post is the short version. The updated PULSE paper gives the full technical story: the compute-visibility principle, the PULSESync and PULSELoCo algorithms, and the experiments behind the numbers above.
You can read it here: https://arxiv.org/pdf/2602.03839
 | A guest post by
|