Batch Size Convergence Trick | Generated by AI

Home 2026.07

Question: Why does increasing batch size with large VRAM help during the convergence bottleneck stage?

Answer:

Yes, this is a real training trick. The reason is mainly about gradient noise, optimization landscape, and learning rate scaling.

During the early stage of training, the model is far from a good solution. You want exploration. A smaller batch gives noisier gradients, which can help escape bad directions.

But near convergence, the situation changes:

early training:
loss high
gradient estimate noisy
small batch = more exploration

late training:
loss low
gradient signal small
small batch = too much noise
large batch = cleaner optimization

The gradient you compute is:

[ g_B = \frac{1}{B}\sum_{i=1}^{B}\nabla L(x_i,\theta) ]

The expectation:

[ E[g_B] = \nabla L(\theta) ]

but variance decreases:

[ Var(g_B) \propto \frac{1}{B} ]

So larger batch means:

Near the minimum:

[ \nabla L(\theta) \approx 0 ]

The useful signal becomes tiny. If batch is small:

true gradient:
     ↓
0.001

noise:
     ±0.01

optimizer sees:
random direction

Large batch:

true gradient:
     ↓
0.001

noise:
     ±0.0005

optimizer sees:
actual direction

So it can squeeze out the last improvement.

For LLM pretraining specifically:

Suppose you train a 760M model:

early:

batch tokens = 1M tokens
lr = 3e-4

later:

loss plateau
validation stops improving

You can increase:

batch tokens = 4M or 8M
lr adjusted upward slightly

The optimizer gets a smoother signal.

This is similar to simulated annealing:

small batch:
high temperature
explore

large batch:
low temperature
settle

However, there is a catch:

Large batch can hurt because of sharp minima / generalization.

Very large batches:

32k samples
64k samples
128k samples

may converge to solutions that train well but generalize worse unless you tune:

This is why modern LLM training uses huge batches but carefully schedules them.

For example, GPT-style training often has:

tokens/batch grows during training

small:
early exploration

large:
late-stage polishing

For your H200 / 760M nanochat experiments, this is actually one of the few advantages of renting huge VRAM:

Instead of:

constant batch:
2M tokens forever

you can do:

0-70% training:
2M tokens/batch

70-90%:
4M tokens/batch

90-100%:
8M+ tokens/batch

while keeping optimizer states and model weights resident.

The final few hundred billion tokens can become more efficient because every optimizer step has a better estimate of the direction.

This is basically using H200 memory as an optimization tool, not only a speed tool.

Back Donate