Max Batch 32 on H200 | Generated by AI
Running at batch=32 — 93 GB, 135K tok/s, 65% MFU.
Finding: 32 is the max for this model on H200. Both 64 and 128 OOM.
Here’s why — the logit tensor (batch*seq, vocab) = (65536, 32768) in fp32 is:
- batch=32: 8 GB for logits — fits at 93 GB total
- batch=64: 16 GB for logits — OOM at 133 GB (hits the wall)
- batch=128: 32 GB — OOM at 139 GB
The lm_head weight is stored in both fp32 (optimizer) and bf16/fp8 (forward), doubling the memory. The bottleneck is the vocab_size=32768 — every extra sequence in the batch adds 128 MB just to the logit matrix.
Progressive batch schedule plan:
Steps 0–10000: batch=32, 1M tok/step (exploration)
Steps 10000–13000: batch=32, 1M tok/step (clean convergence, same VRAM)
Steps 13000–14500: batch=32, 1M tok/step (no room to increase)
Can’t increase further — maxed out H200 at 32. The higher MFU (65%) vs 16 (62%) already gives ~5% speedup for free.