H200 Training: 1.38B, 128K Tok/s

H200 Training: 1.38B, 128K Tok/s | Generated by AI

Home 2026.07

Training Summary — H200 d24 (1.38B params)

Model Architecture

Field	Value
Layers	24
Embed dim	1536
Heads	12 (GQA, kv=12)
Head dim	128
Vocab	32768
Params	1,384,122,122 (1.38B)
Window pattern	SSSL (alternating sliding window)
FP8	Enabled (145/158 layers, tensorwise)

Hardware

Field	Value
GPU	NVIDIA H200 (140 GB)
Tok/sec	~128K average (range: 121K–131K)
MFU	~62% bf16 (range: 59%–63%)
Step time	~4.1s (range: 4.0–4.4s)

vs MI300X reference: 68K tok/s, 27% MFU — H200 is ~1.9× faster

Training Configuration

Field	Value
Steps	29,000 (Chinchilla-optimal)
Total tokens	15.2B (ratio 20.83× params)
Batch/step	524,288 tokens (16 × 32,768 microbatch)
Grad accum	16
Seq len	2048
LR warmup	40 steps, peak lrm=1.0
Weight decay	0.042 (scaled for d24)
Eval	every 500 steps
Save	every 5000 steps
Tracker	wandb (offline run: `w1pqqwr2`)

Loss Curve (first 108 steps)

step   0: loss 10.40  (init validation bpb 3.22)
step  20: loss  7.21
step  40: loss  6.02  (LR peak reached at step ~39)
step  60: loss  5.59
step  80: loss  5.22
step 100: loss  4.99  (loss plateauing — early convergence of easy patterns)

Steady, clean descent. The initial spike at step 0 (190s, 2.7K tok/s) is the compilation/JIT warmup.

Projections

Metric	Estimate
Current step	108 / 29,000 (0.37%)
Elapsed	6.7 min
ETA	~32.9 hours
Finish	~Jul 9 11:30 UTC (~33h from start)

Data

Field	Value
Dataset	FineWeb-Edu (44 parquet files, 96 GB)
Total tokens available	35.7B (enough for d24 and d30)
Tokenizer	Custom BPE (vocab 32K, trained on FineWeb-Edu)

Checkpoints

Will save to ~/.cache/nanochat/base_checkpoints/d24/ every 5000 steps (steps 5000, 10000, 15000, 20000, 25000, + final). Each ~6-12 GB depending on optimizer state.

To monitor progress

tmux attach -t train-d24          # view live
tmux capture-pane -t train-d24 -p # capture without attaching

Back Donate