Training Results and Model Readiness | Generated by AI
You’re right to call me out — I was loose with the numbers. Let me look again clean.
Checked the actual checkpoint metadata and config:
Training completed successfully — here’s what’s real:
| Detail | Value |
|---|---|
| Steps | 8,985 (full dataset epoch, num_iterations=-1) |
| Total tokens | 8,985 × 65,536 = ~589M |
| Start val_bpb (step 7) | 0.9536 |
| Final min val_bpb | 0.4053 |
| Duration | 184.7 min (3h 4m) |
| Peak VRAM | 6,240 MiB |
| Checkpoint | model_008985.pt (793 MB) |
| Base model | d12 pretrained at step 50k (code+SEC+fineweb) |
| SFT data | SmolTalk + 1,000 custom code/SEC + MMLU + GSM8K + SpellingBee |
Corrections from my previous reply:
- The 0.954 bpb I cited was the smoke test at step 7, not the previous run’s final. The real final is 0.4053.
- Previous pretraining bpb (1.418) is in a different directory — these checkpoints are in a fresh
chatsft_checkpoints/d12/dir, not the old ones. - The loss oscillation at the end is mild (0.97–1.13 range with lrm → 0) — expected for cosine decay tail with no warmup.
The model is ready. The tmux session is idle at shell prompt. Want to run inference on it?
Generated by AI. Curating and sharing still takes effort.
If you find it useful, feel free to
donate.
WeChat: @lzwjava
·
X: @lzwjava ·
Say hi 👋