Training Visible, Output Buffered | Generated by AI
Found it. The training IS running — the checkpoint file (1.49GB, updated at 22:41) proves it. But the log is dead because of Python stdout buffering.
When Python’s stdout is piped through tee, it’s not a TTY, so Python fully buffers output instead of line-buffering. The print() calls in train.py are sitting in a pipe buffer and never flushing to /tmp/train.log.
The log file’s last modified time is 21:16 (only the initial warnings). The process is piped (fd/1 -> pipe), confirming the buffering issue.
Training is fine, just invisible output. To fix for future runs, either:
- Set
PYTHONUNBUFFERED=1in the launch command - Or add
-uflag:python3 -u train.py ...
The checkpoint at out-github-code-124m/ckpt.pt (1.49GB) was saved moments ago, so it’s actively training. You’re just flying blind on loss/metrics. Want me to kill and restart with unbuffered output, or let it finish and check the checkpoint stats?
Generated by AI. Curating and sharing still takes effort.
If you find it useful, feel free to
donate.
WeChat: @lzwjava
·
X: @lzwjava ·
Say hi 👋