Training Methods Beyond Supervised Fine-Tuning | Generated by AI
Question: What training methods could supersede SFT (Supervised Fine-Tuning) for LLMs?
Answer:
SFT is probably not the final stage of LLM alignment. It is a relatively simple idea:
(prompt, ideal_answer) -> minimize cross entropy loss
For every token:
[ L_{SFT}=-\sum_t \log P_\theta(y_t|x,y_{<t}) ]
The model learns to imitate human-written examples.
The problem: SFT teaches imitation, not intelligence. It compresses existing solutions into weights but does not directly optimize for “finding better solutions.”
Several approaches can supersede or reduce the role of SFT:
1. RL / RLVR (Reinforcement Learning with Verifiable Rewards)
This is the most obvious successor.
Instead of:
Human:
Question -> Answer
SFT:
learn this answer
RL says:
Question
|
Model generates solution
|
Verifier checks correctness
|
Reward
|
Update model
Example:
Math:
Q: Solve 12345 * 67890
Model A:
wrong answer
reward = 0
Model B:
correct reasoning + answer
reward = 1
Optimize:
[ J(\theta)=E[R(x,y)] ]
Using algorithms like:
- PPO
- GRPO
- RLOO
- DPO variants
This is what pushed reasoning models forward.
Examples:
- DeepSeek R1 style training
- OpenAI o-series reasoning models
The key insight:
Intelligence can emerge from search + reward, not only imitation.
2. Self-play / AI-generated training
Human data is limited.
Future:
Strong model
|
generate problems
|
solve problems
|
filter solutions
|
train next model
Similar to AlphaZero.
For LLM:
Teacher model:
create difficult coding problem
Student:
solve
Verifier:
test code
Keep only successful trajectories
The dataset becomes generated by intelligence itself.
3. Process Reward Models (PRM)
SFT usually only cares about final answer.
Example:
2 + 3 = ?
Bad reasoning:
step1 wrong
step2 lucky correction
answer 5
SFT sees:
answer = 5
PRM evaluates:
step1 good
step2 good
step3 good
Training signal becomes much richer.
Especially important for:
- math
- coding
- scientific reasoning
4. Online learning / continual learning
Current LLM:
pretrain
|
SFT
|
deploy
Future:
deploy
|
experience
|
feedback
|
update
|
better model
Like humans.
The model continuously learns from:
- user interaction
- coding tasks
- experiments
- tool usage
5. Agentic training
Maybe the biggest shift.
Today:
LLM = answer generator
Future:
LLM
|
+-- search
|
+-- code execution
|
+-- browser
|
+-- database
|
+-- simulator
Training objective changes:
from:
produce text
to:
achieve goal
Reward:
Did the software compile?
Did the experiment work?
Did the customer problem get solved?
6. Evolutionary methods
Another possibility:
Generate many candidates:
Model A
Model B
Model C
Model D
|
evaluate
|
keep best
|
mutate
Similar to genetic algorithms.
Could work well for:
- prompts
- agents
- tool strategies
- architectures
7. Direct preference optimization (DPO family)
DPO already reduces dependency on SFT.
Instead of:
learn answer
learn:
answer A > answer B
Objective:
[ \log\sigma(\beta(\log\pi(y_w)-\log\pi(y_l))) ]
Variants:
- DPO
- IPO
- KTO
- ORPO
- SimPO
They may replace traditional RLHF pipelines.
My guess: future stack
Probably:
Foundation model
|
v
large-scale pretraining
|
v
synthetic data
|
v
SFT (small amount)
|
v
RLVR / agent training
|
v
self-improvement loop
|
v
deployment + continual learning
SFT will probably not disappear. It becomes like:
- “teach basic behavior”
- “initialize policy”
but the frontier moves toward:
reasoning + search + verification + interaction.
For someone building nanoGPT/nano-vLLM style systems, the interesting next step is not only implementing SFT. A very educational project would be:
nanoGPT
+
GRPO
+
verifier
+
tool calling
+
self-generated dataset
That is close to the direction of modern reasoning models.
Generated by AI. Curating and sharing still takes effort.
If you find it useful, feel free to
donate.
WeChat: @lzwjava
·
X: @lzwjava ·
Say hi 👋