85B Model on 4×P40 | Generated by AI
Question: Can 4×Tesla P40 (24GB ×4 = 96GB VRAM) directly run an 85B parameter model?
Answer:
Yes, technically you can run an 85B model on 4×P40, but not in FP16. You need quantization, and performance will be limited.
Let’s calculate.
VRAM requirement
An 85B parameter model:
FP16
85B params × 2 bytes
= 170GB
Need:
170GB + KV cache + runtime overhead
≈ 180GB+
❌ 96GB cannot fit.
INT8
85B × 1 byte
≈ 85GB
Add overhead:
85GB + KV cache
≈ 90-100GB
Borderline.
4×P40:
24GB × 4 = 96GB
Maybe possible with aggressive memory management, but tight.
4-bit quantization (realistic)
85B × 0.5 byte
≈ 42.5GB
Add:
+ model metadata
+ KV cache
+ runtime
Usually:
50-60GB
So:
4×P40 96GB
has plenty of room.
Examples:
Llama 3.1 70B Q4
Qwen 72B Q4
DeepSeek 67B Q4
are comfortable.
85B Q4 should also fit.
But speed is the issue
P40 has no Tensor Cores.
Your compute:
1 × P40
≈ 12 TFLOPS FP32
4 × P40
≈ 48 TFLOPS FP32
Compare:
RTX 4090
≈ 83 TFLOPS FP32
But LLM inference is not only FLOPS; memory bandwidth matters.
P40:
346 GB/s × 4
≈ 1.4 TB/s
A 4090:
1 TB/s
So multi-P40 has decent bandwidth.
Multi-GPU inference
You need tensor parallelism:
85B model
model layers
|
---------------------
| | | |
P40 P40 P40 P40
24G 24G 24G 24G
Software:
llama.cpp
Example:
./llama-server \
-m qwen-85b-q4.gguf \
-ngl 999 \
--tensor-split 1,1,1,1
vLLM
CUDA_VISIBLE_DEVICES=0,1,2,3 \
vllm serve model \
--tensor-parallel-size 4
Expected performance
Rough estimate:
| Hardware | 70B Q4 tok/s |
|---|---|
| RTX 4090 | ~20-35 tok/s |
| 2×3090 | ~30-50 tok/s |
| 4×P40 | ~5-15 tok/s |
| CPU only | <5 tok/s |
So 4×P40 is a capacity machine, not a speed machine.
For a home AI server:
RTX 4070
↓
training / coding / experiments
4×P40 96GB
↓
always-on LLM inference server
↓
70B-85B quantized models
is actually a reasonable architecture.
The sweet spot is probably 2×P40 or 4×P40 + llama.cpp + GGUF, because the biggest limitation is not VRAM — it is the lack of Tensor Cores and older CUDA architecture.