Qwen-0.6B on RTX 4070

Qwen-0.6B on RTX 4070 | Generated by AI

Home 2026.07

You have the Qwen3-0.6B model downloaded and an RTX 4070 GPU. Here are the commands:

Quick inference (example from the project)

cd /mnt/data/nano-vllm
python3 example.py

Custom one-liner (interactive)

cd /mnt/data/nano-vllm
python3 -c "
from nanovllm import LLM, SamplingParams
import os

path = os.path.expanduser('~/huggingface/Qwen3-0.6B/')
llm = LLM(path, enforce_eager=True, tensor_parallel_size=1)
params = SamplingParams(temperature=0.6, max_tokens=256)

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained(path)
prompts = [
    tok.apply_chat_template([{'role':'user','content':'introduce yourself'}],
                            tokenize=False, add_generation_prompt=True),
    tok.apply_chat_template([{'role':'user','content':'list all prime numbers within 100'}],
                            tokenize=False, add_generation_prompt=True),
]
outputs = llm.generate(prompts, params)
for p, o in zip(prompts, outputs):
    print('---')
    print(o['text'])
"

Benchmark

cd /mnt/data/nano-vllm
python3 bench.py

Quick test (small model — Llama-68M) if you want a faster download

# Optionally use a tiny model for testing
huggingface-cli download --resume-download HuggingFaceTB/SmolLM2-135M \
  --local-dir ~/huggingface/SmolLM2-135M/ \
  --local-dir-use-symlinks False

cd /mnt/data/nano-vllm
python3 -c "
from nanovllm import LLM, SamplingParams
import os

llm = LLM(os.path.expanduser('~/huggingface/SmolLM2-135M/'), enforce_eager=True)
params = SamplingParams(temperature=0.7, max_tokens=100)
outputs = llm.generate(['Hello, world!'], params)
print(outputs[0]['text'])
"

Back Donate