Processing Codeparrot Dataset Pipeline | Generated by AI
Done. Two new scripts committed:
| Step | Script | Produces |
|---|---|---|
| 1. Download | scripts/download/download_codeparrot_clean.py |
.json.gz shards in datasets/codeparrot-clean/ |
| 2. Convert | scripts/extract/convert_codeparrot_for_nanochat.py |
text-col parquet in datasets/codeparrot-clean-nanochat/ |
| 3. Tokenize | scripts/extract/tokenize_github_code.py (existing) |
.npy token shards in datasets/codeparrot-clean-tok/ |
Smoke test (valid set, 1 shard, 61k Python files) — passed ✅
To run the full pipeline (all 53 train shards + 1 valid shard, ~12.7 GB compressed):
Step 1 — download:
python3.11 scripts/download/download_codeparrot_clean.py
Step 2 — convert to nanochat format:
python3.11 scripts/extract/convert_codeparrot_for_nanochat.py
Step 3 — tokenize:
python3.11 scripts/extract/tokenize_github_code.py \
--input-dir /mnt/data/zz/datasets/codeparrot-clean-nanochat \
--output-dir /mnt/data/zz/datasets/codeparrot-clean-tok
Timing estimate: step 1 ~15-20 min (14 MB/s observed), step 2 ~20-30 min, step 3 ~10-15 min. Total pipeline ~1 hr for 5.3M Python files.
The dataset is pure Python — deduplicated, filtered for quality (autogenerated flag, alpha fraction, line stats). Exactly what you need for a Python-focused nanoGPT or nanochat experiment.