karpathy/nanoGPT completed
https://github.com/karpathy/nanoGPT (opens in a new tab) · run 36eec7b0-9f2d-4382-a1aa-902b04420262 · 2026-08-26 10:40 UTC
Summary
Coverage of the repository is partial, as two non-text assets were skipped and left unread according to the inventory.
The nanoGPT repository is designed for training and finetuning medium-sized GPT models, serving as a simplified rewrite of minGPT. The codebase includes model definitions, training scripts, sampling tools, and preparation scripts for datasets such as Shakespeare and OpenWebText, along with Jupyter notebooks analyzing transformer sizing and scaling laws.
The project is organized around core Python modules including train.py, model.py, sample.py, and bench.py, accompanied by configuration files and dataset preparation scripts. Dependencies involve PyTorch, numpy, transformers, datasets, tiktoken, wandb, and tqdm.
What this summary is based on: read 24 of 24 selected files; sent 65 of 65 extracted chunks. Nothing was left out.
Generated by gemini-3.5-flash-lite from ~22421 estimated tokens of extracted text (chars//4 heuristic, not a tokenizer). Claims in the prose should be checkable against the file inventory below.
Repository
| Requested ref | (default branch) |
|---|---|
| Resolved ref | master |
| Default branch | master |
| Language | Python |
| Description | The simplest, fastest repository for training/finetuning medium-sized GPTs. |
| Stars | 62502 |
Classification
Every entry GitHub returned is accounted for: 6 directories + 24 selected + 2 skipped + 0 failed = 32 entries.
Why files were skipped
| Reason | Count | |
|---|---|---|
| non_text_asset | 2 | view |
Extracted content
24 of 24 selected files were read, producing 65 chunks.
Showing 20 of 65 chunks, ordered by source path. Every chunk records where it came from and where in that file.
| Source | Type | Chunk | Cell | Chars | Preview |
|---|---|---|---|---|---|
| .gitattributes | code | 0 | — | 214 | # Override jupyter in Github language stats for more accurate estimate of repo code languages # reference: https://githu… |
| .gitignore | code | 0 | — | 100 | .DS_Store .idea .ipynb_checkpoints/ .vscode __pycache__/ *.bin *.pkl *.pt *.pyc input.txt env/ venv/ |
| LICENSE | code | 0 | — | 1072 | MIT License Copyright (c) 2022 Andrej Karpathy Permission is hereby granted, free of charge, to any person obtaining a… |
| README.md | markdown | 0 | — | 13846 | # nanoGPT  --- **Update Nov 2025** nanoGPT has a new and improved cousin called [nano… |
| bench.py | code | 0 | — | 4815 | """ A much shorter version of train.py for benchmarking """ import os from contextlib import nullcontext import numpy as… |
| config/eval_gpt2.py | code | 0 | — | 208 | # evaluate the base gpt2 # n_layer=12, n_head=12, n_embd=768 # 124M parameters batch_size = 8 eval_iters = 500 # use mor… |
| config/eval_gpt2_large.py | code | 0 | — | 215 | # evaluate the base gpt2 # n_layer=36, n_head=20, n_embd=1280 # 774M parameters batch_size = 8 eval_iters = 500 # use mo… |
| config/eval_gpt2_medium.py | code | 0 | — | 216 | # evaluate the base gpt2 # n_layer=24, n_head=16, n_embd=1024 # 350M parameters batch_size = 8 eval_iters = 500 # use mo… |
| config/eval_gpt2_xl.py | code | 0 | — | 213 | # evaluate the base gpt2 # n_layer=48, n_head=25, n_embd=1600 # 1558M parameters batch_size = 8 eval_iters = 500 # use m… |
| config/finetune_shakespeare.py | code | 0 | — | 645 | import time out_dir = 'out-shakespeare' eval_interval = 5 eval_iters = 40 wandb_log = False # feel free to turn on wand… |
| config/train_gpt2.py | code | 0 | — | 681 | # config for training GPT-2 (124M) down to very nice loss of ~2.85 on 1 node of 8X A100 40GB # launch as the following (… |
| config/train_shakespeare_char.py | code | 0 | — | 1132 | # train a miniature character-level shakespeare model # good for debugging and playing on macbooks and such out_dir = '… |
| configurator.py | code | 0 | — | 1758 | """ Poor Man's Configurator. Probably a terrible idea. Example usage: $ python train.py config/override_file.py --batch_… |
| data/openwebtext/prepare.py | code | 0 | — | 3167 | # saves the openwebtext dataset to a binary file for training. following was helpful: # https://github.com/HazyResearch/… |
| data/openwebtext/readme.md | markdown | 0 | — | 489 | ## openwebtext dataset after running `prepare.py` (preprocess) we get: - train.bin is ~17GB, val.bin ~8.5MB - train h… |
| data/shakespeare/prepare.py | code | 0 | — | 1132 | import os import requests import tiktoken import numpy as np # download the tiny shakespeare dataset input_file_path = … |
| data/shakespeare/readme.md | markdown | 0 | — | 161 | # tiny shakespeare Tiny shakespeare, of the good old char-rnn fame :) After running `prepare.py`: - train.bin has 30… |
| data/shakespeare_char/prepare.py | code | 0 | — | 2344 | """ Prepare the Shakespeare dataset for character-level language modeling. So instead of encoding with GPT-2 BPE tokens,… |
| data/shakespeare_char/readme.md | markdown | 0 | — | 209 | # tiny shakespeare, character-level Tiny shakespeare, of the good old char-rnn fame :) Treated on character-level. Af… |
| model.py | code | 0 | — | 16345 | """ Full definition of a GPT Language Model, all of it in this single file. References: 1) the official GPT-2 TensorFlow… |
The Cell column is the notebook cell this chunk came from; it is blank for files that are not notebooks. Notebook markdown and code cells are kept as separate chunks.