PentaCMD-47M — English in, commands out
English in · Commands out
A decoder-only transformer, trained from scratch, that turns a plain-English instruction into a single terminal command across five tools — reaching ~86.7% exact-match, and 100% on git.
Install & run on a CPU-only machine
# clone + install the two deps$ git clone …/pentacmd$ pip install torch tokenizers# family hint + instruction$ python inference.py git \$ "undo my last commit"→ git reset --soft HEAD~1Live — real outputs, on a loop
### Task (git):
### Command:
// the idea
Beginners know what they want to do — not the exact command.
Big models can already do this. PentaCMD is the opposite bet: a tiny, specialised model built entirely by hand — data, tokenizer, architecture, training, evaluation, release — to learn the whole ML lifecycle by doing every step.
The same English (“go to the src folder”) maps to different commands in different shells. So the model is handed the target family as an input hint — never asked to guess it. Every example, every source, follows one locked three-field schema.
the prompt format
### Task (git): undo my last commit but keep my changes
### Command:
→git reset --soft HEAD~1
Because family is an input, the dedup key is (family, instruction) — identical English across different shells is kept and disambiguated by the hint.
From raw text to trained weights
Every example flows through one path: real and synthetic sources are merged, deduped, split leak-free, tokenized with a byte-level BPE trained on the train split only, then fed to a from-scratch transformer — checkpointed on exact-match, not loss.
SOURCES
NL2Bash + synthetic
DEDUP
(family, instruction)
LEAK-FREE SPLIT
90 / 5 / 5 · 0 overlap
BPE TOKENIZER
byte-level · vocab 12k
TOKEN .bin
~6.54M train tokens
TRANSFORMER
nanoGPT · 47.2M
TRAIN
T4 · exact-match ckpt
Dataset composition · 299,329 pairs
grown ~8× from a 36,329-pair first cut
git · 120,000
synthetic
python · 90,000
synthetic
bash · 41,329
NL2Bash ~11K + synth 30K
powershell · 28,000
synthetic
npm · 20,000
synthetic
Split leak-free 90/5/5 → train 269,396 / val 14,967 / test 14,966. Data went where it paid off — most to git and python (the families with the most headroom), least to near-ceiling npm and PowerShell. Verified: 0 commands and 0 instructions span splits.
Architecture
Decoder-only transformer, nanoGPT-style — pre-norm blocks, causal self-attention, GELU MLP, weight-tied LM head.
Shape
8 layers · 10 heads · 640 width · 256 context · 47.2M params (up from a 20.0M baseline).
Tokenizer
Byte-level BPE, vocab 12,000 — git, npm install, --soft, Get-ChildItem become single/short tokens.
Training
AdamW · cosine LR 6e-4→6e-5 · 300 warmup · batch 32 · fp16 · 14,000 steps · best at 11,000.
The results
Per-family exact-match
Thin white tick = first-word accuracy (the right command / verb): ~100% on the four synthetic families. Test ≈ validation (~86.7%) → no overfitting.
Training trajectory · exact-match on val
Structure is learned in the first 2k steps (first-word ~96%, dashed). The copy circuit comes online around step 3k, then a steep takeoff to the 86.7% peak at 11k.
first model · 20.0M · <1M tokens
0.0%
final V0 · 47.2M · ~6.54M tokens
0.0%
The problem-solving arc
Build the dataset from scratch
3-field schemaStarted from the real-world NL2Bash corpus (12,607 English↔bash pairs), then generated natural, beginner-phrased synthetic data for git / npm / python / PowerShell. Locked a clean three-field JSONL schema early.
Design for honesty
0 leakageMerged everything and split it leak-free — grouped by command so paraphrases like “install react” / “install react package” cannot leak across train and test.
First model — 20M params
58.7%A per-family breakdown was the key tool: ~90% first-word accuracy but the wrong argument (npm uninstall lodash → npm uninstall handlebars). bash, anchored by research-hard NL2Bash, sat at 6.7% and dragged the average.
Diagnose the real bottleneck
right metricTwo insights: validation loss was misleading (it rose while real accuracy climbed) — so switched to checkpointing on exact-match; and the model was data-starved — 20M params but under 1M tokens.
Scale data ~8× — for variety, not volume
×8 dataGenerated thousands of distinct package / branch / file names so the model learns to copy an arbitrary argument instead of memorising a small set. Grew to 299,329 pairs and a 47.2M model to match.
Result — 47M params
86.7%git solved at 100%, npm and PowerShell at 97–99%, and a model that copies arguments it has never seen — trained hands-off in ~53 minutes as a committed background job.
// honest engineering details
The trailing-space bug
A single trailing space in the inference prompt (### Command: vs ### Command:) made the model drop the first word of every command. Removing it instantly fixed outputs.
Background runs
Interactive Kaggle sessions kept dying on idle-timeout. Long jobs moved to committed “Save & Run All” runs that survive the laptop being closed.
How it was built
PentaCMD wasn’t built in a research lab — it was built solo and AI-native: AI copilots compress the loop from idea to implementation, while every decision that actually moves the score — the metric, the data strategy, the model size — stays a deliberate human call. The whole project is one tight loop that ran again and again.
Frame & research
Define the task and study transformers, tokenizers and training — AI as a research copilot to move fast.
Generate data
Build programmatic synthetic-data engines — templates × large entity pools — for natural beginner phrasing.
Train from scratch
A from-scratch nanoGPT on a free Kaggle T4 GPU, run hands-off as a committed background job.
Diagnose by metric
Per-family exact-match breakdowns expose exactly where — and why — the model is failing.
Iterate & ship
Feed findings back into data + model size; then document and publish to GitHub and Hugging Face.
// what I learned
Pick the right metric
For a leak-free split, validation loss misleads — it can rise while real accuracy climbs. Score on the task metric (exact-match), not loss.
Data variety beats volume
Argument-copying only generalises when the model sees thousands of distinct arguments — not more rows of the same few.
Diagnose before scaling
Per-family breakdowns showed bash was the anchor and git / python had the headroom — so data went where it actually paid off.
Small details matter
A single trailing space in the prompt silently broke every output. Tiny formatting bugs can masquerade as model failures.
Respect the tooling
Long jobs belong in committed / background runs — interactive sessions die. Reliable infrastructure is part of the model.
Live examples
Correct — verified, run locally on CPU
“undo my last commit but keep my changes”
→git reset --soft HEAD~1
“create a branch called payments and switch to it”
→git switch -c payments
“install the express package”
→npm install express
“create a virtual environment”
→python -m venv venv
“list the files in the Downloads folder”
→Get-ChildItem Downloads
“find all text files in the current folder”
→find . -type f -name "*.txt"
Honest misses — kept for transparency
“install pandas version 2.1.0”
✗pip install langchain==2.1.0
right structure, wrong package
“find all python files here”
✗find . -name "*.c"
right command, wrong extension
Use it yourself
Clone & install
Grab the repo and install the only two dependencies.
git clone …/pentacmd · pip install torch tokenizers
One-shot a command
Pass a family hint and an English instruction; read the command back.
python inference.py git "…"
Or go interactive
Run inference.py with no args for an interactive prompt loop.
python inference.py
Bring the weights
best_model.pt + tokenizer.json + modeling + inference — also on Hugging Face.
huggingface.co/SumanDebnath943/PentaCMD-47M
best_model.pt
~180 MB
trained weights + config
tokenizer.json
vocab 12k
byte-level BPE tokenizer
modeling_pentacmd.py
the GPT class
model architecture
inference.py
load + generate
one-shot or interactive
# python inference.py <family> "<instruction>"$ python inference.py git "create a branch called payments and switch to it"### Command: git switch -c payments$ python inference.py powershell "list the files in the Downloads folder"### Command: Get-ChildItem DownloadsGet the project
Code is MIT. Weights & synthetic data are CC BY-NC 4.0. The bash portion inherits NL2Bash’s own license — flagged in LICENSE, the README, and the model card.
The numbers
Version 0, honestly
Where it lags & why
python — 69.3%
Trained on five installers (pip / uv / poetry / conda / pipx), so “install X” is genuinely ambiguous about which tool. Package names are also long and multi-token, which makes exact copying harder.
bash — 68.0%
The test includes research-hard NL2Bash pipelines (placeholder args, nested -exec). Synthetic beginner-bash scores high; NL2Bash drags the average. ~68% is near the realistic ceiling for that data.
Plausible-but-wrong args
The model can emit the right structure with a wrong argument — generated commands should be reviewed before running.
Version 1 — in progress
WIP- 01Lift python 69 → 80s: regenerate data with pip as the dominant installer so “install X” is unambiguous.
- 02Lift usable bash: more synthetic beginner-bash; down-weight or trim the hardest NL2Bash rows.
- 03Scale further if warranted — more data and/or a larger model, same exact-match checkpointing.
- 04Live demo: wrap inference.py’s command_for() in a small web UI.
- 05Licensing: verify NL2Bash’s exact terms; loosen the model license if permitted.
open source · built from scratch
Small model. Big utility.
Clone the repo, install two packages, and translate your first instruction in under a minute. Weights, tokenizer, and model card are open on Hugging Face.
PentaCMD-47M · Version 0 released · Version 1 in progress · built by Suman Debnath