Ten Minutes, 16 Megabytes

The Ablation Stages

Eight focused sweeps anchored most of what follows. Each is labeled Stage N throughout the article and links back to its raw TSV. The storyline below is the bird's-eye view; the structural breakdown and per-cell contribution heatmap sit behind dropdowns.

Data

Hypotheses

1. H1 — Vocab size has a low ceiling. Smaller vocabularies are better, but not too small; around 8192 BPE is the right anchor for this challenge.
2. H2 — Tokenization should preserve modeling difficulty. If the per-token problem becomes too easy, this model family plateaus before the wall-clock budget runs out.
3. H3 — Full-word tokens are wasteful. Pruning long words that subword tokens can already reconstruct should free vocab budget without hurting BPB.
4. H4 — Capitalization is a redundant axis. Folding capitalized variants back into a lowercase token plus a <cap> marker should shrink the vocab usefully.
5. H5 — FineWeb noise dilutes the signal. Stripping long URLs and other rare, awkward patterns should improve per-token learning.

Findings

H1 — confirmed. Across many of the later successful PRs, 8192 converged as the practical sweet spot for SentencePiece BPE. We treated it as the anchor for everything that followed and only varied vocab size as part of structured reductions (see H4).

H2 — confirmed indirectly. Training on only 1024 tokens kept showing a low ceiling. The interpretation: if tokenization over-simplifies the sequence, this model family peaks too early under the wall-clock cap.

H3 — refuted. Pruning subword-coverable long words came out roughly +0.001 bpb, directionally worse than the baseline. Whatever vocab budget the prune freed up did not compensate for the harder modeling problem and longer sequences it created.

H4 — confirmed by Stage 5. The focused tokenizer sweep put numbers on the <cap> idea, sweeping a single knob p across cells.

Against the no-<cap> anchor, the sweet spot landed at p = 0.2 (220 tokens dropped, 1.027× sequence-length inflation): -0.00093 bpb, roughly twice the single-seed noise floor. Parameter savings from the smaller embedding bank offset the inflation cost cleanly. A more aggressive p = 0.5 traded a little BPB for far more bytes (-0.00050 bpb at -145 KB artifact) — the cleaner pick whenever the size budget is the binding constraint.

Two side findings under H4 turned out more interesting than the headline. At p=1.0 (remove every eligible cap token, 1.135× inflation), BPB was still slightly better than the anchor (-0.00020) — the full-removal cost was essentially zero, with parameter savings exactly offsetting the inflation. And uniform random removal underperformed inverse-frequency by only +0.00014 at the same vocab size, which suggests the dominant variable is the inflation ratio itself, not which specific tokens get removed. The corollary: a frequency-gated variant (keep tokens above a 1% frequency floor) was actively worse (+0.00058), because it kept the inflation cost without buying back the parameter savings.

H5 — partial. We wrote a corpus-level normalization pipeline that decodes each FineWeb document back to text, applies a small set of cleanups — long URLs, email addresses, repeated or unusual punctuation, and a handful of other rare-pattern rules — and re-encodes the result through the same SentencePiece tokenizer before writing back to the FineWeb shard binaries. Because normalization runs once as a corpus pre-pass, it has no training-time or eval-time cost; it only changes what the model sees. We kept the cleaned dataset in the final stack but never isolated its contribution in a clean ablation, so we treat it as a small confirmed-positive of unknown magnitude rather than a measured win.

PR Lineage

Model Architecture

The starting point was the 9-layer baseline. Depth recurrence was already there, and that community trick had already proved itself, so the question was what else to add or remove under a brutal training budget.

Hypotheses

1. A1 — MHA is worth the parameter cost. Replacing GQA (kv=4) with full MHA (kv=8) should buy enough quality to justify the extra KV-bank parameters.
2. A2 — A custom local-attention head can recover local structure cheaply. A per-head split with a sliding window should claw back BPB without breaking the budget.
3. A3 — Architectural ideas from frontier work transfer. DeepSeek-style engrams — which Kevin Clark brought to this challenge as value embeddings (PR #1218) — and embedding factorizations should help here too.
4. A4 — Attention-side gates carry real signal. SparseAttnGate (nprime06, PR #1787) and SmearGate (aquariouseworkman, PR #65), each layered on top of vanilla attention, should contribute measurable BPB.
5. A5 — The SmearGate BOS-mask matters under packed-document training. Without the cross-document leak fix from PR #1855 (codemath3000), the smear residual silently reaches across document boundaries.
6. A6 — Raw capacity beats clever components. Wider MLPs and more layers should win out over fancier architecture tweaks when training time is the binding constraint.
7. A7 — Smarter RoPE variants help. NTK-aware base scaling, YaRN long-context extrapolation, and alternate partial-rope schemes should improve on the inherited rope_dims=16 baseline.
8. A8 — Depth recurrence helps under a fixed parameter budget. Looping a small subset of layers extra times (sharing parameters but adding compute depth) — the trick from PR #1204 (msisovic, "Mini Depth Recurrence") popularized by PR #1394 (clarkkev) — should improve BpB without bloating the artifact.

Findings

A1 — toss-up on a fixed byte budget (Stage 4). Replacing GQA (kv=4) with MHA (kv=8) added KV-cache parameters that immediately overflowed the 16 MB cap, so MHA has to claw back the parameters from somewhere. We traded either MLP width (MLP_MULT 4.0 → 3.5) or one layer of depth (L=11 → L=10): MHA_M35 came in at 1.06513 (+0.00374 vs the GQA baseline of 1.06139) and MHA_L10 at 1.06352 (+0.00213). Stripped naively, both routes lose. But once the Stage 1/2 free wins are layered on top (see the follow-up paragraph), the gap shrinks to within seed noise — so we treat A1 as near-parity rather than a clean refutation. The unconstrained MHA win we had hoped for is being bought largely by extra MLP capacity, not by MHA itself; whether the architectural property of full multi-head attention is worth taking at near-zero net BPB is a judgment call.

Two follow-up observations from the same sweep tightened the picture. Shifting PARALLEL_START_LAYER from 8 to 7, a no-cost architectural knob, bought another -0.00047. And the Stage 1/2 free wins (LQER top_k=1, GateQuant on, TTT batch_size=16) stacked cleanly on top of MHA_M35 for -0.00277 bpb, bringing the best cap-legal MHA point within +0.00097 of the GQA baseline. That trio appears to be near-additively free across architecture choices.

Addendum (PR #1987): the toss-up tipped to MHA. Our final submission PR #1987 commits to MHA (kv=8) with MLP_MULT = 3.5 — i.e., the MHA_M35 shape from Stage 4 — once it's paired with the 9-hparam tuning stack from PR #1855 and the lrzip pergroup compressor (≈ −270 KB artifact savings). 3-seed mean lands at val_bpb 1.06184, beating the previous PR #1948 (GQA-baseline) by −0.00058. The Stage 4 numbers above remain a faithful read of MHA at that point in the search tree; they just understated how much surrounding-stack tuning was still on the table.

Addendum 2 (Stage 8 / PR1855NEW): a late scout flipped the sign again. We ported MHA to a newer pr1855 fork (PR1855NEW) and found that MHA hurts there by +0.00104 at single seed vs the same fork's GQA baseline (1.06217 MHA vs 1.06113 GQA). The newer fork has apparently absorbed via XSA, looping, and parallel-residual variants whatever MHA was capturing on the older base, so MHA on top is purely a byte cost without BPB benefit. There is also a methodology lesson worth flagging: a Stage 8 codebase ablation showed PR #1855's reported gap over v2b (−0.00134) came entirely from its 9-hparam tuned defaults, not from its structural codebase changes — running PR #1855 with v2b's config gave 1.06246 ± 0.00039, statistically indistinguishable from v2b's 1.06242 ± 0.00013. The reading we now favor across A1: MHA was less a structural win than a stepping stone — the route by which we surfaced the 9-hparam tuning stack — and the 9-hparam stack itself is what carries forward.

A2 — refuted by systems, not by theory. The implementation was inherently inefficient and could not take advantage of the optimized FlashAttention-style ecosystem, so whatever benefit it might have had on paper got crushed by practical cost.

A3 — refuted in this regime. We looked at several ideas from frontier work — most prominently DeepSeek-style engrams, which are structurally the same family as value embeddings: a separate small per-token embedding table, projected into the attention's KV dimension and added directly to V at the last few layers. We also explored embedding factorizations along the same theme. None of them paid off here. The cleanest representative we have a clean number for is Kevin Clark's 4096-vocab value-embedding stack, whose 3-seed mean lands at val_bpb 1.09785 — about +0.035 above the SmearGate-era frontier. With only about ten minutes of training time, this family of tricks simply does not become useful fast enough.

A4 — confirmed by Stage 1. The two attention-side gates carried most of the architectural budget on this stack. Removing SparseAttnGate cost +0.00356 bpb, the single largest swing in the sweep. SmearGate was worth another +0.00235 when ablated. Two smaller signals also showed up: replacing the sparse gate with a dense GatedAttn variant was strictly worse (+0.00180), and re-enabling the GateQuant flag that #1851 had deliberately turned off was essentially free (-0.00011) — a small neutral lever worth keeping in mind.

A5 — confirmed (Stage 1 B2 cell). The BOS-mask fix that distinguishes the #1851 SmearGate from the original #1797 version was worth +0.00056 on its own, almost exactly the size we had guessed before running the sweep.

not_bos = (input_ids[:, 1:] != BOS_ID).to(x.dtype).unsqueeze(-1)
x = torch.cat([x[:, :1], x[:, 1:] + g * x[:, :-1] * not_bos], dim=1)

A6 — confirmed. Vanilla attention plus MLP was extremely hard to beat under this budget. If anything reliably helped, it was making the MLPs wider; most other architectural tweaks did not show enough return on investment. Stage 4 reinforced the lesson from the other direction: pushing MLP_MULT from 3.5 down to 3.0 cost +0.00463, confirming that MLP capacity is the real budget bottleneck. Layer depth gave more BPB headroom than MLP width when shrinking to fit. The cleanest parameter-saving lever was unrelated to attention — embedding tying — and it was already on by default in the inherited baseline.

A7 — refuted in this regime. Both YaRN and NTK-aware base rescaling are designed for the case where inference sequence length exceeds training sequence length; under parameter-golf both phases run at seq_len = 2048, so the extrapolation regime never engages. We also tested several alternate partial-rope-dim values, and none beat the inherited rope_dims = 16 from the modded-nanogpt baseline. The final submission keeps standard partial RoPE: rope_dims = 16, rope_base = 1e4, no YaRN, no NTK scaling. The ROPE_YARN flag is wired into the hyperparameters but stays off in the submitted config.

inv_freq[i] = 1 / base ^ (2i / rope_dims)

A8 — confirmed (inherited from upstream). Depth recurrence originated as "Mini Depth Recurrence" in PR #1204 (msisovic) and was popularized via PR #1394 (clarkkev) with a simpler "loop layers 4–5 twice" implementation, then carried forward through dexhunter's SP8192 + QK5 baseline into our stack. In the final submission we tightened the schedule slightly: NUM_LOOPS = 2, LOOP_START = 3, LOOP_END = 5, ENABLE_LOOPING_AT = 0.35 — looping layers 3–5 (three layers, each visited three times, all parameters shared) and switching the schedule on at 35% of training instead of the original 50%. Total per-step layer applications go from 11 (vanilla) to 17 at the same parameter count.

[0, 1, 2, 3, 4, 5, 3, 4, 5, 3, 4, 5, 6, 7, 8, 9, 10]

PR Lineage

Optimizer

Hypotheses

1. O1 — AdamW should not be that bad at small scale. On a relatively small dataset, standard AdamW with tuned betas usually holds its own against fancier optimizers; we expected it to come within striking distance of Muon.
2. O2 — Muon's communication overhead is the dominant remaining slack. Reducing the post-backward collective traffic (a reordering we called the "0427 trick", inspired by the observation that DDP already syncs gradients) should give a meaningful throughput win — either by removing all-reduce wire time, or by exposing whatever else is hiding under the optimizer step.
3. O3 — torch.compile knobs unlock training-time throughput. Disabling compile or switching to dynamic shapes might leave more wall-clock for actual training inside the 600 s cap.
4. O4 — Larger effective batch with LR scaling improves BPB. Doubling effective batch with sqrt(k) or linear LR scaling should beat the 1.0x baseline.

Findings

O1 — refuted. In our experiments, AdamW consistently lagged Muon. The "small-scale parity" intuition turned out to be wrong, and we did not look back. From there we spent time looking inside Muon itself to see where the remaining slack might be.

O2 — mostly refuted by Stage 6. The hypothesis named communication overhead — specifically wire time spent in collectives — as Muon's dominant slack. We tested it with the 0427 trick: a Muon variant that exploits the fact that DDP's backward already synchronizes gradients across ranks, so a deterministic Newton-Schulz iteration can run independently on every rank, skipping the extra all_gather and reduce_scatter a sharded Muon would otherwise need. Same-shape matrices across layers are batched into a single NS call. The result was a small win, about +0.0005 bpb.

Stage 6 made the cost structure quantitative. Out of a 4.82 ms training step, exposed collective wait time was only 112 µs — about 2.3% of the step. The wire we had been targeting was already well-pipelined; the headroom for any "save wire time" intervention was tiny by construction. The real cost was hiding inside the optimizer's packed-grad path: opt.ar_packed took 2.77 ms per step (~57% of the whole step), but only 56 µs was wire — the remaining ~2.7 ms was tensor pack/unpack overhead (cat + copy_ of many small gradients inside _all_reduce_packed_grads). A fused pack via torch._foreach_* or a custom kernel could plausibly recover 20–30% throughput at 1.0x batch.

Reading the +0.0005 win against that profile: the wire-time saved by removing two extra collectives is on the order of ~100 µs/step (well under 5% of the eliminated collectives' wall time), which would translate to a barely-measurable BPB delta. The pack/copy work skipped on those same collectives is on the order of ~2 ms/step — much closer in magnitude to the win we actually observed. So the cleanest reading is that the 0427 trick worked, but not for the reason the hypothesis named: it bought a little pack/copy savings as a side effect of removing collectives, while the wire-time mechanism the hypothesis pointed at was never the bottleneck.

Verdict: O2 is mostly refuted. The cleanly-formulated version would be "pack/copy in the optimizer's gradient path is Muon's main slack" — a different mechanism than "communication overhead", and one we could only state confidently after Stage 6 surfaced it. The 0427 trick is still worth keeping (small gain in the predicted direction), but it is a partial workaround for a problem we now know is upstream of the collective itself.

O3 — refuted by Stage 2. Turning torch.compile off entirely crashed at serialization — the surrounding codepath quietly assumed compile-on. Switching the main training compile from static to dynamic=True was strictly worse (+0.00258 bpb vs the compile-on diagnostic baseline). Under a 600 s wall-clock cap there was no hidden throughput win in this knob; static compile was already the right default.

O4 — refuted by Stage 3. The wall-clock cap made the headline simpler than we expected: doubling effective batch at a fixed LR reliably hurt. The 1.0x baseline came in at 1.06342 bpb; 1.5x batch at the same LR landed at 1.06570 (+0.00228), because fewer effective updates fit in 600 seconds. The classic sqrt(k) rule helped slightly at 1.5x (-0.00039 vs fixed) and linear LR over-corrected (+0.00057). A sub-sqrt Muon factor looked promising at 1.75x batch, but those larger-batch runs were not iso-step, so a clean rerun is needed before reading too much into them.

The harder boundary under O4 was memory. Every 2.0x batch cell OOMed in fused-CE backward on 80 GB H100s, regardless of LR scale. The practical ceiling on this stack was around 1.75x, and even there throughput collapsed (about 2.6 Mtok/s versus 6.6 Mtok/s at 1.0x). Under this budget, larger batch simply was not the right axis to push.

PR Lineage

Quantization

Quantization still felt somewhat like a black box to us, even though we had done it before. Two questions kept coming up: which compensation tricks pay off, and are BPB and the artifact-byte cap independent constraints we can tune separately.

Hypotheses

1. Q1 — Group quantization improves GPTQ. Per-group statistics should give a more stable estimate of weight distributions and improve quantization quality on this stack.
2. Q2 — QAT should outperform GPTQ-style PTQ. Quantization-aware training, with quantization in the loop, should give the model time to adapt and beat post-training quantization.
3. Q3 — LQER has substantial headroom. Low-rank quantization error compensation, layered on top of GPTQ, should add roughly +0.001 to +0.003 BPB beyond GPTQ alone.
4. Q4 — More aggressive matrix-bit quantization is a viable tradeoff. Pushing matrix bits below the int6 default should buy artifact bytes at an acceptable BPB cost.
5. Q5 — BPB and byte budget are independent constraints. Levers that improve BPB shouldn't fundamentally have to trade against the 16 MB artifact cap.

Findings

Q1 — refuted. Group quantization did give the predicted statistical-stability improvement, but the per-group scale storage compounded across every quantized matrix. The BPB gain was on the order of +0.0001 bpb (under the single-seed noise floor on this stack), while the extra scale storage cost 300–450 KB after compression — enough to push the cap-legal artifact past the 16 MB limit. Cleanest concrete example of Q5: BPB and bytes are joint constraints, and group quantization sits on the wrong side of that joint.

Q2 — untested. Our intuition still says QAT should work better than GPTQ-style PTQ on a long enough training run, but we never got a successful QAT run during this round. Inside a 600 s wallclock cap the engineering overhead of a working QAT loop kept eating the gain we were hoping it would unlock.

Q3 — refuted by Stage 1. LQER in total was worth about +0.00038 bpb when fully ablated — well below the +0.001 to +0.003 range we had loosely predicted. Most of the LQER sub-knobs (sym vs asym, group sizes, factor bits) sat within noise of one another, and rank=8 actively hurt (+0.00077). One small, real win surfaced: dropping LQER top_k from 3 to 1 improved BPB by 0.00044. The default appears to over-spend compensation on tensors that did not really need it.

Q3 addendum (Stage 8): the LQER top_k optimum is codebase-dependent. We re-ran the LQER landscape (top_k ∈ {1, 2, 3, 4, 5}) on the PR #1855 base and found top_k = 2 wins (mean 1.06211, σ 0.000056 — the lowest σ in the whole landscape, and a clean −0.00052 vs the top_k=3 anchor). On the same codebase, top_k=1 is statistically indistinguishable from top_k=3. The Stage 1 result still holds for the PR #1851 codepath; the takeaway is that LQER's compensation sweet spot interacts with the surrounding stack rather than being a single-codebase constant.

Q4 — refuted by Stage 1 and Stage 2. Pushing matrix bits more aggressive (down to 5) was a disaster at +0.0269 bpb. Stage 2 confirmed the floor more cleanly: matrix_bits=4 was unrecoverable (pre-quant BPB jumped to 1.43), and adding LQER rank=8 as a rescue did not bring it back (1.28 vs the 1.06182 anchor). Int6 sits near the actual floor for this model size; the LQER compensation is not the bottleneck that is saturating below it.

Q5 — refuted (the most consequential finding). Stage 1 made the coupling impossible to ignore: embed_bits=8 and a looser MLP clip were the two settings that genuinely improved BPB (-0.00080 and -0.00141) — but both pushed the artifact past the size limit. Stage 2 quantified the other direction: dropping embed_bits from 8 to 6 cost only +0.00080 bpb while shrinking the artifact by 242 KB — one of the cleaner BPB-vs-bytes tradeoff levers in the stack. The takeaway across both sweeps: bit-budget gains and BPB gains are joint constraints that have to be co-optimized, not independent knobs.

PR Lineage

Test-Time Compute

Test-time compute felt the most like a backdoor lottery ticket. The space was noisy, and a lot of it felt less principled than the earlier parts of the stack. The guiding heuristic, at least from our perspective, was to align the trained distribution with the test-time distribution as much as possible — beyond that, many of the gains felt opportunistic rather than deeply structural.

Hypotheses

1. T1 — TTT itself gives meaningful BPB headroom. Letting the model train briefly on each test chunk before scoring should beat the frozen-model baseline by a clearly measurable margin.
2. T2 — Multi-phase TTT beats single-phase. Splitting TTT eval into multiple cumulative phases (each phase trains on the data scored in earlier phases) should beat a single all-at-once pass.
3. T3 — Smaller TTT batches are too noisy. The standard intuition says larger batches give cleaner LoRA gradients, so the bs=64 baseline should beat bs=16.
4. T4 — SMT (Sparse Matrix Tuning, our own technique from arXiv 2405.15525) at TTT should help. Selecting only the top-K highest-gradient 64×64 blocks per matrix and zeroing updates outside that mask should focus TTT capacity on the parameters that matter and beat full LoRA TTT.

Findings

T1 — confirmed by Stage 1. TTT was the place where the numbers most clearly justified the engineering. The baseline post-TTT score (1.06172) sat ~0.013 bpb below the pre-TTT floor (1.07507). That single delta was several times larger than every gated-attention or quantization knob we touched — TTT is by a wide margin the largest single lever in this search tree.

T2 — confirmed. Phased TTT preferred three phases: collapsing to one cost +0.00118, while five phases were within noise. The alpha and weight-decay values inherited from S9 were each slightly worse than the #1851 defaults, by small margins. A side wrinkle: replaying the older #1797 configuration on top of the #1851 stack came out at 1.06149 bpb, very slightly ahead of the #1851 baseline (1.06172). The two changes (BOS-mask off, GateQuant on) interact non-additively — at this resolution, leaderboard ordering is not a clean total order even within a single code lineage.

T3 — refuted by Stage 2. The data went the other way. At fixed LoRA rank 96, dropping batch size from 64 to 16 improved post-TTT BPB from 1.06195 to 1.06169. Going to rank 192 with batch size 16 was the best of the four cells (1.06166). The "noisier gradient hurts BPB" story did not hold up — TTT seemed to prefer more, smaller updates than the conventional bigger-batch-cleaner-gradient intuition suggests.

T4 — refuted. SMT selects the top-K (default keep_frac = 0.25) 64×64 blocks of each matrix's gradient and zeros updates outside that mask. The mask is computed once at chunk 0 by running a single forward+backward over the chunk's training-eligible sequences (_smt_select_masks), then frozen and applied as a per-step gradient mask for every remaining chunk. In principle, focused updates on the highest-gradient blocks should be cleaner than spreading TTT capacity thinly across the whole matrix.

In practice, the chunk-0 gradient signal turned out to be too unstable a basis for a frozen mask. The blocks that look most active in the first chunk are not consistently the blocks that matter for the rest of TTT — the test-time distribution drifts across chunks, and any one-shot mask bakes in the early signal at exactly the moment when the model has the least information. Full-rank LoRA TTT, which gets to re-allocate capacity at every chunk, was meaningfully more stable end-to-end. The final submission keeps SMT off and uses standard LoRA rank 80, batch size 16, three phases Phased TTT instead.

PR Lineage

A Note on Working with Agents

Looking back at the search tree above, much of the substance was carried by agentic LLMs. Codex and Claude Code were not adjacent helpers — they were inside almost every step. We wrote prompts; they wrote scripts. We sketched a sweep; they wired up the runner, kicked off training on the HPC, scraped the logs, parsed the TSVs, ran the regressions, and handed back a summary that often included a sensible next experiment to try.

There was a threshold somewhere in the middle of the project where the loop tightened to the point of feeling almost autonomous. Once we had fed enough context — the codebase, the constraints, the prior PRs, the way the budget was being scored — the agent could take a one-line research goal, build the conda environment on a fresh HPC node, run the experiment to completion, write the results to disk, generate the diff against the previous run, and propose what it thought we should look at next. We mostly nudged.

This is not entirely a happy observation. The work above includes plenty of moments that felt genuinely ours — the choice of hypotheses, the willingness to keep a tossed-up MHA result, the suspicion that the SmearGate BOS-mask was the right size. But the rote engineering — the part that, ten years ago, would have been the work — was largely automated away. The fraction of the effort that was distinctively human kept shrinking.

We are not lamenting the agents themselves; they are extraordinary, and this project would not have been possible at this depth without them. What is harder to sit with is how fast the line between operator and author moved. There is a version of this writeup that could have been, almost entirely, a transcript of what we asked the agents and what they handed back. We did not write it that way — but the fact that we could have is what makes the lament a real one.

A second thing worth flagging is the substrate. The agentic loop runs on hardware that has not gotten cheaper or more available. A 600-second wall-clock cap on 8× H100s is a small experiment by frontier standards, but every iteration of the search tree above still cost real GPU-hours. Most of the project's calendar time was spent waiting for jobs to slot into a shared queue, not waiting for the agent to think. Compute is now the binding constraint on how fast agentic research can move — not the agent's bandwidth, not the human's. We were lucky to have access; many researchers in this design space are not.

A third thing to say out loud is that the slow parts left in the loop are the parts that most need a human. Picking which hypothesis is worth a sweep, recognizing that the chunk-0 SMT mask was unstable rather than just noisy, deciding to call A1 a "toss-up" instead of a "refutation" — those judgment calls were ours. The agents could implement and measure, but the framing of what to measure, and the read of why a number was the size it was, still benefited noticeably from the human. Research efficiency has gone up enormously; the parts that didn't speed up are the parts where intuition and an honest second look at the data still matter, and for now those still seem to be a place where human researchers have an edge over LLM agents. We expect that gap to close. It hasn't yet.

An aside on autoresearch. Alongside the manual sweeps we also ran a Karpathy-style autoresearch hive — 24 agents across desktop 4090s, a 6× 4090 box, and a cloud H100 node, following the standard design (LLM proposes a train.py diff → fixed wall-clock train+eval → val_bpb → git commit if better, reset if not). After 1,114 completed runs over April, the hive's best result was val_bpb ≈ 1.0906 — about 0.029 bpb behind what two humans did in the same window.

A few reasons the loop didn't carry the way it does on nano-gpt: there is no cheap proxy here — parameter-golf's eval is the 10-minute hard cap, so the 12-experiments-per-hour flywheel that makes the ratchet compound on nano-gpt never spun up. With 24 agents each following their own greedy ratchet, they drifted to different local minima rather than converging on one. The wins that mattered for the 1.06 frontier weren't single-knob — they required correlated multi-knob optima (the 9-hparam stack), counterintuitive direction reversals (MHA only paying off when paired with the right surrounding tuning), specific bug fixes (the SmearGate BOS-mask), and byte-budget-aware engineering (lrzip pergroup) — none of which a greedy one-step ratchet can surface. And the ratchet has no mechanism for committing to a worse-looking intermediate, which is exactly how the manual track escaped the 1.09 basin.

Honest read: the hive reproduced roughly what the modded-nanoGPT community had already published — useful, but not a frontier-mover here. The 1.06 frontier moves were all human-found; the agent's contribution kicked in after a human had picked the direction.

Citation

If you would like to cite this writeup:

@misc{li2026paramgolf,
  author       = {Li, Billy and Shen, Tim},
  title        = {Ten Minutes, 16 Megabytes: A Parameter-Golf Field Report},
  year         = {2026},
  month        = apr,
  howpublished = {\url{https://www.junchengbillyli.com/llm-notes.html}},
  note         = {Submission: \url{https://github.com/openai/parameter-golf/pull/1987}}
}