Looped mdlARC: 37% on ARC-AGI-2 Kaggle with a 100M transformer trained from scratch


Contents

This is the third version of mdlARC, a series where I try to find out how far an autoregressive transformer can get on ARC when only trained on ARC puzzles -- no text, no LLMs (Vakde, 2025, Vakde, 2026).

The 37% model is a 100M looped autoregressive transformer. It was pretrained offline on ARC puzzles and then finetuned on the hidden tasks' demos inside Kaggle’s 12 hrs on 4xL4s.

The pretraining dataset was 3,826 ARC puzzle families: the public ARC-AGI-2 tasks and combinations of them made by LLM-written generators. 42 variations of each puzzle with another LLM pipeline.

New records on Public Eval: 70% on ARC-1 with a 19M model

20304050607080100K1M10M100MParameters trained (log scale)ARC-AGI-1 public eval, top-2 (%)↖ betterCompressARC: 20%. A 76K network trained on each puzzle alone, with no pretrainingCompressARC: 20%76K, trained on each puzzle aloneTRM: 44.6%. A 7M network plus about 430M of per-puzzle embeddingsTRM: 44.6% (7M + 430M embeddings)HRM: 40.3%. A 27M network plus about 538M of per-puzzle embeddingsHRM: 40.3% (27M + 538M embeddings)mdlARC v1, Dec 2025: 27.5%, 28M parametersmdlARC v2, Mar 2026 (the “44% in 67 cents” post): 44%, 76.6M parametersmdlARC v1: 27.5%Dec ’25mdlARC v2: 44%Mar ’26Looped mdlARC, 1.05M parameters: 49.0%Looped mdlARC, 2.0M parameters: 55.625%Looped mdlARC, 8.4M parameters: 66.25%Looped mdlARC, 18.9M parameters: 70.25%1.05M: 49.00%2.0M: 55.63%8.4M: 66.25%Looped mdlARC (this post)19M: 70.25%+ RE-ARC, scale aug and answer filtersLooped mdlARC 19M + RE-ARC, scale augmentation and answer filters: 81.38%. Without RE-ARC: 71.88%81.38%72× fewer parameters, higher score

Figure 3. ARC-AGI-1 public eval against parameters trained, log scale. Looped mdlARC (blue) scores higher than HRM, TRM, CompressARC and my earlier models with far fewer parameters. HRM and TRM are plotted at their full trained size: the small network they report plus their per-puzzle embeddings. CompressARC trains a 76K network on each puzzle alone. The 19M point adds ranking by the model’s own likelihood (68.38 without). The top point is the same model with RE-ARC, scale augmentation and answer filters added; RE-ARC accounts for 9.5 of its 11 extra points (appendix B.7). It is outside the 1,307-task setting. Hover a point for details.

ARC-AGI Public Eval was used for quickly testing new ideas. These were trained only on 1307 puzzles (official public puzzles + ConceptARC dataset; this includes the evaluation tasks’ demos, never their answers). Training was transductive like HRM/TRM.

I got many surprising results:

This is great as a sample efficiency benchmark, so I'm launching it as ARC Data Squeeze:
"How high can you score when trained on just these 1307 puzzles"

Full results are in appendix B.

Results vs other non-LLM approaches

MethodParametersTest-time trainingARC-AGI-1ARC-AGI-2
Looped mdlARC, 19M18.9M—70.25 / 79.8815.14 / 20.56
TRM (Jolicoeur-Martineau, 2025)7M + ~430M per-puzzle embeddings—44.67.8
HRM (Wang et al., 2025)27M + per-puzzle embeddings—40.35.0*
CompressARC (Liao and Gu, 2025)76Keach task alone, from scratch20—

* As reported by TRM; HRM’s paper gives only the ARC-AGI-1 score. ARC Prize re-ran HRM on its semi-private sets and measured 32 and 2 (ARC Prize, 2025). Looped mdlARC’s ARC-AGI-2 runs reuse the ARC-AGI-1 recipe unchanged. HRM and TRM also train with translation augmentation, placing each grid at a random position on a 30×30 canvas; my models don’t use it, and ARC Data Squeeze doesn’t allow it.

Many other results add RE-ARC (Hodel, 2024), which generates 1000 new examples of the 400 ARC-AGI-1 training tasks. When I add RE-ARC, my model is still SOTA at an incredible 81% on ARC-1.The 70.25 / 79.88 record adds ranking by the model’s own likelihood. The Kaggle writeup quotes 71.88 / 81.50 for the same model with scale augmentation and rule-based answer filters borrowed from transductARC; ARC Data Squeeze’s rules don’t allow those, so they count only for its Open table. The RE-ARC runs use them too; against that run, RE-ARC itself adds 9.5 points (appendix B.7).

MethodParametersExtra dataTest-time trainingARC-AGI-1ARC-AGI-2
Looped mdlARC, 19M + RE-ARC18.9MRE-ARC—81.38 / 88.0020.14 / 32.08
transductARC (Schweighofer, 2026)~75MRE-ARC—76~17
Loop-OWM (Gao et al., 2026)10.6M + ~2M per-task tablesRE-ARC, ARC-GENyes68.522.5
LoopViT (Shu et al., 2026)18MRE-ARCyes65.814.2
VARC (Hu et al., 2025)18MRE-ARCyes54.58.3

transductARC’s numbers are means over several seeds; its best of 8 seeds scores 77.5 and 19.4.

Looping is the huge win

For months, my 1B non-looped models plateaued at 30–32% on Kaggle (the best was 33.89). I added loops in the last 3 days of the competition. Both looped submissions (76M, 100M) beat every non-looped one on the first try.

On ARC Data Squeeze, an 8.4M model went from 47.44 to 66.25 with looping. At the same logical depth, it beats non-looped ones with 4× more parameters, by 2.3 to 7.6 points. Full table and caveats: appendix B.3.

4550556065708121620242832Depth: block passes per tokenARC-AGI-1 top-2 (%)Unique layers, 8 blocks, 8.4M: 47.44Unique layers, 12 blocks, 1.5× params: 53.56Unique layers, 16 blocks, 2× params: 55.75Unique layers, 24 blocks, 3× params: 56.25Unique layers, 32 blocks, 4× params: 58.631.5× params2×3×4×Looped, 8.4M, no loop: 47.44Looped, 8.4M, ×2: 55.88Looped, 8.4M, ×3: 60.13Looped, 8.4M, ×4: 61.88Looped, 8.4M, ×6: 66.25×2×3×4×6no loop: 47.44Looped: 8.4Mat every depthUnique layers:more blocks,more params

Figure 4. Looping against unique layers at the same depth, ARC-AGI-1. The looped model stays at 8.4M parameters at every depth (blue). The unlooped models add blocks and parameters (grey) and still score lower. The ×6 point uses deep supervision.

Novelty and significance

The most important contribution: an existence proof that LLMs aren’t needed {#main-contribution}No pretrained LLM is in the model or used at inference. LLMs did write the generators for its pretraining puzzles (appendix A.2). But the result still holds. On ARC Data Squeeze there is no LLM anywhere, not even to make data.

This matters because the field converged on LLMs, concluded nothing else works, and stopped trying on Kaggle. Nearly every top ARC-AGI-2 entry fine-tunes an LLM. Apart from mine, the best non-LLM writeup scores a very low 12% (HRM/TRM stayed under 8%).

Hence, more than any technique, I think my most important contribution is to show that this is possible so that the field doesn't stay stuck working on LLMs.

Looped mdlARC is the existence proof and counterexample: it's trained on ARC puzzles alone with pure DL (no RL) yet it scores 37%, matching Opus 4.5 on Max reasoning. Today's frontier reasoning LLMs obv score far higher, but 37% is nowhere near this model’s ceiling. It had to fit Kaggle’s 4xL4s, it’s badly undertrained because looping went in during the competition’s last 3 days, and its score was still rising at the deadline.

There’s a second reason this matters. François Chollet created ARC partly as an argument against pure deep learning: he bet that solving it would need program search (Chollet, 2019). He had written that reasoning and abstraction are “out of reach for deep learning models”, however much data you give them (Chollet, 2017).

Before this result, reasoning LLMs were the only method to score this high on ARC-AGI-2. However, they don’t prove him wrong: Chollet reads their long chain of thought as a search, and called o3 “a form of deep learning-guided program search” (Chollet, 2024). Evidence: Without search, base LLMs scored 0% on ARC-AGI-2 during launch, and even now perform abysmally (ARC Prize, 2026).

Looped mdlARC does no search at all, so it’s a far stronger result: no chain of thought and no program search, just the answer written one cell at a time. Even its test-time adaptation is plain gradient descent. It still matches Opus 4.5 on Max reasoning, the strongest evidence yet that pure deep learning may be enough. It’s astounding to see some of the puzzles it solves, especially ones where people need genuine multi-step reasoning. Take ARC-AGI-2 task 7b80bb43 (Figure 5). Almost no frontier model solves it (2 of the 66 model settings ARC Prize has published answers for, and Opus 4.5 at none of its thinking budgets), yet most people do (7 of the 9 who tried it). My 19M ARC Data Squeeze model, trained from scratch on public ARC tasks alone, gets it right.

ARC-AGI-2 public evaluationtask 7b80bb43

Demo pairs

Test input

→

The 19M model’s answer

Figure 5. A hard ARC-AGI-2 task the model solves. Some lines have a piece knocked loose into a short diagonal; the answer removes each diagonal piece and closes the gap it left, while gaps with no diagonal next to them stay open. Of the 66 AI model settings ARC Prize has published answers for, only 2 solve it (Claude Opus 4.6 at max thinking and Gemini 3.1 Pro), and Opus 4.5 fails at every thinking budget; 7 of the 9 people who tried it solved it. Shown is the top-voted answer of the 19M ARC Data Squeeze model, trained from scratch on public ARC tasks only.

To be clear, I don’t think pure deep learning (compression) is the complete picture. My result points out that (1) the entire field drastically underestimates how powerful pure compression (just next token prediction, without search) is and (2) I think it will be enough to solve ARC-AGI-1 and 2. But the ceiling for intelligence is much higher, and we need new ideas for this.

Lastly, I agree with Chollet's definition of intelligence: "skill-acquisition efficiency, given priors and experience" (Chollet, 2019). LLMs make that nearly impossible to measure, because pretraining on the internet gives them priors no one can account for. Measuring it means controlling both: training from scratch on ARC puzzles alone (no hidden priors) and on a fixed set of them (fixed experience). That’s what ARC Data Squeeze, the benchmark I’m launching, does.

Technical ideas not tried on ARC before

Borrowed, with credit. The rest mixes ideas others have used, on ARC or elsewhere: the whole puzzle in context (Franzen et al., 2024), voting over augmented views (Franzen et al., 2024), test-time training (Akyürek et al., 2024), multi-token prediction (Gloeckle et al., 2024; Schweighofer, 2026) and drafting tokens with it (Stern et al., 2018), deep supervision (Lee et al., 2014; Jolicoeur-Martineau, 2025) and input injection (Geiping et al., 2025). Everything else is a standard decoder-only transformer.

New Benchmark: ARC Data Squeeze!

ARC Data Squeeze is a sample efficiency benchmark. Train from scratch on the 1,307 public tasks (about 2M tokens) and score as high as you can on ARC-AGI-1 and ARC-AGI-2, with any architecture and any compute. No LLMs, no pretrained weights and no outside data. Like the NanoGPT slowrun (NanoGPT slowrun, 2026), it fixes the data and lets compute vary, but it scores what the model solves, not its loss. Hand-written rules, DSLs and program search are off limits too, so the gains have to come from learning (rules).

I think sample efficiency is an important ML problem. LLMs are trained on so much data that it becomes difficult to discern how strong the learning algorithm is. Fixing the data also makes the score a measure of Chollet’s skill-acquisition efficiency.

On ARC, it’s now obvious that with enough data both ARC-1 and ARC-2 would become trivial: RE-ARC’s generated examples alone add 9.5 points to my model (appendix B.7) and other top submissions are Qwen models post-trained on lots of synthetic data. On the other hand, it’s super interesting to know how far a model gets when data is severely restricted.

The current SOTA is my looped mdlARC v3: 19M parameters, 70.25% on ARC-AGI-1 and 15.14% on ARC-AGI-2, for about $13 of compute per run (full results).

Open challenges: Can a 1M model reach 70% (49% today)? What's the best looping implementation? How do newer research directions like FBTs, recirculation, H-nets perform? The grand one: can we beat GPT-6 Astra by discovering new sample-efficiency tricks?

The repo has the full rules, a list of ideas to try and how to submit; full results are in appendix B.

Future directions and beyond ARC

I believe a model trained from scratch can pass 85% on ARC-AGI-2, the bar for ARC Prize’s bonus prize, and 95% on ARC-AGI-1, with enough sample-efficiency tricks.

Nothing I’ve seen suggests a ceiling yet. The model is badly undertrained, its own candidates already contain the right answer 5–15 points more often than its top two picks, and none of the pieces is tuned. The open question is which sample-efficiency tricks will get us there (such as the best way to loop) and that’s what ARC Data Squeeze is for.

Universality: How general is the recipe?

I tried to keep the mdlARC recipe as general as possible. The obvious question is, where else could it work? Could a model trained from scratch beat LLMs on math? On parts of coding that don't need English? On ARC-3 or other interactive games? I haven't tested it outside ARC yet, but that's the plan next.

It's general because it's a plain autoregressive looped transformer: no vision backbone, no program search, no DSL, and very little ARC-specific stuff inside the model (mostly things to make training faster, cheaper or fit on Kaggle). It’s the same recipe that runs LLMs, so whatever makes it more sample efficient here should carry over.

The loop, for example, is just extra depth for every token, and other groups already see looping help language models reason (Geiping et al., 2025; Saunshi et al., 2025) and learn algorithms in context (Yang et al., 2023). Balanced augmentation order works for any training that uses augmentations. Hub-and-spoke attention is one of the Kaggle compromises: without the 4xL4 limit I’d use full bidirectional attention across the demos and the test input, and I expect it would score much higher.

Method and rationale

For the rest of the paper, I'll assume you know how ARC puzzles work. If not, click the drop down below.

ARC puzzles explained

The Abstraction and Reasoning Corpus (Chollet, 2019) is a test of abstract reasoning. Each task shows a few demonstration pairs: an input grid and the output grid it should become. All the pairs follow one hidden rule, and you must produce the exact output for a new test input in at most two attempts. Every task uses a different rule, so solving them means learning concepts like objects, counting and symmetry, and combining them into new ones. It’s fascinating to watch a transformer learn that from so little data. Try one:

Demo pairs: what is the rule?

Test input

→

Your answer?

Figure 6. Real ARC-AGI-1 evaluation tasks. Work out the rule from the demo pairs, guess the output for the test input, then reveal the answer. The 8.4M looped model, trained only on the 1,307 public tasks, solved every task shown here.

ARC-AGI-1 has 400 public training and 400 public evaluation tasks. ARC-AGI-2 (Chollet et al., 2025) is harder: 1,000 training and 120 public evaluation tasks, and the competition is scored on hidden ones, with 12 hours on four L4 GPUs and no internet. Most entries start from a pretrained LLM (Akyürek et al., 2024; Franzen et al., 2024; Sorokin and Puget, 2025); mdlARC starts from random weights and learns only from ARC grids. Its name comes from the minimum description length principle (MDL; Rissanen, 1978; Grünwald, 2007): the model that explains the data in the fewest bits, counting its own size, should generalise best.

Here are the main problems I had to solve, and what solved each one:

ProblemWhat solved itEvidence
An output cell has to find its input cell3D RoPE, with the grids stacked (positions)1D positions: 44% → ~25%
Pairs have no order, and the test pair is specialSeparate test planes + hub-and-spoke mask (positions, attention)–
Fine-tuning must fit in 12 hours on 4xL4sA small model; hub-and-spoke cost grows linearly with the demos (attention)–
Depth without more parametersLoop the middle 4 layers six times (loop)8.4M model: 47.44 → 66.25
The rule has to come from the demosDemos in context, no per-task embeddings (why)Embeddings halve the loop’s gain (+16.8 → +8.5)
Little dataAugmentations, balanced order; LLM-made puzzles on Kaggle (details)–
Slow decodingMulti-token prediction + speculative decodingSame answers, faster
Picking two answersVoting over augmented views + likelihood re-ranking+1.9 (B.8)

Training and inference overview

Like LLMs, mdlARC is an autoregressive transformer. The demo pairs and the test input grid are fed in as context, and the model writes the answer (the test output grid) one cell at a time, choosing the grid’s size itself by deciding when to stop.

Training uses teacher forcing, and the loss covers only the output grid being predicted. Any pair whose answer we know can play the test pair: in each epoch, every solved pair takes one turn as the target, with a random subset (at least half) of the task’s other solved pairs as its demos. At inference, each view leaves out a different demo.

Figure 7. Training and inference on ARC-AGI-1 task d037b0a7 (every coloured cell drips to the bottom). It plays by itself; the tabs and arrows step through it. The votes in step 3 are made up for the example.
Why demos in context, not per-task embeddings?

My first two versions gave every task its own learned embedding, added to every token, and showed the model only the test input (Vakde, 2025, Vakde, 2026). That reached 44% in March, but the rule lived in a lookup table: without the embedding, the March model fell to 24.8%. This version puts the demos in the context and has no per-task parameters, for three reasons:

  • A new task needs no new parameters. The model has to read the rule from the demos every time, which is what ARC asks for. A table grows with every training puzzle (TRM’s holds about 430M values), and it counts towards the model’s size.
  • With the demos in context, the embeddings add nothing. At 8.4M without a loop: 48.13 with them, 47.44 without.
  • They fight the loop. They halve its gain, from +16.8 to +8.5 points (evidence).

Other details:

For ARC Data Squeeze and small scale tests, we use a single training stage: every model trains from scratch only on the 1,307 public tasks, including the evaluation tasks’ demos but never their answers. This is a transductive approach followed by HRM (Wang et al., 2025), TRM (Jolicoeur-Martineau, 2025) and my earlier versions (Vakde, 2025; Vakde, 2026). We use 80 inference augmentations. The record run also re-ranks the top 20 voted answers by the model’s own likelihood, which added 1.9 points (appendix B.8).

It is much easier to have this single phase of training (lesser hparams, tuning, less brittle, etc). But we are forced to do 2 stage for Kaggle since we aim for highest score and need to train on much larger data as training from init on 4xL4s in 12hrs is impossible.

Converting a puzzle into a sequence of tokens

I create a separate sequence for each test pair in a puzzle. Each sequence ends with that test pair, and is prepended with the demo pairs, in any order (the model can’t tell which comes first).

Each grid is flattened row by row. The grid cells become a color token (0-9), and we have 4 special tokens: <start>, <next_line>, <input_output_separator> and <end_pair>.

One demo pairinputoutput…becomes these tokens, read left to rightS↵|↵Einput rowsoutput rowsS start (once, at the very beginning) · ↵ new row · | input ends, output begins · E end of pairA whole task is one sequence: the demo pairs, then the test input. The model writes the answer.demo 1demo 2demo 3test inputtest outputone cell at a time; loss only hereread all at once (bidirectional)Each token also gets a 3D position: its row, its column, and which grid it belongs to.Nothing records which demo comes first, so the demos are an unordered set.

Figure 8. How a task becomes a sequence. Colours are tokens; four special tokens mark the structure.

Positioning the tokens, grids and pairs

Transformers are equivariant to permuting token positions. To differentiate tokens in a sequence, we usually break the symmetry using position encodings and attention masks. mdlARC uses both.

To describe a token's position in a puzzle, we need at least 4 variables. Obviously you can decompose the dimensions differently (like different bases for the same vector space). Here's one example:

V1/V2 put one pair in each sequence, so 3D RoPE (Su et al., 2021) was enough: the input and output grids sit on top of each other in 3D, which lines each output cell up with its input cell, so the model doesn’t have to count tokens to find it. (Switching to ordinary 1D positions in March dropped the model from 44% to about 25% (Vakde, 2026).) V3 puts several pairs in one sequence, so it needs the 4th variable too.

Unfortunately 4D RoPE doesn't work because the last dimension is complicated:

To solve these problems, I use 2 tricks: I split the 4th variable into 2, and use the attention mask to distinguish pairs without ordering.

Every token gets a 3D position: (row, column, grid)grid1234(row, col, grid)(1, 2, 1)demo inputs (spokes)every demo shares this plane;the mask keeps them apart(1, 2, 2)demo outputs (spokes)every demo shares this plane(1, 2, 3)test input (hub)one plane, seen by every demo?(1, 2, 4)test outputbeing written001122columnrowNew-row, separator and end tokens take the column after their row's last cell; the start token is (0, 0, 0).

Figure 9. Every token’s position is (row, column, grid), and rotary embeddings act on all three. A cell in an output grid lines up with the same cell in its input. The test input has a plane of its own, so it sits in the same place for every demo: the hub. All the demos share the two demo planes, the spokes, and the attention mask decides which demo a token can see (next figure).

Put together, every demo sees the same picture: its own input and output on the two demo planes, as if it were the only demo, and the test input on its own plane, at the same relative position for every demo. The demos replace each other on the shared planes, while the test input never moves. That's the shape of hub-and-spoke attention: the test input is the hub, and the demos are the spokes.

This is a cool (but anti bitter lesson) trick. It kills 2 birds with one stone, because the same mask also cuts the cost of quadratic attention dramatically

Hub-and-spoke attention

The mask has to solve two problems:

A generic sparse pattern, like a sliding window, would cut through a task: a demo output could lose sight of its own input. Hub-and-spoke keeps every pair whole and linked to the test, and drops only the demo-to-demo links.

Full attentionHub-and-spoke attentionD1D1D2D2D3D3ininoutoutD1D1D2D2D3D3ininoutoutcost grows with (number of demos)²cost grows linearly with the number of demosdemos see each other at identical positionsa demo sees only its own pair and the test inputattendstest pair to all demos: a second softmax, gated per headRows attend to columns. D = demo pair, in/out = test input and output (output is causal).

Figure 10. Who attends to whom, for three demo pairs. Left, full attention: each demo also sees the other demos, whose cells sit at the same positions as its own. Right, hub-and-spoke attention: each demo sees only its own pair and the test input (blue), so the mask tells it which cells are its own. The test pair reads all the demos through a second, separately normalised attention (orange), mixed in by a learned gate in each head.

I use a custom sparse attention variant (Figure 10, right):

This custom attention variant provides a couple more advantages: Bidirectional attention is stronger than causal (the test output is causal though). And because the test tokens read the demos through their separate softmax, the demos can't crowd out the test pair, however many there are.

The positioning and custom attention stay the same in all the blocks. The last major piece is how the layers are stacked

Loop the layers

Out of the 8 transformer layers, I loop only the middle 4 layers six times. That gives 28 logical layers of depth, from just 8 blocks of weights.

demos +test input +output so farB1B2B3B4B5B6B7B8headnext cell× 6preludecodapasses 1–5 also exit(training only, weight 0.3)token embeddings reach every block through a learned skip; the prelude output is re-added at each pass28 block passes from 8 blocks: 2 + 4 × 6 + 2

Figure 11. Looped mdlARC. Blocks 3–6 run six times, so 8 blocks of weights give 28 block passes. Dashed: during training only, passes 1–5 also exit early and are scored. Green: the input reaches every block directly.

I experimented with multiple different looping strategies -- loop all 8 layers, early layers only, late layers only, middle + late, early + middle, multi loops, etc. Middle layer looping seemed to perform the best.

I used 2 additional tricks in the loops:

In short training runs, deep supervision improved the performance of undertrained models, so I used it in the final Kaggle submission (which was severely undertrained). There's no clear evidence if it helps in the trained / overtrained regime (perf is within seed noise, but not many sweeps). Input injection was part of the loop design from the start. I didn't remove it because the models clearly utilise it -- in almost every run its scale grows from 0 to 0.2-2.6 (obv requires proper investigation, will continue in ARC Data Squeeze).

Input injection is separate from the skip connection that brings the token embeddings into every block, which mdlARC already had before the loop. The skip connections were taken from the nanoGPT speedrun, and they always improved perf.

Deep supervision and input injection, with the loop unrolledinputB1–B2the loop, unrolled: blocks 3–6 run six timesinput injection: the output of B1–B2 is added back before passes 2–6pass 1B3–B6+B7–B8+ headloss 1pass 2B3–B6+B7–B8+ headloss 2pass 3B3–B6+B7–B8+ headloss 3pass 4B3–B6+B7–B8+ headloss 4pass 5B3–B6+B7–B8+ headloss 5pass 6B3–B6B7–B8+ headloss 6training only: 0.3 × their mean, so 0.06 eachweight 1the test answerAll six exits share the same blocks 7–8 and head. Circle area shows each loss’s weight.

Figure 12. Deep supervision and input injection. During training, the states after loop passes 1–5 also go through blocks 7–8 and the head and are scored; their mean loss is added with weight 0.3, and only pass 6 is used at test time. Before each of passes 2–6, the output of blocks 1–2 (green) is normalised, scaled by a learned number that starts at 0, and added back. At full training length, neither changes the score beyond seed noise (appendix B.4).

The full recipe behind the 37% submission (its data, pretraining, fine-tuning inside Kaggle’s 12 hours, and answering) is in appendix A.

Why does looping help?

What the evidence says

A possible explanation: a bias toward step-by-step rules

At equal depth, both models take the same number of steps, but the looped one must use the same step every time. Sharing weights across depth is like a convolution sharing weights across space: a constraint that pays off when it matches the data. Many ARC rules are step-by-step procedures, like extending a line until it hits something or filling a region, so a model that must repeat one step should find these rules from fewer examples, even with fewer parameters. Each looped weight is also trained by all six passes.

How to test it: compression

mdlARC is named after the minimum description length principle: the best model is the one that gives the shortest description of the data, counting the model itself (Rissanen, 1978; Grünwald, 2007). For a neural network, the useful version is the prequential code (Blier and Ollivier, 2018). You never send the trained weights. You send the architecture, the training recipe and a random seed, which cost a few bytes whatever the model’s size. Then you send the data one example at a time, each coded by the model trained on everything before it. The total length measures how fast the model learns.

On this view, fewer parameters don’t help by themselves. Bigger models usually learn faster, which fits my bigger models, looped or not, scoring higher (appendix B.3 and B.5). What should help is the right bias: a looped model that finds each rule from fewer examples gives a shorter code.

I haven’t measured this yet, and training loss can’t show it: at every depth the unlooped model fits its training data faster, yet scores lower on new tasks. The real test is a prequential code: train each model on a growing share of the tasks, and score each new share before training on it.

It is NOT refining the answer

HRM/TRM loop over the whole answer: they keep a draft of the full output grid and rewrite it again and again. ARC Prize’s analysis of HRM found that this outer refinement loop drove much of its gain (ARC Prize, 2025). My model never goes back to a cell it has written. Its loop is extra depth for each token: the same four blocks run six times before the next cell is written. It still adds 15–19 points, so on ARC a large gain from looping doesn’t need answer refinement.

These explanations are hypotheses. Like most things in deep learning, the science isn’t really clear, so we can't be sure of anything other than "divine benevolence"

Open questions

Prior work

Looping builds on a long line of work. Weight sharing across depth goes back to Universal Transformers (Dehghani et al., 2018) and ALBERT (Lan et al., 2019). The closest design is the recurrent-depth model of Geiping et al., 2025: a prelude, a looped middle block and a coda, with the input fed back into every pass. Encode–Think–Decode (Koishekenov et al., 2025) and retrofitted recurrence (McLeish et al., 2025) loop the middle layers of pretrained LLMs, and looped models can match much deeper ones on reasoning tasks (Saunshi et al., 2025). On ARC, HRM (Wang et al., 2025) and TRM (Jolicoeur-Martineau, 2025) refine a draft of the whole answer, LoopViT (Shu et al., 2026) and Loop-OWM (Gao et al., 2026) loop vision models, and the next-best LLM-free Kaggle entry this year was also a looped transformer.

Hub-and-spoke attention is built from known pieces. Fusion-in-Decoder (Izacard and Grave, 2021) encodes a question together with each retrieved passage, separately from the other passages, and its decoder reads them all: the same hub and spokes, with the question copied once per passage. Star Attention (Acharya et al., 2024) does something similar for long contexts, with the first block of the context as the hub. For in-context learning, Structured Prompting (Hao et al., 2022), Parallel Context Windows (Ratner et al., 2023) and FiD-ICL (Ye et al., 2023) encode the demos separately and let the query read all of them, but the demos never see the query. Mixing two separately normalised attentions with a learned gate goes back to Memorizing Transformers (Wu et al., 2022), whose gate is one learned number per head; here it depends on the token. What I haven’t seen elsewhere is the combination: one shared hub inside a single sequence, at the same position relative to every demo; a second, gated softmax that lets the test pair read the demos without them crowding out its own tokens; and training from scratch with this mask. ReHub (Borreda et al., 2025) uses the same hub-and-spoke picture in a graph transformer: a few virtual hub nodes relay information between the graph’s nodes. Here the hub is not an added node: it is the test input itself.

Code, data and weights

Appendix

Show the appendix: how the 37% model was built, full ARC Data Squeeze results and model details

Scores are top-2 / oracle on the public evaluation sets; a single number means no oracle was logged. One seed per row unless stated.

A. How the 37% submission was built

A.1 Overview

Data. The Kaggle model was pretrained on 161,798 puzzles in 3,826 families. They start from the 1,120 public ARC-AGI-2 tasks. Frontier LLMs, working in a coding agent, then wrote two kinds of generator programs, each checked automatically and none reviewed by a person:

  • Variations. The LLM summarises a task's rule and writes a program that makes new puzzles with the same rule, about 42 per task. This is an upgraded version of RE-ARC (Hodel, 2024).
  • Compositions. Following BARC (Li et al., 2024) and NVARC (Sorokin and Puget, 2025), the LLM combines the rules of two ARC-AGI-2 tasks into a new task and writes its generator.

The model learns only from the grids these programs output. It never sees the programs or any text. I couldn't afford to test whether the variations helped.

Pretraining. One run of 440 epochs, 14.1 hours on 8 B300 GPUs (about $600). Each epoch shows one puzzle from every family. Every solved pair becomes a target, with a random subset of the other pairs as context. Every sequence is randomly rotated or reflected, and its colours are permuted. Variations, demo subsets and augmentations are cycled in a shuffled, balanced order, which beat independent random sampling in earlier mdlARC runs. It extends Jordan’s alternating flip (Jordan, 2024), which balances a single yes-or-no flip, to augmentations with many values (A.3). The optimiser is NorMuon (Li et al., 2025) for the weight matrices and AdamW for the rest.

Test-time fine-tuning. On Kaggle, the whole model is fine-tuned for 180 epochs on the demo pairs of the hidden tasks (Sun et al., 2020; Akyürek et al., 2024), split across the four L4s. A time guard shortens fine-tuning if it would run past hour 10.

Answering. Each task is answered 40 times, under 40 different rotations, reflections and colour permutations, each time leaving one demo out of the context. The answers are mapped back to the original orientation, identical grids are counted, and the two most common become the two attempts. This voting scheme is standard on ARC (Franzen et al., 2024). Three extra output heads, trained to predict 2–4 tokens ahead (Gloeckle et al., 2024), draft tokens that the model then checks. This speculative decoding (Stern et al., 2018; Leviathan et al., 2023) only changes speed, never answers.

Surviving the rerun. Kaggle reruns your notebook on hidden tasks and returns no logs. So the notebook first writes a valid placeholder answer file, every stage catches its own errors, and failures are encoded in the size of the output file. That is how I learned that one run got a 0.00 because fine-tuning hung on all GPUs.

A.2 Pretraining data

PartFamiliesInstances
Public ARC-AGI-2 tasks (1,000 training + 120 evaluation, with answers), each with its LLM-written variations1,12048,160
LLM-written compositions of two ARC-AGI-2 tasks (1,988 eval–eval, 300 train–train, 418 mixed parents)2,706113,638
Total3,826161,798

No ConceptARC, RE-ARC or NVARC data, and no text or code. Generators were written mainly by GPT Sol-class models in Codex CLI; each passed automated checks and reproduces its puzzles exactly when re-run. No puzzle was reviewed by a human. Whether the ~42 variations per task helped is untested.

The generators, the seeds behind every puzzle, and the prompts and code of the pipeline that wrote them are public in the repo’s data_generation/ folder.

A.3 Pretraining

Data per epoch.

  • An epoch shows one instance from every puzzle family. A family is an original task, or a composition, with about 42 generated variations.
  • Every solved pair in that instance becomes a training target. Its context is a random subset of the other pairs: at least half, at most 4.
  • Variations, demo subsets and augmentations are cycled in shuffled order rather than drawn independently, like random reshuffling of the training data (Mishchenko et al., 2020). Each puzzle gets each of the 8 rotations and reflections once every 8 visits, and every 8 consecutive sequences also use each of them once. Colour permutations follow a Latin square: within each block of visits, every colour is sent to every allowed colour once. Demo counts and demo subsets are also cycled, so each option comes up equally often. Jordan’s alternating flip (Jordan, 2024) does this for a single yes-or-no flip. Earlier ARC work already gives each puzzle all 8 rotations and reflections (Akyürek et al., 2024; Franzen et al., 2024); I haven’t found earlier work that also balances them across consecutive sequences, or balances colour permutations. In earlier mdlARC experiments this balanced order beat independent random sampling.
  • Each family has about 43 versions, so 440 epochs is roughly 10 passes over the generated data.
  • Each sequence is augmented with one of 80 transformations: one of 8 rotations or reflections, plus a permutation of the colours that appear in the context.

Optimizer.

NorMuon (Li et al., 2025)AdamW
Applied toLinear weight matrices inside the blocksEmbeddings, output heads (including MTP), biases, norms, scalar gates
Peak learning rate0.006930.00125
OtherMomentum 0.95 (warmed up from 0.85), β2 0.95
  • Weight decay 0.1 on the feed-forward weights and the other matrices, 0.01 on the attention weights and token embeddings; one-dimensional parameters get none.
  • Gradient clipping 1.0. Seed 42.

Schedule. Warmup–stable–decay: linear warmup over the first 0.5% of epochs, then a constant learning rate, then linear decay over the last 20% of epochs to 10⁻⁴ × peak.

Two phases.

PhaseEpochsStartPeak LR (NorMuon / AdamW)DecayTime
10 → 270Random init0.00693 / 0.00125Epochs 216 → 2708.7 h
2270 → 440Phase-1 weights, optimizer state kept0.005544 / 0.001 (0.8×), re-warmed over 2 epochsEpochs 352 → 4405.4 h

Phase 1 decayed its learning rate to near zero before phase 2 resumed, so phase 2 re-warms an already-converged model rather than continuing a single schedule. It reused the data build from phase 1.

Batch, hardware and cost.

  • Batch 160 (8 GPUs × 20 sequences), no gradient accumulation.
  • 8×B300 with FlashAttention 4, about 1.9 minutes per epoch.
  • 14.1 hours in total, about 113 B300-hours (~$600).
  • A checkpoint was kept every 50 epochs, plus epochs 352 and 440. The final checkpoint is 910 MB.

Due to lack of time to test them for accuracy, I didn't optimise the GPU kernels before running. In hindsight, this was stupid. I could have saved about $300.

A.4 Test-time fine-tuning

  • Starts from the epoch-440 checkpoint. All weights are trained, with a fresh optimizer and the same schedule shape: decay starts at epoch 144 of 180.
  • Every demo pair of every hidden task is a target, with up to 3 of the task's other demos as context. The cap is 3 because the worst-case sequence (7,444 tokens) needs 20.68 GiB per L4 under data-parallel training; after a CUDA out-of-memory error the cap drops to 2.
  • 180 epochs, 80 augmentations. NorMuon 0.0008 / AdamW 0.00014, with weight decay and dropout as in pretraining.
  • One sequence per step, gradient accumulation 10 on each of the 4 GPUs: effective batch 40. One model is fine-tuned on all hidden tasks, with data-parallel training across the 4 GPUs.
  • Time guard. Fine-tuning must finish by hour 10 of the notebook. If measured epoch times predict an overrun, the epoch count is cut to fit 90% of the remaining time. A run already under way may overrun by up to 0.3 h; decoding absorbs it. There are up to 4 attempts.
  • How the recipe was chosen. It was tuned by pretraining on a custom data subset for clean evaluation -- trained without the val40 tasks, scoring on val40 and val120 (appendix A.7). The number of optimizer updates mattered more than the number of epochs, and repeats of the same setup varied by several points (14 fine-tunes of one checkpoint: mean 23.4, standard deviation 3.7 top-2). Full fine-tuning beat LoRA and freezing on val40: 32.50 vs 20.83–22.50 (LoRA) and 12.50 (frozen), top-2. 180 epochs was a judgement call: on heavy100 (the d768 e330 model), 90 epochs scored 72.67 / 78.67 and 180 scored 71.33 / 76.33, within that noise.

A.5 Decoding

  • Work split. The tasks are split into four groups balanced by estimated fine-tuning and decoding cost (largest task first, into the lightest group). Each GPU decodes its own group, with its own augmentation seed (42, 1, 2, 3).
  • Two passes. A quick pass with 8 views, as one shard, so every task has an answer early. Then a full pass with 40 views, in 4 shards per GPU, if its estimate (quick-pass time × 40/8 × 1.3) fits before hour 11.3. The full pass's votes replace the quick pass's; a crashed shard loses only its own tasks.
  • Each view uses one augmentation and a leave-one-out context: n − 1 of the task's n demos, up to 6. Decoding is greedy, with batch 16 and a prefill budget of 32,768 tokens; the multi-token heads draft tokens and the model verifies them.
    • 6 demo pairs was a mistake and likely reduced the score of my final submission. The final model was pretrained with a cap of 4 demo pairs, and fine-tuned with a cap of 3.
  • Voting. Undo each view's augmentation, count identical grids, discard any that copy the test input, and submit the two most common as the two attempts.

A.6 Fitting in 12 hours on 4×L4

  • A valid placeholder submission.json (each test input copied back as its answer) is written first, and every stage catches its own errors, so a failure leaves a valid file instead of an error.
  • FlashAttention is installed from an offline wheel.
  • Reruns on the hidden set return no logs, so failures are encoded in the size of the output file.
  • On the public file, the whole run was projected to take about 4.3 hours.
  • One of the three submissions that fine-tuned with data-parallel training scored 0.00. The size code showed the fine-tune had failed on all GPUs, most likely a hang in GPU-to-GPU communication on the rerun machines. The other two scored, including the 37.08.

A.7 Held-out sets for the Kaggle model

  • val40: 40 ARC-AGI-2 evaluation tasks, held out to tune fine-tuning. A separate development model was pretrained without them: its data (2,546 families, 108,008 puzzles) leaves out these 40 tasks, every composition built from them and any exact copies, so its scores on them are clean.
  • val120: val40 plus 80 held-out compositions.
  • new80: 80 new compositions of ARC-AGI-2 evaluation tasks, made after the training package was frozen. Their parent tasks are in training, so new80 tests new combinations, not new tasks. The submitted checkpoint scores 22.50 / 30.00 (fine-tuning with up to 4 context demos instead of 3).

B. ARC Data Squeeze: rules and full results

B.1 Rules in brief

ARC Data Squeeze follows the NanoGPT slowrun (NanoGPT slowrun, 2026), which fixes the data and lets compute vary, but it scores what the model can solve rather than its loss.

  • Fixed data. Train only on what the benchmark's prepare_data.py builds: the 1,307 public tasks.
  • Answers stay hidden. You may train on the evaluation tasks' demos and use their test inputs, but never their test outputs.
  • From scratch. No LLMs or pretrained models anywhere in the pipeline. Using them as coding assistants is fine.
  • Both benchmarks. Train one model on the ARC-AGI-1 data and one on the ARC-AGI-2 data, with the same method.
  • Limited inductive biases. Allowed: the grid layout (positions, which grid is which, hub-and-spoke attention), whole-task rotations, reflections, colour swaps and demo order, and voting or ranking with the model's own likelihood. Not allowed: scaling, translation or cropping, hand-computed objects or counts, rule-based filters on candidate answers, DSLs and program search.
  • Records need public code at a commit and weights for one seed. Each time an entry passes 90% top-2 on ARC-AGI-1, data is removed: first ConceptARC, then everything but each benchmark's official tasks.

Full rules: github.com/mvakde/arc-data-squeeze.

B.2 The data

Human-made public tasks only, deduplicated by task ID (773 IDs are shared between ARC-AGI-1 and ARC-AGI-2; near-duplicates under different IDs were not checked). No generated data and no language models.

ARC-AGI-1 runsARC-AGI-2 runs
Training set of the evaluated benchmark400 ARC-AGI-1 training tasks (1,718 pairs)1,000 ARC-AGI-2 training tasks (4,308 pairs)
Other ARC set, not already included347 ARC-AGI-2 tasks (1,457 pairs)27 ARC-AGI-1 tasks (123 pairs)
ConceptARC (Moskvichev et al., 2023)160 tasks (907 pairs)160 tasks (907 pairs)
Evaluated tasks, demos only400 tasks (1,363 pairs; 419 test outputs withheld)120 tasks (359 pairs; 167 withheld)
Pairs per epoch5,4455,697

The Data Squeeze models train from scratch on this set for 330 epochs, with no generated data and no test-time fine-tuning, and answer each task with 80 views. The record run also re-ranks the top 20 voted candidates by the model's own likelihood.

Results are on ARC-AGI-1 public evaluation (400 tasks) unless noted. The demos of the evaluated tasks are in training, but their test outputs never are, as in mdlARC, HRM and TRM.This is transductive learning: the model may learn from the unlabelled parts of the test tasks. No run is ever tuned on the evaluation answers.

B.3 Looping vs depth (8.4M looped models, d256)

Depth (block passes)Looped, 8.4MUnlooped, same depthUnlooped size
8(no loop)47.441×
12×2: 55.8853.561.5×
16×3: 60.1355.752×
20 vs 24×4: 61.88 / 72.8856.253×
28 vs 32×6 + deep supervision: 66.25 / 77.2558.634×

The 8-, 12- and 16-block scores are means of two runs.

At a fixed size, looping adds 15–19 points. At every depth it beats unique layers with up to 4× fewer parameters, and in the last two rows with fewer passes too (83% and 88% of the FLOPs per answer).

The 32-block model is the fairest test: it is deeper, has 4× the parameters and uses more compute per answer, and the looped model still beats it by 7.6 points.

A caveat: I ran out of time to tune the baselines. No model here, looped or unlooped, was tuned on its own, and a tuned unlooped model may close some of the gap. I’ll check this in ARC Data Squeeze, where tuned baselines are welcome entries.

B.4 What the loop needs (d256 ×6)

VariantTop-2 / oracle
Full recipe: deep supervision 0.3 + input injection66.25 / 77.25
No deep supervision61.75 / 75.13
No prelude injection64.13 / 76.13
Plain loop: no deep supervision, no prelude injection62.00 / 75.38
No prelude injection and no token-embedding skip61.50 / 73.13
Deep-supervision weight 1.064.62 / 75.88
Dropout 0.268.50 / 76.13
  • Each row is a single run, so gaps of a few points between rows may be noise. Looping itself is far outside that: the unlooped model scores 47.44.
  • Deep supervision matters most in short training. At ×3, after 60 epochs (d512, 8 views), the loop scores 34.13 without it and 39.25 with it. After 330 epochs (d256), it scores 60.1 without it and 59.4 with it.
  • Loop count saturates at ×6: ×8 scores 66.5 / 75.25 at d256; at d384, ×8 scores 67.88 / 78.50 vs 68.38 / 79.62 for ×6; at 1M, ×11 scores 48.88 / 60.75 vs 49.00 / 60.62.
  • The loop learns a reusable step. Trained at 6 passes and run with 3–8 passes, d256 scores 57.6 / 63.5 / 66.0 / 66.25 / 66.0 / 65.25 (top-2).
  • Per-task embeddings halve the loop gain: +16.8 top-2 without them (47.44 → 64.25 at ×4 + deep supervision), +8.5 with them (48.13 → 56.63). Teacher-forced exact accuracy barely changes (42.18 vs 43.25), so the loss is in the consistency of votes across views.

B.5 Small models

ModelParametersARC-AGI-1
My March 2026 mdlARC, unlooped (Vakde, 2026)76.6M44
Unlooped, today's recipe75.6M53.75
TRM (reported; Jolicoeur-Martineau, 2025)7M network + ~430M per-puzzle embeddings44.6
Looped ×6, d80, 4 heads, FFN 2560.71M33.38 / 48.12
Looped ×6, d96, FFN 2560.90M41.25 / 55.50
Looped ×6, d96, FFN 2960.99M41.50 / 55.88
Looped ×6, d96, FFN 3201.05M49.00 / 60.62
Looped ×6, d1442.0M55.63 / 68.00
Looped ×6, d1282.1M51.25 / 65.25
Looped ×6, d2568.4M66.25 / 77.25
Looped ×6, d38418.9M68.38 / 79.62
Looped ×6, d384, + likelihood ranking (ARC Data Squeeze record)18.9M70.25 / 79.88
Looped ×6, d384, + scale augmentation, rule-based filters and likelihood ranking (B.8; Open table)18.9M71.88 / 81.50

Below about 1M parameters the scores fall quickly.

B.6 ARC-AGI-2 public evaluation (120 tasks, same protocol)

ModelParametersARC-AGI-2
Looped ×6, d384, + likelihood ranking (ARC Data Squeeze record)18.9M15.14 / 20.56
Looped ×6, d38418.9M13.89 / 20.56
Looped ×8, d2568.5M12.64 / 18.89
TRM (reported; Jolicoeur-Martineau, 2025)7M + embeddings7.8
HRM (as reported by TRM)27M + embeddings5.0

B.7 Beyond the rules: adding RE-ARC

With RE-ARC's generated examples of the ARC-AGI-1 training tasks (Hodel, 2024), the 18.9M model reaches 81.38 / 88.00 on ARC-AGI-1 (plain votes 78.75) and 20.14 / 32.08 on ARC-AGI-2. transductARC, the closest from-scratch work, reports 76 on ARC-AGI-1 with RE-ARC at about 75M parameters (68 without), and about 17 on ARC-AGI-2 (Schweighofer, 2026). The RE-ARC runs also use scale augmentation (on the original tasks only, never on RE-ARC’s examples), rule-based filters and likelihood ranking. The same model with those three and no RE-ARC scores 71.88 / 81.50 (B.8), so RE-ARC itself adds 9.5 points on ARC-AGI-1.

B.8 Tricks used only in some runs

Three tricks from transductARC that the Kaggle submission does not use:

  • Scale augmentation: each sequence is upscaled by an integer block factor before the rotation or reflection; decoding mixes scaled views into its 80.
  • Rule-based filters: checks that reject implausible candidate grids.
  • Likelihood ranking: the top 20 candidates are re-scored by the model's likelihood.

On the d384 ×6 model, likelihood ranking alone takes ARC-AGI-1 from 68.38 / 79.62 to 70.25 / 79.88 (the ARC Data Squeeze record). Scale views take it to 71.00 / 81.50, and all three to 71.88 / 81.50. ARC Data Squeeze allows only likelihood ranking, so the other two count for its Open table.

B.9 Selection is the bottleneck

Across runs the oracle is 5–15 points above top-2: the right answer is often generated but not chosen.

B.10 Compute per run

Training plus the full evaluation:

ModelARC-AGI-1Compute per run
mdlARC v1, Dec 2025 (Vakde, 2025)27.52.9 A100-hours, $1.80
mdlARC v2, Mar 2026 (Vakde, 2026)442 hours on an RTX 5090, $0.67
Looped mdlARC, 19M (the record)70.25 / 79.883.7 H200-hours, ~$13
Looped mdlARC, 1.05M49.00 / 60.623.8 A100-hours, including a 57-minute evaluation
TRM (Jolicoeur-Martineau, 2025)44.6 (reported)about 3 days on 4 H100s (reported)

The 100M Kaggle model is the exception: its offline pretraining took 14.1 hours on 8 B300 GPUs (about $600).

C. Model details

C.1 Sizes

ModelWidthHeadsFFNBlocks / passesParameters
Kaggle submission10241626888 / 2899.8M
Earlier looped Kaggle model7681230728 / 2875.6M
Data Squeeze models80–3844–6256–15368 / 280.71M–18.9M

Blocks 1–2 run once, blocks 3–6 run six times, blocks 7–8 run once: 28 block passes from 8 blocks of weights. Parameter counts are trainable parameters.

C.2 Tokens and positions

  • Grids are written row by row with 14 tokens: the 10 colours plus start, newline, separator and end.
  • The sequence opens with one start token. Each pair then reads input rows, separator, output rows, end. The demo pairs come first, then the test input. The model writes the test output token by token and finishes with end, so the output size is never fixed in advance.
  • Positions use 3D rotary embeddings (Su et al., 2021) over row, column and grid (demo input, demo output, test input, test output). No position records which demo comes first, so the demos form an unordered set.

C.3 The transformer block

  • Pre-norm blocks with RMSNorm, multi-head attention with 3D RoPE, and a SwiGLU feed-forward layer (Shazeer, 2020).
  • Each block also gets a learned skip connection from the token embedding, scaled by a scalar that starts at 0. The prelude's normalised output is re-added at the start of loop passes 2–6, scaled by one learned scalar that starts at 0.
  • Deep supervision (Lee et al., 2014): during training, the states after loop passes 1–5 also run through blocks 7–8 and the output head; the mean of their losses is added with weight 0.3. Only the final pass is used at test time.
  • The output head is separate from the input embedding (untied). There are no task embeddings.
  • Dropout 0.1 on the residual branches and the embeddings. Attention dropout is 0.1 in the Data Squeeze models and 0 in the Kaggle model, whose FlashAttention 4 kernel has no attention dropout.

C.4 Hub-and-spoke attention

  • Demo tokens attend only within their own pair and to the test input (the hub). No cross-demo attention matrix is ever built, so demos share information only through the hub.
  • Test-pair tokens use two attentions with separate softmaxes: one over the test pair itself (the test input bidirectionally, the output causally) and one over all the demos. A per-head gate, a sigmoid of a linear map of the hidden state, mixes them. It is initialised to an even mix.
  • Attention cost grows linearly with the number of demos instead of quadratically.
  • There is no ablation against full attention.

C.5 Multi-token prediction

  • Three extra linear heads on the final hidden state predict the tokens 2, 3 and 4 positions ahead, with loss weight 0.3, on output tokens only (Gloeckle et al., 2024).
  • During training they add a learning signal. During decoding they draft tokens for speculative decoding (Stern et al., 2018; Leviathan et al., 2023), which is exact: it changes speed, never answers.

C.6 Loop variants that did not help

Dense mixing across passes, iteration embeddings, gates, per-iteration norms and parallel streams were all neutral against the plain loop in controlled tests; pointer-style variants scored 63.0–65.6 vs 66.25; H-Net chunking with a ×6 loop scored 58.5 / 72.63 vs 66.25 / 77.25. A d768 model combining several variants scored higher than its plain counterpart, but that run changed other settings too, so their effect is not isolated.

C.7 Changing the loop count after training

The loop learns one step it can repeat. Take a trained model and change only how many times the loop runs:

Demo pairs

Test input

Correct answer

Figure 13. The same trained weights, run with the loop repeated 3 to 8 times (it was trained with 6). Each frame is a real, full answer to an ARC-AGI-1 evaluation task from the 8.4M model, showing its top-voted answer; wrong cells are outlined. More loops fix the first two tasks. The third gets worse past the trained depth. The small chart is the score on all 400 tasks.

Nothing else changes between these frames: same weights, same input. With too few loops the model gets the task wrong. With the six it was trained on, it gets it right.

References