A 100M transformer trained only on ARC puzzles scores 37.08% on ARC-AGI-2. Looping beats models 10× larger, and a 1M model beats my 76M one.

I trained a 100M parameter autoregressive transformer from scratch on ARC puzzles alone. No language, no code, no pretrained LLM: the model has only ever seen ARC grids. It scores 37.08% on ARC-AGI-2 on Kaggle¹. Across all 746 ARC Prize 2026 writeups, that’s the highest score without an LLM, about 3× the next best.

¹ This is the Kaggle public score of my final submission (57908007). It was submitted before the deadline but scored after the leaderboard froze, so the official board shows my previous looped model at 35.69% (rank 141). Some training puzzles came from LLM-written generator programs (see Data); the model itself never saw them.

This is an upgrade to the mdlARC work I published over the past year. The biggest change is looping the middle layers, which gives us extraordinary gains (better scores than 10x larger non-looped models).

The score can probably go much higher: I only started looping in the last week, so the implementation is rough, and the final 100M model submission was severely undertrained and unoptimised. The best looping strategy is still an open question, and it matters: frontier LLMs today are widely rumoured to loop.

How I got here: Six months ago, I scaled up mdlARC to 1B parameters, and it plateaued at ~33% on Kaggle (6th place). I came back with two weeks left; of everything I tried, looping the middle layers blew the rest away.

What’s new

• Looping is a huge win. At a fixed 8.4M parameters, looping the middle layers lifts ARC-AGI-1 from 47% to 62-66% (2 seeds). At the same depth, it beats a stack of unique layers with 4× more parameters.
• A 1M model beats my 76M one: The looped 1M model scores 49% on ARC-AGI-1, above my March 2026 mdlARC (44%), with 73× fewer parameters.
• No hidden parameters. TRM’s “7M” model is misleading. It also learns 430M of per-puzzle embeddings. My models are truly small: my 1M looped model, with no other parameters, beats it on ARC-AGI-1 (49% vs 44.6%)
  • TRM’s looping might actually be harming its score: It is easy for any autoregressive transformer trained on the same dataset to perform better
  • Looped mdlARC is much cheaper to train (2 H200 hours vs 3 4xH100 days)
  • Looped mdlARC is still autoregressive, so the justification for looping is MDL and not iterating the answer
• Best from official data alone. A 19M model reaches 72% on ARC-AGI-1 and 14% on ARC-AGI-2 (TRM: 45% and 8%). With ReARC it reaches 81%, above transductARC’s 76% with 4× fewer parameters.
• Hub-and-spoke attention. This makes cost linear in the number of demos, so fine-tuning fits on Kaggle’s 4×L4.

New Sample efficiency Benchmark.
To find out how to loop and test other sample-efficiency ideas, I am launching ARC Data Squeeze: how high can a model score, trained from scratch only on the ~1,300 public, human-made ARC tasks?

• It measures sample efficiency across puzzles, not just inside one.
• Can’t test data efficiency on Official ARC because it allows unlimited pretraining. LLMs post-trained on huge synthetic sets can score arbitrarily high, like the surge of scores at the end.
• Current SOTA: my 19M looped model, 72% on ARC-1 and 14% on ARC-2.
• About 2 H200-hours a run, so it is open to the GPU-poor

Results

Kaggle ARC-AGI-2 scores

For months, my non-looped models plateaued at 30–32%. Both looped models beat them on their first submission, with up to 10× fewer parameters. These entries also differ in data, fine-tuning and decoding, so the tests below isolate the loop.

ARC Data Squeeze (ARC-1 public eval, full tables in appendix)

Every model is trained from scratch, only on the ~1,300 public human-made tasks.

Why looping works

Hard to truly know why. “Divine benevolence” is the honest answer. Anything else is overclaiming. But if I were to guess, I’d say minimum description length (MDL): among models that fit the data, prefer the one that takes the fewest bits to describe (itself included) as they generalise better.

Intuitively, looping adds depth without increasing parameters or weights. Parameter count is our rough proxy for description length; we don’t measure code length directly. Some Evidence:

• Extra compute depth is not enough to reach At equal or lower compute depth, loops beat extra unique layers by 2.3 to 7.6.
• Extra description length undoes the gain. (alert: untuned baseline) Per-task embeddings halved the loop gain: Maybe a model that can look a rule up stops working it out. TRM carries 430M parameters of such embeddings.

Notes: The baselines aren’t tuned well. Further analysis will continue on the Data Squeeze Benchmark

Architecture

An autoregressive transformer reads the demo pairs and the test input bidirectionally, then writes the test output one cell at a time, causally. The loss covers only the output. With no task embeddings, every rule must come from the demos. Token and position details are in the appendix.

Middle layer loop. Blocks 1–2 run once, blocks 3–6 run six times, blocks 7–8 run once: 28 block passes from 8 blocks. A plain loop does most of the work, two details add a little:

• Deep supervision. In training, each of the first five passes also exits through blocks 7–8 and the head, with the mean exit loss weighted 0.3.
• Input re-injection. Every block receives the token embeddings through a learned skip, and the prelude’s output is added back at the start of each later pass.

Hub-and-spoke attention (HSA).

Demo pairs can reach 20K+ tokens (ten 30×30 pairs). This makes offline pretraining incredibly expensive and kaggle finetuning on 4xL4s nigh impossible.

In HSA, each demo pair attends only to itself and the test input. The test input (the hub) attends to its own tokens (the output causally) and, through a second, separately normalised attention, to all the demos; a learned per-head gate mixes the two. Attention cost grows linearly with the number of demos instead of quadratically

Training and decoding

Training has two stages
A randomly initialised model is pretrained offline for 440 epochs on 3,826 puzzle families: one YOLO run of 14.1 h on 8×B300. Inside Kaggle’s 12 hours on 4×L4, all its weights are then fine-tuned for 180 epochs on the demo pairs of the 240 hidden tasks.

Balanced Sampling. Each epoch shows one variation per family; every solved pair is a target, with half to 4 other pairs as context. Variations, demo subsets and the 80 augmentations (rotation or reflection, plus colour permutation) are cycled in a balanced shuffled order, which beat random sampling in earlier mdlARC runs.

Decoding. Each task is decoded greedily under 40 augmentations, and each view leaves one demo out of the context. Answers are mapped back, identical grids are counted, copies of the test input are dropped, and the two most common are submitted. Three extra heads, trained to predict 2–4 tokens ahead, draft tokens for speculative decoding. This is exact: it changes speed, never answers.

Data

Kaggle model: the 1,120 public ARC-2 tasks, including the 120 evaluation tasks with answers, plus 2,706 synthetic tasks. Frontier LLMs (mainly GPT Sol-class, in Codex CLI) wrote two kinds of generators, all machine-checked and none human-reviewed:

• Variations: the LLM summarises a task’s rule, writes a generator and reviews it, giving ~42 variations per task: an upgraded ReARC. Whether they helped is unclear; testing was too costly.
• Compositions: following BARC and NVARC, the LLM combines two ARC-2 tasks into a new rule and generator in one shot.

Conclusion

Looping is a cheap, general way to add depth. It drops into any decoder, adds 19 points on ARC-AGI-1 at a fixed size, and lets a 2M model beat a 76M one.

ARC Data Squeeze is open to everyone, at about 2 H200-hours a run.

Code: github.com/mvakde/mdlARC · ARC Data Squeeze: github.com/mvakde/arc-data-squeeze · Weights: kaggle.com/models/mithilavakde/mdlarc

Appendix

Scores are top-2 / oracle on the public evaluation sets; a single number means no oracle was logged.

A. Comparison with previous ARC methods

B. Full ARC Data Squeeze results

ARC-AGI-1 public evaluation (400 tasks) unless noted. Every model is trained from scratch on the Data Squeeze set (appendix E.2) for 330 epochs and decoded with 80 views. The demos of the evaluated tasks are in training, but their test outputs never are, as in mdlARC, HRM and TRM. One seed per row unless stated.

B.1 Looping vs depth (8.4M looped models, d256)

At a fixed size, looping adds 15–19 points. At every depth it beats unique layers with up to 4× fewer parameters, and in the last two rows with fewer passes too. No variant, looped or unlooped, was tuned separately.

B.2 What the loop needs (d256 ×6)

• The two seeds of the full recipe differ by 4.25 top-2 points, so every other gap in this table is within noise. Looping itself is not: the unlooped model scores 47.44.
• Loop count saturates at ×6: ×8 scores 66.5 / 75.25 at d256; at d384, ×8 scores 67.88 / 78.50 vs 68.38 / 79.62 for ×6; at 1M, ×11 scores 48.88 / 60.75 vs 49.00 / 60.62.
• The loop learns a reusable step. Trained at 6 passes and decoded with 3–8 passes, d256 scores 57.6 / 63.5 / 66.0 / 66.25 / 66.0 / 65.25 (top-2).
• Per-task embeddings halve the loop gain: +16.8 top-2 without them, +8.5 with them. Teacher-forced exact accuracy barely changes (42.18 vs 43.25), so the loss is in the consistency of votes across views.

B.3 Small models

Below about 1M parameters the scores fall quickly.

B.4 ARC-AGI-2 public evaluation (120 tasks, same protocol)

B.5 Beyond Data Squeeze: adding ReARC

With ReARC’s generated examples of the ARC-1 training tasks (Hodel, 2024), the 18.9M model reaches 81.38 / 88.00 on ARC-AGI-1 (plain votes 78.75) and 20.14% / 32.08% on ARC-AGI-2. transductARC, the closest from-scratch work, reports 76% on ARC-AGI-1 with ReARC at about 75M parameters (68% without), and ~17% on ARC-AGI-2. The ReARC runs also turned on scale augmentation, guards and likelihood ranking, so ReARC’s own effect is not isolated.

B.6 Selection is the bottleneck

Across runs the oracle is 9–14 points above top-2: the right answer is often generated but not chosen.

C. Model details

C.1 Sizes

Blocks 1–2 run once, blocks 3–6 run six times, blocks 7–8 run once: 28 block passes from 8 blocks of weights. Parameter counts are trainable parameters.

C.2 Tokens and positions

• Grids are written row by row with 14 tokens: the 10 colours plus start, newline, separator and end.
• Each pair reads start, input rows, separator, output rows, end. The sequence is the demo pairs followed by the test input. The model writes the test output token by token and finishes with end, so the output size is never fixed in advance.
• Positions use 3D rotary embeddings over row, column and role (demo input, demo output, test input, test output). No position records which demo comes first, so the demos form an unordered set.

C.3 The transformer block

• Pre-norm blocks with RMSNorm, multi-head attention with 3D RoPE, and a SwiGLU feed-forward layer.
• Each block also gets a learned skip connection from the token embedding, scaled by a scalar that starts at 0. The prelude’s normalised output is re-added at the start of loop passes 2–6, scaled by one learned scalar that starts at 0.
• Deep supervision: during training, the states after loop passes 1–5 also run through blocks 7–8 and the output head; the mean of their losses is added with weight 0.3. Only the final pass is used at test time.
• The output head is separate from the input embedding (untied). There are no task embeddings.
• Dropout 0.1 on the residual branches and the embeddings. Attention dropout is 0.1 in the Data Squeeze models and 0 in the Kaggle model, whose FlashAttention 4 kernel has no attention dropout.

C.4 Hub-and-spoke attention

• Demo tokens attend only within their own pair and to the test input (the hub). No cross-demo attention matrix is ever built, so demos share information only through the hub.
• Test-pair tokens use two attentions with separate softmaxes: one over the test pair itself (the test input bidirectionally, the output causally) and one over all the demos. A per-head gate, a sigmoid of a linear map of the hidden state, mixes them. It is initialised to an even mix.
• Attention cost grows linearly with the number of demos instead of quadratically.
• Intuition: a person solving an ARC task compares each demo with the test input; the demos are independent examples of one rule, so the model need not compare them with each other. There is no ablation against full attention.

C.5 Multi-token prediction

• Three extra linear heads on the final hidden state predict the tokens 2, 3 and 4 positions ahead, with loss weight 0.3, on output tokens only.
• During training they add a learning signal. During decoding they draft tokens for speculative decoding, which is exact: it changes speed, never answers.

C.6 Loop variants that did not help

Dense mixing across passes, iteration embeddings, gates, per-iteration norms and parallel streams were all neutral against the plain loop in controlled tests; pointer-style variants scored 63.0–65.6 vs 66.25; H-Net chunking with a ×6 loop scored 58.5 / 72.63 vs 66.25 / 77.25. A d768 model combining several variants scored higher than its plain counterpart, but that run changed other settings too, so their effect is not isolated.

D. Training and decoding details (Kaggle model)

D.1 Pretraining

Data per epoch.

• An epoch shows one instance from every puzzle family. A family is an original task, or a composition, with about 42 generated variations.
• Every solved pair in that instance becomes a training target. Its context is a random subset of the other pairs: at least half, at most 4.
• Variations, demo subsets and augmentations are cycled in shuffled order rather than drawn independently. In earlier mdlARC experiments this balanced order beat independent random sampling.
• Each family has about 43 versions, so 440 epochs is roughly 10 passes over the generated data.
• Each sequence is augmented with one of 80 transformations: one of 8 rotations or reflections, plus a permutation of the colours that appear in the context.

Optimizer.

• Weight decay 0.1 on the feed-forward weights and the other matrices, 0.01 on the attention weights and token embeddings; one-dimensional parameters get none.
• Gradient clipping 1.0. Seed 42.

Schedule. Warmup–stable–decay: linear warmup over the first 0.5% of epochs, then a constant learning rate, then linear decay over the last 20% of epochs to 10⁻⁴ × peak.

Two phases.

Phase 1 decayed its learning rate to near zero before phase 2 resumed, so phase 2 re-warms an already-converged model rather than continuing a single schedule. It reused the data build from phase 1.

Batch, hardware and cost.

• Batch 160 (8 GPUs × 20 sequences), no gradient accumulation.
• 8×B300 with FlashAttention 4, about 1.9 minutes per epoch.
• 14.1 hours in total, about 113 B300-hours (~$600).
• A checkpoint was kept every 50 epochs, plus epochs 352 and 440. The final checkpoint is 910 MB.

D.2 Test-time fine-tuning

• Starts from the epoch-440 checkpoint. All weights are trained, with a fresh optimizer and the same schedule shape: decay starts at epoch 144 of 180.
• Every demo pair of every hidden task is a target, with up to 3 of the task’s other demos as context. The cap is 3 because the worst-case sequence (7,444 tokens) needs 20.68 GiB per L4 under data-parallel training; after a CUDA out-of-memory error the cap drops to 2.
• 180 epochs, 80 augmentations. NorMuon 0.0008 / AdamW 0.00014, with weight decay and dropout as in pretraining.
• One sequence per step, gradient accumulation 10 on each of the 4 GPUs: effective batch 40. One model is fine-tuned on all hidden tasks, with data-parallel training across the 4 GPUs.
• Time guard. Fine-tuning must finish by hour 10 of the notebook. If measured epoch times predict an overrun, the epoch count is cut to fit 90% of the remaining time. A run already under way may overrun by up to 0.3 h; decoding absorbs it. There are up to 4 attempts.
• How the recipe was chosen. It was tuned with a development checkpoint trained without the val40 tasks, scoring on val40 and val120 (appendix E.3). The number of optimizer updates mattered more than the number of epochs, and repeats of the same setup varied by several points (14 fine-tunes of one checkpoint: mean 23.4, standard deviation 3.7 top-2). Full fine-tuning beat LoRA and freezing on val40: 32.50 vs 20.83–22.50 (LoRA) and 12.50 (frozen), top-2. 180 epochs was a judgement call: on heavy100 (the d768 e330 model), 90 epochs scored 72.67 / 78.67 and 180 scored 71.33 / 76.33, within that noise.

D.3 Decoding

• Work split. The tasks are split into four groups balanced by estimated fine-tuning and decoding cost (largest task first, into the lightest group). Each GPU decodes its own group, with its own augmentation seed (42, 1, 2, 3).
• Two passes. A quick pass with 8 views, as one shard, so every task has an answer early. Then a full pass with 40 views, in 4 shards per GPU, if its estimate (quick-pass time × 40/8 × 1.3) fits before hour 11.3. The full pass’s votes replace the quick pass’s; a crashed shard loses only its own tasks.
• Each view uses one augmentation and a leave-one-out context: n − 1 of the task’s n demos, up to 6. Decoding is greedy, with batch 16 and a prefill budget of 32,768 tokens; the multi-token heads draft tokens and the model verifies them.
• Voting. Undo each view’s augmentation, count identical grids, discard any that copy the test input, and submit the two most common as the two attempts.

D.4 Tricks used only in Data Squeeze runs

The best Data Squeeze runs add three tricks from transductARC that the Kaggle submission does not use:

• Scale augmentation: each sequence is upscaled by an integer block factor before the rotation or reflection; decoding mixes scaled views into its 80.
• Symbolic guards: rule-based checks that reject implausible candidate grids.
• Likelihood ranking: the top 20 candidates are re-scored by the model’s likelihood.

On the d384 ×6 model they take ARC-AGI-1 from 68.38 / 79.62 to 71.00 / 81.50 (scale views) and 71.88 / 81.50 (with guards and ranking).

D.5 Fitting in 12 hours on 4×L4

• A valid placeholder submission.json (each test input copied back as its answer) is written first, and every stage catches its own errors, so a failure leaves a valid file instead of an error.
• FlashAttention is installed from an offline wheel.
• Reruns on the hidden set return no logs, so failures are encoded in the size of the output file.
• On the public file, the whole run was projected to take about 4.3 hours.
• One of the three submissions that fine-tuned with data-parallel training scored 0.00. The size code showed the fine-tune had failed on all GPUs, most likely a hang in GPU-to-GPU communication on the rerun machines. The other two scored, including the 37.08.

E. Data details

E.1 Kaggle pretraining data

No ConceptARC, ReARC or NVARC data, and no text or code. Generators were written mainly by GPT Sol-class models in Codex CLI; each passed automated checks and reproduces its puzzles exactly when re-run. No puzzle was reviewed by a human. Whether the ~42 variations per task helped is untested.

E.2 ARC Data Squeeze set

Human-made public tasks only, deduplicated by task ID (773 IDs are shared between ARC-1 and ARC-2; near-duplicates under different IDs were not checked). No generated data and no language models.

E.3 Held-out sets for the Kaggle model

• val40: 40 ARC-2 evaluation tasks withheld, with their descendant compositions and exact matches, from a development package (2,546 families, 108,008 instances). Used to tune fine-tuning.
• val120: val40 plus 80 held-out compositions.
• new80: 80 new compositions of ARC-2 evaluation tasks, made after the training package was frozen. Their parent tasks are in training, so new80 tests new combinations, not new tasks. The submitted checkpoint scores 22.50 / 30.00 (fine-tuning with up to 4 context demos instead of 3).

References

• Akyürek et al. (2024). The Surprising Effectiveness of Test-Time Training for Abstract Reasoning.
• Chollet (2019). On the Measure of Intelligence.
• Chollet et al. (2025). ARC-AGI-2.
• Dehghani et al. (2018). Universal Transformers.
• Franzen, Disselhoff and Hartmann (2024). The LLM ARChitect.
• Geiping et al. (2025). Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach.
• Hodel (2024). Addressing the Abstraction and Reasoning Corpus via Procedural Example Generation (RE-ARC).
• Jolicoeur-Martineau (2025). Less is More: Recursive Reasoning with Tiny Networks (TRM).
• Li et al. (2024). Combining Induction and Transduction for Abstract Reasoning (BARC).
• Liao and Gu (2025). ARC-AGI Without Pretraining (CompressARC).
• Moskvichev, Odouard and Mitchell (2023). The ConceptARC Benchmark.
• NVARC (2025). ARC Prize 2025 solution.
• Schweighofer. transductARC. github.com/kschweig/transductARC
• Vakde (2025). New Pareto Frontier on ARC-AGI. mvakde.github.io/blog/new-pareto-frontier-arc-agi
• Vakde (2026). 44% on ARC-AGI-1 in 67 cents. mvakde.github.io/blog/44-on-arc-1
• Wang et al. (2025). Hierarchical Reasoning Model (HRM).