Nemotron reasoning benchmark, decomposed

The NVIDIA Nemotron Model Reasoning Challenge presents 863 puzzles in nine categories. Each gives 3 to 10 input/output examples generated by a hidden rule and asks for the output on a held-out input, scored by exact match. The strongest public pipeline treats the benchmark uniformly. This project does not.

Step 1: measure, do not assume

Re-expanding the prior pipeline's collapsed category buckets and counting per-category unsolved puzzles shows the difficulty is not spread out at all. 47 of the 66 remaining unsolved puzzles sit in cryptarithm_deduce, which is 77% of that category and most of the leaderboard headroom. Five categories are already closed by deterministic solvers.

Step 2: split the hard category by mechanism

Every cryptarithm_deduce instance is a five-character string with an operator in the middle, and the hidden rule branches on the operator. Empirically there are two modes:

• Mode A, positional permutation. For most * and + cases the output is a permutation (with repeats and drops) of the four non-operator input characters. Solvable by exhaustive search over a tiny positional DSL.
• Mode B, substitution plus arithmetic. For -, /, and non-standard operators the output characters do not appear in the input at all, so a positional DSL provably cannot produce them. The rule is a symbol-to-digit bijection, arithmetic in an unstated base, and the inverse bijection.

This explains mechanically why a uniform permutation-style solver leaves 77% of the category unsolved: it only fits Mode A.

Step 3: one solver per mode

The Mode A solver enumerates output patterns grouped by operator, keeps the patterns consistent with every example, and answers only when all survivors agree. It scores 7/7 on the gold-labeled validation slice and abstains with a structured reason otherwise, which cleanly identifies Mode B puzzles for routing.

The Mode B attempt is a version-space program-synthesis engine over three atoms (pick a position, emit a constant, apply a learned bijection to a position), backed by Z3. It produces 30 raw predictions on the 47 unknown puzzles, tiered by how many examples constrain the bijection. Only the 4 medium-trust predictions were admitted to the training corpus; the 26 low-trust ones are documented and excluded.

Negative results, reported as such

• Three frontier reasoning models (DeepSeek R1, GPT-5.5, Claude Opus 4.8) were used as teachers on Mode B with a verification-first prompt. They failed in three different ways and produced zero usable traces. One silently corrupted the puzzle text and answered confidently; one honestly said "uncertain".
• A depth-3 functional DSL on the residual bit_manipulation puzzles: 0/8.
• The operator-independent fallback on the two *_guess categories: 0 safe cracks, supporting the conclusion that those need operator extrapolation no rule miner can do without a transfer prior.

Fine-tuning track

Teacher-student SFT with cross-model agreement filtering, Hyperband hyperparameter search, multi-seed runs, and LoRA adapter-soup merging, in a set of reproducible notebooks. A 419-row augmented SFT corpus is released with the repository.