OpenWeights · research · 2026-09-07

What the reruns changed · a second pass over the window matrix

After the window matrix was published, every cell was checked for a failed or noisy run, and three things were rerun the same day.

Purpose

Settle the three cells the first pass left in doubt: the 2.6B's grades, which its 640-token cap had cut short; the Poco's Qwen3 32k run, which had died without a recorded cause; and whether a second run of the same file on another phone varies as much as it did on the Poco.

How it was graded

Same prompts and graders as the matrix: GSM8K by the last number in the reply, IFEval by the benchmark's instruction checkers, BFCL by one call to the right function with the right arguments. Greedy, thinking off. Every run also recorded prefill and decode time per prompt and resident memory, shown in the timing panel and table. The 2.6B rerun raised only the reply cap, from 640 to 2048, on its 60 GSM8K and IFEval prompts; the repeats used all 90.

How long it ran

RunWhereWall clock
Qwen3 32k with a logPoco2 min, killed
2.6B 2k repeat, 90 promptsPoco48 min
2.6B 2k at 2048, 60 promptsPoco80 min
2.6B 32k at 2048, 60 promptsPoco66 min
2.6B 4k and 32k at 2048, 1.2B 4k repeatTest Lab, 3 phones22 launches, 12:40 to 18:51

The Test Lab span includes a pause from 13:19 to 15:05 for the daily quota. Generation time alone: 2.6B at 2048, 130 to 213 minutes per phone.

The 2.6B

The 640-token cap suppressed its grades

GSM8K 10 to 19 of 30 and IFEval 3 to 6 at the 640 cap; 23 to 28 and 18 to 21 at 2048, on every phone and window.

The window

At 2k, 14 of 60 replies reached the window limit

Prompt plus reply stopped at the window size. At 4k and 32k no reply reached the window; the ones still cut reached the 2048 reply cap.

Same file, twice

One cloud repeat, the same low agreement

The Exynos 2400 repeat of the 1.2B agreed with its first run on 7 of 90 raw replies, at least as variable as the Poco controls. The 2.6B repeat moved grades by 2 per set.

Qwen3 32k

Both memory guards report a 6 GB threshold

MIUI stopped the Poco run at 8.65 GB against a 6 GB threshold, as Samsung's Heimdall stopped the two Galaxys. The Pixel, with no such guard, finished at 8.5 GB.

The 2.6B under two reply caps

Prompts passed out of 30, per phone and window. The hollow marker is the 640-token cap the suite uses, the solid one the 2048-token rerun. BFCL was not rerun; its 640-cap grades and calls stay in the original rows.

The chart scrolls sideways on a narrow screen.

Where the long replies were cut

The 60 GSM8K and IFEval replies of each 2048-cap cell: finished, cut by the window (prompt plus reply reached the window size), or cut by the reply cap.

The chart scrolls sideways on a narrow screen.

FinishedCut by the windowCut by the 2048 cap

Reply length at 2k and at 32k

Dimensity 9400, 2048 cap, 60 replies per window, both panels on one scale. Each bar is a 128-token bin of prompt plus reply. At 2k, 14 replies end at the window limit of 2048 tokens; at 32k, 10 of 60 reached the reply cap and the rest finished earlier.

The chart scrolls sideways on a narrow screen.

2048 tokens, the 2k window

Same file, run twice

Five controls: the same export, same prompts, same phone, a second time. How many replies came back byte-identical (raw stream and shown text over 90 prompts, parsed tool calls over the 30 BFCL prompts), and how far each grade moved.

The chart scrolls sideways on a narrow screen.

Raw streamShown textTool calls

Qwen3-1.7B at 32k against the 6 GB rule

Resident memory (RSS) of the test process right after the 32k export loaded, on each phone, with the phone's RAM. The dashed line is the 6 GB threshold the killed runs' logs reported; that threshold is measured in PSS at kill time, a different measure from the bars, so the bars show which phones were near it, not the moment they crossed it. Three of four processes were stopped within the first dozen prompts.

The chart scrolls sideways on a narrow screen.

6 GB, the PSS threshold Samsung Heimdall and MIUI reported when they stopped the processThe phone's RAM

Time to first token and time per output token

Every rerun recorded prefill and decode time per prompt. TTFT here is the runtime's prefill of the whole prompt, the engine's part of the wait before the first token; TPOT is decode time over generated tokens. Each row pairs a rerun with the earlier run of the same cell: the hollow marker is the earlier run, the solid one the rerun. Medians over the prompts each run completed, so they are per-cell figures, not a paired speed test.

What to take from it: the capped rerun's lower TTFT is a prompt-mix effect, not a speed change. Its 60 GSM8K and IFEval prompts are shorter than the 30 BFCL prompts with their tool definitions, which the 640-cap rows include. TPOT, which does not depend on prompt length, agrees within 5% on the Dimensity 9400 and Exynos 2400, differs by 8 to 11% on the 8 Elite and on the Tensor G5 at 4k, and by 26% on the Tensor G5 at 32k (81 then 102 ms), the size of the launch-to-launch swing seen on that phone before. Of the five repeats, three agree within 4% on both figures; the 1.2B repeats at 32k on the Dimensity and at 4k on the Exynos were 7 to 12% slower the second time.

The chart scrolls sideways on a narrow screen.

Why the 2.6B needed a rerun. Its chat template opens a thinking block before every answer, and the suite runs thinking off with a 640-token cap. Most of its GSM8K and IFEval replies were still reasoning when generation reached the cap, so those grades confound answer quality with the 640-token limit. The rerun keeps the same prompts, graders and thinking-off rule and only raises the cap to 2048. The 640-cap rows stay in the matrix as their own condition.
What the cut replies mean. A reply cut by the window exceeded the export's prompt-plus-reply length in this test; a reply cut by the cap is one the benchmark chose not to wait for. In every cell 10 to 15 of 60 replies were cut. At 2k on the Poco all 14 reached the window limit and none the cap; in the seven other cells, at 4k and 32k on every phone, 10 to 15 reached the 2048 cap and none the window. Some prompts make this model reason past 2000 tokens.
The kill rule. Samsung's Heimdall wrote "Trigger Global kill before GC, Usage 8.67 GB, Threshold 6 GB" on the S25 Ultra and S24+. MIUI on the Poco wrote "used too many pss resource, pss used 8653468kb, pss threshold 6291456kb". Different code, same number: on those two phones a process whose PSS passes 6 GB is stopped. The 32k Qwen3 export needs about 7 GB of KV cache before the first prompt; it measured 6.1 to 6.4 GB resident after load on the three runs that were stopped and 8.5 GB on the Pixel run that finished.
What was not rerun. The cloud phones' suite timings, which differed by up to threefold between two launches of the same Tensor G5 cell and may reflect different thermal states. A repeat would be as launch-dependent; a paired probe on those phones is required to compare their speed.
Cost. The cloud reruns took 22 Test Lab launches, one set per launch and continued where the 38-minute budget cut them; 20 ran and 2 were refused when the Blaze plan's quota of 50 physical-device tests per day ran out at 13:19. They resumed at 15:05 after the reset and finished at 18:51. The Poco ran its four steps in 3 h 20 min.

The rerun tables

The rerun cells and controls. The full matrix, with the 640-cap rows and the fixed-prompt speed probe, is on the window page.

Where the numbers come from. Grades, controls and memory figures are the rendered matrix's. The cut reasons and the reply-length histogram come from the graded JSON reports, which record prompt and reply token counts per prompt; a reply is counted as cut by the window when prompt plus reply reached the window size, and as cut by the cap when the reply alone reached 2048. Phone RAM is the manufacturer's figure for each device. Launch times and counts are from the run logs.
Terms. A cell is one export on one phone over the prompt set. An export is the model compiled for ExecuTorch at a fixed context window. Greedy means the most likely token is always taken, so a run should repeat exactly. Thinking off means the model is not given room to reason before answering, though the 2.6B's template does so anyway. GSM8K is grade-school maths, IFEval instruction following, BFCL tool calling. Raw stream is every token generated, shown text is what a user would see after tool-call markup is removed, parsed tool calls are the function calls the app would act on. TTFT is time to first token, measured here as the runtime's prefill of the prompt; TPOT is time per output token, decode time over generated tokens. RSS is the process's resident memory; PSS is its proportional share, the figure the memory guards read. The KV cache is the memory a model keeps per token of window. Heimdall is Samsung's memory guard. Test Lab is Firebase's rented-phone service.
Where things are. Raw and graded reports under tools/eval/results: the capped rerun as -cap2048 files, continued sets as @n, second runs under a repeat- prefix, the Poco kill evidence under qwen3-32k-poco-rerun/. The note: executorch-window-matrix.md. The exports on Hugging Face: LFM2.5-2.6B, LFM2.5-1.2B, Qwen3-1.7B.