OpenWeights · research · 2026-09-07

Does the exported window matter? · ExecuTorch, 2k to 32k

ExecuTorch fixes a model's context window when the model is exported. This page asks whether that choice changes anything but memory. Pick a model or condition; every chart follows.

Purpose

Decide whether a full sweep of five windows, five models and four chips (about 50 device hours) was worth running, by measuring the same weights exported at 2k, 4k, 8k, 16k and 32k on the same prompts. Five windows of LFM2.5-1.2B ran on the Dimensity 9400; the other model and phone pairs ran the smallest and largest window.

How it was graded

90 prompts per cell: 30 GSM8K (the last number in the reply must equal the reference), 30 IFEval (every instruction in the prompt must be satisfied, checked by the benchmark's own code), 30 BFCL (exactly one call to the right function with the right arguments). Greedy decoding, thinking off, replies capped at 640 tokens (384 for BFCL). The 2.6B was also rerun at a 2048 cap.

How long it ran

Model	Dimensity	8 Elite	Tensor G5	Exynos
LFM2.5-1.2B	63	12	43	35
LFM2.5-2.6B, 640 cap	129	75	128	117
LFM2.5-2.6B, 2048 cap	146	130	213	195
Qwen3-1.7B	33	19	57	25

Minutes the model spent generating, summed over every run in that cell; 23.7 hours in all. Loading, cooling and Test Lab queueing came on top. Run on 2026-09-07.

Answers and tool calls

No consistent directional effect found

Two windows agree on 12 to 19 of 90 replies; the same file run twice agrees on 7 to 15. Grades move both ways between windows by about as much as between repeats.

Speed

Paired medians close; Qwen3 32k slower

Same prompt, same turn, Dimensity 9400: LFM2.5 windows within 2.4% of each other. Qwen3 32k was 5.9% slower on prefill and 4.7% on decode than 2k.

Memory

The one thing the window buys, at load

LFM2.5 pays 25 to 33 KB per token of window; Qwen3-1.7B pays 224 KB, so its 32k export needs over 6 GB before the first prompt and 12 GB phones kill it.

Where the window did matter

At 2k, 14 of 60 long replies hit the window

The 2.6B reasons before it answers. At 2k, prompt plus reply reached the window and was cut; at 4k and above only the reply cap cut them. Grades barely moved.

The full sweep

Screen the endpoints before any full sweep

Compute the KV cost, run the smallest and largest window per model and chip, and add middle windows only where those two diverge or fail.

Memory after load

Resident memory of the test process right after each export loaded, Dimensity 9400, from the fixed-prompt probe. The picked model is drawn solid.

Speed, paired probe

Each window's throughput as a ratio of the model's smallest window, same 929-token prompt, same turn against same turn, six pairs per file. Dot is the median, bar the range. Dimensity 9400 only.

PrefillDecode1.00 means no difference

Grades by window

Identical replies

Two things the matrix found that it was not looking for. Every compiled model this app had ever run was fed no start-of-text token: the ExecuTorch 1.4.0 runtime adds one only when asked and the app never asked. LFM2.5-1.2B shrugs it off; LFM2.5-2.6B scored 0 of 30 on GSM8K on every chip until the engine wrote the token itself, and every 2.6B cell here is from the fixed build. The 1.2B cells remain BOS-less as measured: comparable among themselves, and a description of that deployed configuration. And a phone that has dozed throttles instrumentation two to five times with wild variance, so the local runner now wakes the phone and holds it awake.

Two 2.6B conditions. Its own template makes it reason before answering, so under the suite's 640-token cap 16 of 30 GSM8K and 26 of 30 IFEval replies never reached an answer: those rows are cap-censored and kept for their BFCL calls, memory and speed. The rerun at a 2048 cap, GSM8K and IFEval only, is the model's real score: 23 to 28 of 30 and 18 to 21 of 30 on every phone and window, with its same-file repeat moving grades by 2 and its BFCL calls unchanged.

Qwen3 32k on 12 GB phones. Samsung's Heimdall memory guard killed the test process on the Galaxy S25 Ultra and S24+ after 7 to 12 prompts ("Trigger Global kill before GC, Usage 8.67 GB, Threshold 6 GB" in the logcats). A rerun on the Poco with logcat attached died the same way after 7: MIUI's ActivityManager force-stopped it for "used too many pss resource, pss used 8.65 GB, threshold 6 GB". Only the 16 GB Pixel 10 Pro XL finished. Those cells are marked incomplete and carry no grade or timing.

Not matched conditions. The Dimensity 9400 ran in hand, cooled below 42 C before each probe load. The 8 Elite, Tensor G5 and Exynos 2400 ran racked on Firebase Test Lab, where phones throttle within about 40 seconds, and had no paired probe: their suite timings are per-cell medians over different replies and do not support a speed claim either way.

Every number

The full matrix, as rendered from the graded reports by window_report.py. Prefill ms is the runtime's prefill of the whole prompt, the engine-side part of time to first token. ms/token is decode time over generated tokens; replies differ per cell, so it is not a paired comparison. Capped counts replies that ran to the token cap (640, or 384 for BFCL). RSS is resident memory right after load and at the end of the run.

What was run. LFM2.5-1.2B: five windows on the Dimensity 9400, 16k and 32k twice; 4k and 32k on the cloud chips, 4k twice on the Exynos. LFM2.5-2.6B: 2k and 32k on the Dimensity, 2k twice; 4k and 32k on the cloud chips; then GSM8K and IFEval again at a 2048 cap on every one of those cells. Qwen3-1.7B: 2k and 32k everywhere, 2k twice on the Dimensity, 32k twice on the Dimensity to catch the kill. Every file loaded at its own exported window. Exports use ExecuTorch 1.4.0's recipes: 8-bit dynamic activations, 4-bit weights in groups of 32, 8-bit embeddings, fp32 KV cache, prefill chunk 2048.

Review. Codex reviewed the method twice, before and after the data: same-file repeats, a paired speed probe separate from the task eval, a three-way identity metric counted over BFCL only, memory after load, a second architecture, incomplete cells shown as such, and conclusions bounded by the observed run variation. All are in what you see here.

Where things are. The note: executorch-window-matrix.md. The tables: window-matrix.md. Raw and graded reports: tools/eval/results. The exports, each window a file: LFM2.5-1.2B, LFM2.5-2.6B, Qwen3-1.7B on Hugging Face.