# Quantization selection notes

The practical default/mobile recommendation is
**webgpu-q4g32-e4-gram8192-emse**, at **2,345,711,616 weight bytes**. It combines
Gram8192 linears with Q4 embedding-only MSE scale search. It scores **44.8585**
WikiText screening perplexity, **0.07180** mean prompt KL, and **6/6** next-token
top1 matches. Its weights are 9.81% smaller than the Q8 embedding profile.
The tiny short-corpus PPL is 54.878, so it is a size/quality tradeoff rather than
a universal improvement on every diagnostic.

The balanced full-size choice is **webgpu-q4g32-e8-gram8192**: 2,600,908,800
weight bytes, signed Q4 group32 linears, Q8 embedding, FP16 scales/norms, and
the same WGSL tensor layout as the original export. The trained two passes
share one physical copy of every parameter. Calibration hooks asserted two
visits per physical linear, gathering both activation distributions.

On the identical 4,080 scored WikiText-2 test tokens, its perplexity is
**45.2374**, compared with source FP32 **41.7738** and original absmax Q4g32
**48.8285**. That is 7.35% lower perplexity than the original Q4 candidate and
48.95% less excess NLL above source, with identical model byte size and no
runtime additions. It still has 8.29% higher perplexity than source FP32.

The six-prompt next-token comparison matches source top1 in 6/6 cases and has
mean KL 0.06777 versus the original Q4's 0.07496. The separate tiny eight-text
corpus gives PPL 52.873 versus original Q4 53.303. These are diagnostic checks;
they do not measure general instruction-following accuracy.

## What was tried

- **Unweighted MSE scale search** worsened heldout PPL to 51.788, despite
  optimizing ordinary weight error. Its mean prompt KL was 0.12777 and top1
  matched 4/6. Lower weight error did not establish better model preservation.
- **512 train calibration tokens** plus grouped covariance scale search and
  one integer coordinate-descent sweep reached PPL 45.796, KL 0.06351, 6/6
  top1. Complete export took 66.19 seconds and peaked at 16.85 GB CUDA allocation.
- **2,048 calibration tokens** reached the best WikiText screening PPL,
  44.827, but worse prompt KL 0.12230 and 5/6 top1. It remains a useful alternate,
  not an unqualified winner. Export took 72.64 seconds.
- **8,192 calibration tokens** produced the balanced choice above. Export took
  71.62 seconds; this is a measured run, not a runtime guarantee.
- **Q4 embedding compact candidates** reuse all Gram8192 linears and reduce
  weights to 2,345,711,616 bytes, saving 255,197,184 bytes (9.81%). Both RTN and
  embedding-only MSE exports were evaluated. RTN gave PPL 45.1335, mean KL
  0.07327 and 6/6 top1; embedding MSE gave PPL 44.8585, mean KL 0.07180 and
  6/6 top1. MSE is recommended for the compact profile. The original full-model
  MSE failure therefore does not imply that embedding-only MSE is also harmful.

## Reproduction and limits

On RunPod, run:

```sh
python remote_group_gram.py --samples 64 --variant webgpu-q4g32-e8-gram8192
python remote_compact_embedding.py --base webgpu-q4g32-e8-gram8192 --variant webgpu-q4g32-e4-gram8192-emse --mse
```

The scripts live in `experiments/quantization`. The report records exact train token IDs,
dataset fingerprint, seed, both-loop activation counts, tensor error metrics,
elapsed time and peak CUDA allocation. The format-compatible compact exporter
is `remote_compact_embedding.py`. Embeddings use ordinary RTN/MSE because they
are lookups, not linear layers with a calibrated input covariance.

This is **grouped quadratic reconstruction**, not full GPTQ or AWQ. It uses
damped 32-by-32 covariance matrices without cross-group terms, scale candidate
selection, and bounded integer coordinate descent. It adds no activation
permutation, scale factors at runtime, or recurrent-depth modification.

The original screening tokens were loaded from the existing results rather
than re-tokenized; all candidates match token SHA256
`c7d809472e5fe905d974d502db6690616919b118e7d4e4963421caed411fe136`.
Calibration used WikiText **train** only. Repeatedly selecting against the same
4,080 test tokens means this is a small development screening set, not an
independent final evaluation. Broader instruction, multilingual, code, and
long-context comparisons remain necessary. PyTorch checks exported weight
quality; browser parity and throughput need their separate runtime evidence.

`quality-summary.json` and `quality-summary.md` are regenerated from raw JSON
using `experiments/quantization/summarize_quality.py`. Raw per-sequence losses,
prompt generations, next-token logits, calibration reports, and export logs
are retained beside these notes.