# Activation-aware Q4 candidate

`remote_group_gram.py` preserves existing signed Q4 group32 packing, FP16 group
scales, Q8 embeddings, four head/embedding parts, and physical shared weights.
The browser needs no kernel or loader changes. Output defaults to
`/workspace/nanbeige/webgpu-q4g32-e8-gram512`; an existing output directory causes
an error, protecting previous artifacts.

## Run on the existing RunPod GPU only

From `/workspace/nanbeige`, with the existing reference and Python dependencies:

```sh
PYTHONPATH=/workspace/nanbeige python -u remote_group_gram.py --max-seconds 1500 > gram512-export.log 2>&1
PYTHONPATH=/workspace/nanbeige python remote_quant_eval.py --variant webgpu-q4g32-e8-gram512
```

Copy the script to the working remote directory first. Use the root's existing
held-out corpus evaluator with this variant as an additional candidate; its
current hardcoded list does not include this candidate automatically. Evaluate
the same 4,080 held-out test tokens and prompt/logit comparisons before choosing
this artifact. Do not select it from calibration error alone.

Default calibration is four deterministic 128-token windows from WikiText-2
**train**, seed 20260907. It records exact token IDs and dataset fingerprint.
Hooks observe both recurrent visits (asserted twice per sample) for all 22
physical layers; each hook therefore sees 1,024 activation rows. Q/K/V share
input statistics, as do gate/up. The head sees 512 rows. The model runs its
unaltered trained two-pass execution, in FP32 with SDPA and no KV cache.

The script stores a 32-by-32 input covariance per input group. It adds 1%
diagonal damping, chooses from ten FP16 scale candidates using quadratic
reconstruction error, then performs one exact coordinate-descent sweep over
integer weights. It retains the pre-sweep candidate if floating point error
makes the final objective worse. Absmax RTN is among the candidates, so the
**damped within-group calibration objective** cannot get worse. Full output
error, held-out perplexity and generation quality have no such guarantee:
cross-group correlations and propagated quantization errors are omitted.

This is a bounded grouped quadratic reconstruction experiment. It is not the
full [GPTQ method](https://arxiv.org/abs/2210.17323), which uses inverse-Hessian
error compensation, nor [AWQ](https://arxiv.org/abs/2306.00978), which applies
equivalent activation-aware channel scaling. Those papers motivate using
activation information rather than only unweighted weight MSE.

Measured on the task's RTX A6000: **66.19 seconds** for calibration plus complete
export, **16,847,380,480 bytes** peak CUDA allocation, and **2,600,908,800 bytes**
of browser weights. The first budget estimate of 10–25 minutes was conservative;
the actual grouped implementation completed much faster. Host RAM was not
measured. The default 1,500-second cap is checked between forward passes
and row batches. CUDA allocations are additionally limited to 34 GiB (or 90%
of device memory, whichever is lower). A timeout leaves a failed report and
partial files, never a completed `manifest.json`. It does not terminate a pod;
the root agent remains responsible for pod lifetime and total cost.

`quantization-report.json` contains method, arguments, calibration provenance,
per-tensor ordinary weight RMSE and baseline/chosen quadratic errors, timings,
peak CUDA allocation, and completion status. `calibration-grams.pt` enables
inspection but is not a browser artifact. `manifest.json` is written only once
every tensor export succeeds. To spend less GPU time set `--sweeps 0`; to improve
calibration coverage within budget raise `--samples` (rename the variant).

## Local lightweight check

```sh
python3 experiments/quantization/check_coordinate_math.py
```

This tiny pure-Python synthetic check verifies that the closed-form update
matches exhaustive minimization over all 16 integer levels, each step decreases
quadratic error, residual-gradient updates stay correct, and nibble packing
round-trips. It does not validate CUDA execution or actual model quality.

## Compact embedding candidate

`remote_compact_embedding.py` reuses an already exported base's linears and
normalizations with hardlinks, quantizes only original source embeddings to
Q4 group32, and emits a new manifest. Changed files use exclusive creation to
avoid overwriting shared inodes. Embeddings are lookup weights, so this step
uses ordinary absmax RTN or `--mse` weight-error scale search; it never applies
a Linear-input Gram matrix to embeddings.

```sh
python -u remote_compact_embedding.py --base webgpu-q4g32-e8-gram512 --variant webgpu-q4g32-e4-gram512
python -u remote_compact_embedding.py --base webgpu-q4g32-e8-gram512 --variant webgpu-q4g32-e4-gram512-emse --mse
```

The `--base` may be changed to the selected Gram candidate; name the output
accordingly. Evaluate each compact variant using the same scripts before
publishing it. The expected byte saving is 255,197,184, giving 2,345,711,616
browser weight bytes with this model's vocabulary and dimensions.

## Reproducible comparison

The extended `scripts/remote_corpus_eval.py` accepts `--variants` (multiple
variant directory names), `--output` (a separate results filename), and
`--tokens-from` (saved token IDs from an earlier screening result). It appends
only missing variants and refuses to merge different token hashes or sequence
lengths, preserving existing measurements. Token-level weighting handles a
partial final sequence correctly. `source-fp32`, when requested, always runs
before any quantized weights are assigned.

`summarize_quality.py` reads the small result JSON files, asserts matching
WikiText token hashes and sequence lengths, and writes `quality-summary.json`
and `quality-summary.md`. It is safe to run locally; it never loads weights.