BitBlame — dynamic quantization forensics (SmolLM2-135M audit)

Which layers lose accuracy when a model is quantized? BitBlame measures per-unit quantization sensitivity with confidence intervals, and turns the measurement into a mixed-precision GGUF recipe. This page shows the preloaded Phase B audit on SmolLM2-135M — no live run.

Per-unit blame — SmolLM2-135M

Per-unit blame scores with 95% BCa confidence intervals

Top-3 units (embed + blk.29 + blk.11) = 57.9% of total measured sensitivity; embedding alone 32.0%. Split-half rank stability rho = 0.9754; 31/31 units pass the significance gate (null-floor max CI-high = 0.00307). Screen-vs-exact ranking agreement: Spearman rho = 0.9286 at 4-bit (top-8).

#	unit	S_u	95% CI	share	gate	alloc
1	embed	0.3674	[0.3471, 0.3927]	32.0%	yes	Q5_K
2	blk.29	0.1604	[0.1377, 0.2286]	14.0%	yes	Q6_K
3	blk.11	0.1366	[0.1276, 0.1434]	11.9%	yes	Q8_0
4	blk.28	0.0652	[0.0585, 0.0739]	5.7%	yes	Q6_K
5	blk.2	0.0503	[0.0413, 0.0612]	4.4%	yes	Q6_K
6	blk.9	0.0351	[0.0267, 0.0466]	3.1%	yes	Q5_K
7	blk.27	0.0229	[0.0210, 0.0253]	2.0%	yes	Q5_K
8	blk.19	0.0182	[0.0173, 0.0192]	1.6%	yes	Q5_K
9	blk.0	0.0174	[0.0153, 0.0202]	1.5%	yes	Q5_K
10	blk.18	0.0167	[0.0156, 0.0182]	1.5%	yes	Q5_K
11	blk.23	0.0160	[0.0143, 0.0179]	1.4%	yes	Q5_K
12	blk.3	0.0153	[0.0131, 0.0184]	1.3%	yes	Q4_K
13	blk.16	0.0149	[0.0137, 0.0160]	1.3%	yes	Q4_K
14	blk.12	0.0142	[0.0132, 0.0153]	1.2%	yes	Q4_K
15	blk.22	0.0139	[0.0130, 0.0152]	1.2%	yes	Q4_K
16	blk.21	0.0136	[0.0126, 0.0146]	1.2%	yes	Q4_K
17	blk.20	0.0133	[0.0123, 0.0157]	1.2%	yes	Q4_K
18	blk.1	0.0132	[0.0112, 0.0155]	1.2%	yes	Q4_K
19	blk.15	0.0130	[0.0119, 0.0139]	1.1%	yes	Q4_K
20	blk.14	0.0127	[0.0115, 0.0142]	1.1%	yes	Q2_K
21	blk.13	0.0126	[0.0115, 0.0137]	1.1%	yes	Q2_K
22	blk.17	0.0122	[0.0108, 0.0160]	1.1%	yes	Q2_K
23	blk.24	0.0118	[0.0111, 0.0126]	1.0%	yes	Q2_K
24	blk.10	0.0118	[0.0107, 0.0131]	1.0%	yes	Q2_K
25	blk.8	0.0115	[0.0105, 0.0128]	1.0%	yes	Q2_K
26	blk.4	0.0107	[0.0089, 0.0134]	0.9%	yes	Q2_K
27	blk.5	0.0104	[0.0090, 0.0124]	0.9%	yes	Q2_K
28	blk.7	0.0098	[0.0089, 0.0112]	0.9%	yes	Q2_K
29	blk.6	0.0095	[0.0085, 0.0112]	0.8%	yes	Q2_K
30	blk.26	0.0086	[0.0082, 0.0091]	0.8%	yes	Q2_K
31	blk.25	0.0076	[0.0072, 0.0083]	0.7%	yes	Q2_K

The measured recipe

Exact-DP knapsack over 6 real GGUF quant types, emitted as llama-quantize --tensor-type overrides. Resulting GGUF: 94,373,920 B vs Q4_K_M baseline 105,453,856 B (ratio 0.8949 — 10.5% smaller). Note: the output.weight=Q5_K line is a no-op on this tied-embedding model (no such tensor exists); unmentioned tensors fall back to Q4_K_M. Full recipe file: recipe.sh in the dataset.

#!/bin/sh
# BitBlame recipe — measured-sensitivity-driven GGUF quantization
# Generated per METHODOLOGY.md §7.4 (spec v1.0).
# Unmentioned tensors fall back to the base type.
set -e

# 1) importance matrix on the calibration mix (§5.1)
llama-imatrix -m model-f16.gguf -f calib.txt -o imatrix.dat

# 2) quantize with per-tensor overrides (measured allocation)
llama-quantize --imatrix imatrix.dat \
  --tensor-type 'blk.0.attn_q.weight=Q5_K' \
  --tensor-type 'blk.0.attn_k.weight=Q5_K' \
  --tensor-type 'blk.0.attn_v.weight=Q5_K' \
  --tensor-type 'blk.0.attn_output.weight=Q5_K' \
  --tensor-type 'blk.0.ffn_gate.weight=Q5_K' \
  --tensor-type 'blk.0.ffn_up.weight=Q5_K' \
  --tensor-type 'blk.0.ffn_down.weight=Q5_K' \
  --tensor-type 'blk.10.attn_q.weight=Q2_K' \
  --tensor-type 'blk.10.attn_k.weight=Q2_K' \
  --tensor-type 'blk.10.attn_v.weight=Q2_K' \
  --tensor-type 'blk.10.attn_output.weight=Q2_K' \
  --tensor-type 'blk.10.ffn_gate.weight=Q2_K' \
  --tensor-type 'blk.10.ffn_up.weight=Q2_K' \
  --tensor-type 'blk.10.ffn_down.weight=Q2_K' \
  --tensor-type 'blk.11.attn_q.weight=Q8_0' \
  --tensor-type 'blk.11.attn_k.weight=Q8_0' \
  --tensor-type 'blk.11.attn_v.weight=Q8_0' \
  --tensor-type 'blk.11.attn_output.weight=Q8_0' \
  --tensor-type 'blk.11.ffn_gate.weight=Q8_0' \
  --tensor-type 'blk.11.ffn_up.weight=Q8_0' \
  --tensor-type 'blk.11.ffn_down.weight=Q8_0' \
  --tensor-type 'blk.13.attn_q.weight=Q2_K' \
  --tensor-type 'blk.13.attn_k.weight=Q2_K' \
  --tensor-type 'blk.13.attn_v.weight=Q2_K' \
  --tensor-type 'blk.13.attn_output.weight=Q2_K' \
  --tensor-type 'blk.13.ffn_gate.weight=Q2_K' \
  --tensor-type 'blk.13.ffn_up.weight=Q2_K' \
  --tensor-type 'blk.13.ffn_down.weight=Q2_K' \
  --tensor-type 'blk.14.attn_q.weight=Q2_K' \
  --tensor-type 'blk.14.attn_k.weight=Q2_K' \
  --tensor-type 'blk.14.attn_v.weight=Q2_K' \
  --tensor-type 'blk.14.attn_output.weight=Q2_K' \
  --tensor-type 'blk.14.ffn_gate.weight=Q2_K' \
  --tensor-type 'blk.14.ffn_up.weight=Q2_K' \
  --tensor-type 'blk.14.ffn_down.weight=Q2_K' \
  --tensor-type 'blk.17.attn_q.weight=Q2_K' \
  --tensor-type 'blk.17.attn_k.weight=Q2_K' \
  --tensor-type 'blk.17.attn_v.weight=Q2_K' \
  --tensor-type 'blk.17.attn_output.weight=Q2_K' \
  --tensor-type 'blk.17.ffn_gate.weight=Q2_K' \
  --tensor-type 'blk.17.ffn_up.weight=Q2_K' \
  --tensor-type 'blk.17.ffn_down.weight=Q2_K' \
  --tensor-type 'blk.18.attn_q.weight=Q5_K' \
  --tensor-type 'blk.18.attn_k.weight=Q5_K' \
  --tensor-type 'blk.18.attn_v.weight=Q5_K' \
  --tensor-type 'blk.18.attn_output.weight=Q5_K' \
  --tensor-type 'blk.18.ffn_gate.weight=Q5_K' \
  --tensor-type 'blk.18.ffn_up.weight=Q5_K' \
  --tensor-type 'blk.18.ffn_down.weight=Q5_K' \
  --tensor-type 'blk.19.attn_q.weight=Q5_K' \
  --tensor-type 'blk.19.attn_k.weight=Q5_K' \
  --tensor-type 'blk.19.attn_v.weight=Q5_K' \
  --tensor-type 'blk.19.attn_output.weight=Q5_K' \
  --tensor-type 'blk.19.ffn_gate.weight=Q5_K' \
  --tensor-type 'blk.19.ffn_up.weight=Q5_K' \
  --tensor-type 'blk.19.ffn_down.weight=Q5_K' \
  --tensor-type 'blk.2.attn_q.weight=Q6_K' \
  --tensor-type 'blk.2.attn_k.weight=Q6_K' \
  --tensor-type 'blk.2.attn_v.weight=Q6_K' \
  --tensor-type 'blk.2.attn_output.weight=Q6_K' \
  --tensor-type 'blk.2.ffn_gate.weight=Q6_K' \
  --tensor-type 'blk.2.ffn_up.weight=Q6_K' \
  --tensor-type 'blk.2.ffn_down.weight=Q6_K' \
  --tensor-type 'blk.23.attn_q.weight=Q5_K' \
  --tensor-type 'blk.23.attn_k.weight=Q5_K' \
  --tensor-type 'blk.23.attn_v.weight=Q5_K' \
  --tensor-type 'blk.23.attn_output.weight=Q5_K' \
  --tensor-type 'blk.23.ffn_gate.weight=Q5_K' \
  --tensor-type 'blk.23.ffn_up.weight=Q5_K' \
  --tensor-type 'blk.23.ffn_down.weight=Q5_K' \
  --tensor-type 'blk.24.attn_q.weight=Q2_K' \
  --tensor-type 'blk.24.attn_k.weight=Q2_K' \
  --tensor-type 'blk.24.attn_v.weight=Q2_K' \
  --tensor-type 'blk.24.attn_output.weight=Q2_K' \
  --tensor-type 'blk.24.ffn_gate.weight=Q2_K' \
  --tensor-type 'blk.24.ffn_up.weight=Q2_K' \
  --tensor-type 'blk.24.ffn_down.weight=Q2_K' \
  --tensor-type 'blk.25.attn_q.weight=Q2_K' \
  --tensor-type 'blk.25.attn_k.weight=Q2_K' \
  --tensor-type 'blk.25.attn_v.weight=Q2_K' \
  --tensor-type 'blk.25.attn_output.weight=Q2_K' \
  --tensor-type 'blk.25.ffn_gate.weight=Q2_K' \
  --tensor-type 'blk.25.ffn_up.weight=Q2_K' \
  --tensor-type 'blk.25.ffn_down.weight=Q2_K' \
  --tensor-type 'blk.26.attn_q.weight=Q2_K' \
  --tensor-type 'blk.26.attn_k.weight=Q2_K' \
  --tensor-type 'blk.26.attn_v.weight=Q2_K' \
  --tensor-type 'blk.26.attn_output.weight=Q2_K' \
  --tensor-type 'blk.26.ffn_gate.weight=Q2_K' \
  --tensor-type 'blk.26.ffn_up.weight=Q2_K' \
  --tensor-type 'blk.26.ffn_down.weight=Q2_K' \
  --tensor-type 'blk.27.attn_q.weight=Q5_K' \
  --tensor-type 'blk.27.attn_k.weight=Q5_K' \
  --tensor-type 'blk.27.attn_v.weight=Q5_K' \
  --tensor-type 'blk.27.attn_output.weight=Q5_K' \
  --tensor-type 'blk.27.ffn_gate.weight=Q5_K' \
  --tensor-type 'blk.27.ffn_up.weight=Q5_K' \
  --tensor-type 'blk.27.ffn_down.weight=Q5_K' \
  --tensor-type 'blk.28.attn_q.weight=Q6_K' \
  --tensor-type 'blk.28.attn_k.weight=Q6_K' \
  --tensor-type 'blk.28.attn_v.weight=Q6_K' \
  --tensor-type 'blk.28.attn_output.weight=Q6_K' \
  --tensor-type 'blk.28.ffn_gate.weight=Q6_K' \
  --tensor-type 'blk.28.ffn_up.weight=Q6_K' \
  --tensor-type 'blk.28.ffn_down.weight=Q6_K' \
  --tensor-type 'blk.29.attn_q.weight=Q6_K' \
  --tensor-type 'blk.29.attn_k.weight=Q6_K' \
  --tensor-type 'blk.29.attn_v.weight=Q6_K' \
  --tensor-type 'blk.29.attn_output.weight=Q6_K' \
  --tensor-type 'blk.29.ffn_gate.weight=Q6_K' \
  --tensor-type 'blk.29.ffn_up.weight=Q6_K' \
  --tensor-type 'blk.29.ffn_down.weight=Q6_K' \
  --tensor-type 'blk.4.attn_q.weight=Q2_K' \
  --tensor-type 'blk.4.attn_k.weight=Q2_K' \
  --tensor-type 'blk.4.attn_v.weight=Q2_K' \
  --tensor-type 'blk.4.attn_output.weight=Q2_K' \
  --tensor-type 'blk.4.ffn_gate.weight=Q2_K' \
  --tensor-type 'blk.4.ffn_up.weight=Q2_K' \
  --tensor-type 'blk.4.ffn_down.weight=Q2_K' \
  --tensor-type 'blk.5.attn_q.weight=Q2_K' \
  --tensor-type 'blk.5.attn_k.weight=Q2_K' \
  --tensor-type 'blk.5.attn_v.weight=Q2_K' \
  --tensor-type 'blk.5.attn_output.weight=Q2_K' \
  --tensor-type 'blk.5.ffn_gate.weight=Q2_K' \
  --tensor-type 'blk.5.ffn_up.weight=Q2_K' \
  --tensor-type 'blk.5.ffn_down.weight=Q2_K' \
  --tensor-type 'blk.6.attn_q.weight=Q2_K' \
  --tensor-type 'blk.6.attn_k.weight=Q2_K' \
  --tensor-type 'blk.6.attn_v.weight=Q2_K' \
  --tensor-type 'blk.6.attn_output.weight=Q2_K' \
  --tensor-type 'blk.6.ffn_gate.weight=Q2_K' \
  --tensor-type 'blk.6.ffn_up.weight=Q2_K' \
  --tensor-type 'blk.6.ffn_down.weight=Q2_K' \
  --tensor-type 'blk.7.attn_q.weight=Q2_K' \
  --tensor-type 'blk.7.attn_k.weight=Q2_K' \
  --tensor-type 'blk.7.attn_v.weight=Q2_K' \
  --tensor-type 'blk.7.attn_output.weight=Q2_K' \
  --tensor-type 'blk.7.ffn_gate.weight=Q2_K' \
  --tensor-type 'blk.7.ffn_up.weight=Q2_K' \
  --tensor-type 'blk.7.ffn_down.weight=Q2_K' \
  --tensor-type 'blk.8.attn_q.weight=Q2_K' \
  --tensor-type 'blk.8.attn_k.weight=Q2_K' \
  --tensor-type 'blk.8.attn_v.weight=Q2_K' \
  --tensor-type 'blk.8.attn_output.weight=Q2_K' \
  --tensor-type 'blk.8.ffn_gate.weight=Q2_K' \
  --tensor-type 'blk.8.ffn_up.weight=Q2_K' \
  --tensor-type 'blk.8.ffn_down.weight=Q2_K' \
  --tensor-type 'blk.9.attn_q.weight=Q5_K' \
  --tensor-type 'blk.9.attn_k.weight=Q5_K' \
  --tensor-type 'blk.9.attn_v.weight=Q5_K' \
  --tensor-type 'blk.9.attn_output.weight=Q5_K' \
  --tensor-type 'blk.9.ffn_gate.weight=Q5_K' \
  --tensor-type 'blk.9.ffn_up.weight=Q5_K' \
  --tensor-type 'blk.9.ffn_down.weight=Q5_K' \
  --tensor-type 'token_embd.weight=Q5_K' \
  --tensor-type 'output.weight=Q5_K' \
  model-f16.gguf model-bitblame.gguf Q4_K_M

echo "recipe done -> model-bitblame.gguf"

Run it yourself: pip install git+https://github.com/habib-analyst/bitblame.git then bitblame scan / probes / recipe / validate / report (see the repo README Quickstart).

Audit results — pre-registered paired test (n = 250)

Comparison	Recipe vs Q4_K_M baseline
Accuracy	11.60% = 11.60%
Mean paired delta	0.00 pp (15 wins, 15 losses, 220 ties)
95% BCa CI	[-4.8 pp, +4.0 pp], p = 0.5338
Verdict	NO_SIGNIFICANT_DIFFERENCE; non-inferiority at 0.5 pp margin FAILS
Size	94,373,920 B vs 105,453,856 B — 10.5% smaller, size gate (≤ 1.01x) passes
Closed loop	Harness exit 0; 272/272 tensor types verified, 0 mismatches
Optimizer bit-cost model	Reproduces 272/272 tensor types and stored bytes exactly (0.0000% error) on both GGUFs
imatrix agreement	Spearman rho = 0.2908 (p = 0.119, n = 30) — BELOW rho >= 0.5 bar; corroboration FAILS

Per-category paired deltas (recipe − baseline)

category	n	FP16	Q4_K_M	recipe	delta	95% CI	verdict
copy	20	0.0%	0.0%	0.0%	+0.00 pp	[+0.0, +0.0] pp	NO_SIGNIFICANT_DIFFERENCE
gsm8k	60	1.7%	3.3%	1.7%	-1.67 pp	[-10.0, +0.0] pp	NO_SIGNIFICANT_DIFFERENCE
humaneval	30	0.0%	0.0%	0.0%	+0.00 pp	[+0.0, +0.0] pp	NO_SIGNIFICANT_DIFFERENCE
mmlu	100	28.0%	20.0%	25.0%	+5.00 pp	[-5.0, +14.0] pp	NO_SIGNIFICANT_DIFFERENCE
needle	40	7.5%	17.5%	7.5%	-10.00 pp	[-27.5, -5.0] pp	SIGNIFICANT_REGRESSION

Needle note: the bootstrap mechanically flags SIGNIFICANT_REGRESSION, but recipe-vs-reference is 0 (both 3/40) — the baseline scored above the FP16 reference by chance.

Methodology (summary)

Stage 0 — cache FP16 logits (1 forward pass per calibration seq).
Stage 1 — mock-noise LOLO scan. Perturb one unit's weights with quantization-calibrated noise; mean per-token KL vs FP16 reference. Bootstrap BCa CIs (B=2000), null-floor significance gate, split-half reliability gate.
Stage 2 — exact top-8 verification. Re-measure top units with exact quantize→dequantize at 3/4/6 bits; power-law fits. Screen-vs-exact agreement gate rho >= 0.7.
Stage 3 — behavioral probe battery. 250 items (100 MMLU + 60 GSM8K + 30 HumanEval + 40 needle + 20 copy), FP16 vs baseline-quant, paired items.
Stage 4 — bit-budget optimizer. Significance-gated exact-DP multiple-choice knapsack over 6 real GGUF quant types → llama-quantize overrides + imatrix guidance.
Stage 5 — closed-loop validation. Re-quantize with real llama-quantize, re-measure, pre-registered paired-bootstrap verdict.

Limitations

SmolLM2-135M only; no transfer claims. Phase C (Qwen3-1.7B) pending.
Mock-noise screen is approximate; screen ranking of the 23 non-top-8 units is unverified.
Calibration N=16/T=256 (spec wants 32x512) — CPU adaptation; split-half rho = 0.9754 rescues ranking reliability.
Greedy decoding only; English calibration only. Per-unit CIs are marginal, not family-wise.
Related work: firefly, mlx-optiq, microant exist — BitBlame is not the first such tool. Its precise claim is narrower: paired-bootstrap CIs on per-layer attribution, significance-gating the allocation, for GGUF/llama.cpp.

Full spec: METHODOLOGY.md in the repo. Full numbers: P2_REPORT.md in the repo.

🔬 BitBlame — dynamic quantization forensics