Muse-Glimmer-30B (2x Tesla T4 NF4): LoRAs as Parametric Memory vs. Context Shimming
Pre-registered unseen holdout (n=120: 60 post-cutoff 2026 Task Object recall + 60 nonprofit Task/Object splits) | 48 QLoRA steps (2327s) | rev a4e59da5
Unseen Holdout Exact-Match Accuracy by Regime & Subtask
Strict canonical JSON exact match; whiskers show 95% Wilson CIs (n=120 overall, n=60 per subtask)
Overall Holdout (n=120)
Subtask A: 2026 Task-Object Recall (n=60)
Subtask B: Nonprofit Task/Object Split (n=60)
0%
20%
40%
60%
80%
100%
Base Closed-Book
0 shim tok
Schema valid: 21.7%
0.0%
0.0%
0.0%
Base + BM25 Top-2 Shim
+222 shim tok
Schema valid: 35.0%
29.2%
58.3%
0.0%
QLoRA-as-Memory (r=16)
0 shim tok (weights!)
Schema valid: 100.0%
62.5%
68.3%
56.7%
Hybrid (QLoRA + 1-Line Shim)
+71 shim tok
Schema valid: 100.0%
55.0%
68.3%
41.7%
Paired Bootstrap 95% CIs (2,000x)
Exact-match delta on identical holdout items
QLoRA (0-ctx) vs Base (0-ctx)
+62.5 pp [+53.3, +70.8]
QLoRA (0-ctx) vs Base + BM25 Shim
+33.3 pp [+22.5, +44.2]
Hybrid vs Base (0-ctx)
+55.0 pp [+46.7, +64.2]
Hybrid vs Base + BM25 Shim
+25.8 pp [+15.8, +35.8]
Base + BM25 Shim vs Base (0-ctx)
+29.2 pp [+21.7, +37.5]
Sanity, Safety & Integrity Gates
Sanity Accuracy (n=40):
97.5% -> 95.0% (JSON intr. 0.0%)
Compositional 2-Object Recall:
Base 0.0% | Shim 25.0% | LoRA 45.8%
Builder != Reviewer (Subtask B):
Base 5.0% -> QLoRA 100.0%
Private-Data Tier Gate (Subtask B):
Base 21.7% -> QLoRA 100.0%
Prompt Shim Overhead Saved:
BM25 Shim +222 tok vs QLoRA +0 tok
Frozen Base / Reload Parity:
disable_adapter=0.0, 8/8 byte-exact
Pre-Registered Rule Verdict:
LoRA vs Base: promote_candidate | Hybrid vs Shim: promote_candidate
Scope note: 100%-owned deterministic synthetic benchmark on Kaggle free 2xT4 ($0.00 spend, private artifacts). Adapter is an experimental evaluation receipt, not an admitted or deployed service.