It also ran against a hard wall the whole time: a single 24 GB consumer GPU. That ceiling is why so much of what follows is about bytes — VRAM allocation that has to be exact, MoE temp buffers you can't afford, a last-tensor upload that's a few hundred MB too small. The constraint was real enough that we priced out slotting a second card (a 16 GB Vega64 or RX 6800 from another machine) just to offload a few layers, and watched offloading one layer too many hang the entire desktop for a minute at a time. When you have no headroom, every sizing bug is fatal instead of merely wasteful. Most of the bugs below are sizing bugs.

Here are the lessons, each one learned the hard way.

The metadata plumbing is the real work

Adding a weight type to llama.cpp sounds like one enum and a decode function. It is not. A tensor's size is computed, checked, and assumed in a dozen places — allocation, mmap, the GGUF loader, graph planning, the VRAM budget. Every one of them assumes size is a closed-form function of shape and type (ggml_nbytes). An SCLP blob is none of those things: it's stored at its actual compressed size, which depends on how many outliers landed in the sidecar, which depends on the weights.

So the loader has to infer each blob's real size — its disk_size — from the gap between consecutive tensor offsets in the file, and thread that number through every path that previously trusted ggml_nbytes. That single concept (disk_size ≠ nbytes) touched four files and was the source of the worst bug in the project. Plumbing a new idea about size through a codebase that has one hardcoded everywhere is the actual work. The compression was the easy part.

The week-long bug: a silent size mismatch

SCLP4 prefill stalled. It didn't crash — it stalled, dropping to 0.16 t/s with the GPU pinned but making no progress. llama-bench hung the same way. Single token generation was fine; only the many-token path died.

The chain: the VRAM allocator reserves space per tensor using disk_size. The last tensor in the file has no "next offset" to subtract from, so its size fell back to ggml_nbytes — which is smaller than the compact SCLP blob. The upload was silently truncated. The GPU didn't crash; it just ground forever on bogus work.

Three things made this expensive to find. It only fired at M > 1 (prefill), because token generation used a different code path. The symptom was a hang, not an error — nothing logged, nothing threw. And the most promising clue was a trap: profiling showed a 430,940,160-byte (~411 MB) host-to-device transfer right at the stall, and that number matched the Q6_K output.weight buffer exactly. It looked like the smoking gun — until we noticed SCLP6 issued the identical 411 MB transfer and never stalled. The fingerprint was real — that transfer genuinely happened — but it pointed at the wrong tensor: output.weight wasn't what stalled us.

What had actually broken was sizing, in the loader. The missing case was the final tensor: with no next offset available, its inferred on-disk size used the old ggml_nbytes fallback instead of the real file span. The decode kernel then read past the truncated upload, picked up a garbage sidecar_count, and scatter-wrote phantom corrections forever. The fix is one line: derive the last tensor's size from the file boundary (file_size − data_offset − tensor_offset) instead of falling back to ggml_nbytes. Finding it took several days of bisecting which tensor, which batch size, which kernel — and learning to distrust the one number that looked like an answer.

The takeaway: a silent size mismatch doesn't crash, it corrupts — and corruption that happens to spin the GPU looks exactly like a performance problem until you stop trusting that framing.

Trust measurements over assumptions

We had three plausible optimizations that should have worked by every rule of thumb. All three were measured, and all three were dead ends.

• F16 decode intermediate. The idea was to decode SCLP to F16 instead of BF16 for the two-pass prefill, on the theory that more mantissa would make the GEMM happier. It measured identical to BF16 on RDNA3, because the GEMM is memory-bound at these shapes and the mantissa bits don't move the needle. We won't retry it.
• hipBLASLt. The newer BLAS path is supposed to be faster. On this hardware it came in at −4% on dense and −31% on the SCLP MoE GEMM — decisively slower, so we left it off.
• Fusing decode into the prefill GEMM. The obvious win was to skip the BF16 intermediate at M > 1 too and decode straight into the matmul. It worked on dense models. On MoE it produced catastrophic perplexity — and untangling why took a detour. The first suspect was a real bug: the fused MoE path corrupted every expert bin past the first (b≥1 tiles were garbage), traced to a wrong expert_offsets stride after the route-sort. Fixing the stride made the corruption vanish — and revealed the actual wall underneath. With routing correct, the fused result still diverged from the two-pass result by a deterministic amount, and it was pure arithmetic: the F32 accumulation order of a tensor-core tile differs from scalar sequential summation by ~1e-3 per multiply, and across 26 MoE layers that compounds. So there were two failures stacked — a fixable indexing bug hiding an unfixable ordering difference. Matching rocBLAS's exact tile order is possible but not worth a ~20% prefill win.

That made three confident guesses, three measurements, and three reversals. The pattern is the lesson: on a specific GPU, with a specific BLAS, at specific shapes, your intuition about what's faster is a hypothesis, not a fact. Measure before you commit, and write down the dead ends so they stay dead.

(One smaller version of the same lesson: mixing static __shared__ with dynamic extern __shared__ in a HIP kernel caused a 3× regression on RDNA3. Pure-dynamic shared memory fixed it. Nothing warned us; the profiler did.)

Perplexity can hide collapse — smoke-test for it

Post 2's mode-collapse story ("own own own own...") had a sharp edge worth isolating. That model's decode was byte-perfect across 508 million weights, and its perplexity number looked bad but not obviously broken on our OOD scale. Perplexity averages log-likelihood over a corpus; a model that has quietly collapsed into repeating one token can still post a number that looks merely bad rather than broken — especially when, as we found, OOD wikitext perplexity is inflated ~50× and you've trained yourself to read big numbers as normal.

The only thing that reliably caught collapse was generating a couple hundred tokens of actual chat and reading them. So that became a hard gate: no quant is "good" until it has produced 200+ coherent tokens to a real prompt. A perplexity table is necessary but not sufficient — the inexpensive qualitative check is the one that catches the failure the metric hides.

Don't kill -9 on WSL2/ROCm

A practical one that cost real hours. This work ran on WSL2 with ROCm. If you kill -9 a llama process while it's mid-GPU-kernel, the ROCm runtime doesn't clean up — the GPU wedges, and everything afterward drops to ~0.16 t/s (the same number as the truncation bug, which made triage briefly confusing). In our setup, the reliable recovery was wsl --shutdown.

So: bound every GPU job with timeout instead of reaching for kill, and never SIGKILL a process that's touching the device. On WSL2 the GPU isn't a resource the OS will reclaim for you. Treat a wedged GPU as a possible environment state, not only a code bug — we spent time hunting a regression that was really a leftover wedge from a previous kill -9.

That ~0.16 t/s reading turned out to be badly overloaded: three unrelated failures all produced it: the truncated last-tensor upload, a genuine kill -9 GPU wedge, and — the one that fooled us longest — running the fused GEMV on an old model whose sidecar wasn't sorted. The fused kernel binary-searches each row's sidecar range assuming the entries are sorted by index; on an unsorted blob the search returns a bogus enormous range, and every row grinds through millions of phantom corrections — producing the same 0.16 t/s. We first diagnosed it as a wedge and reached for wsl --shutdown — which of course "fixed" nothing, because the next run loaded the same stale model. The lesson stacks on the truncation one: when three different bugs share a symptom, the symptom tells you almost nothing. You have to find the fingerprint that distinguishes them.

What the bottom of the stack taught us

Step back and the lessons rhyme:

1. A new invariant is expensive to introduce. "Size isn't ggml_nbytes anymore" was one sentence, four files, and the worst bug.
2. Silent corruption masquerades as slowness. A truncated upload, an unsorted sidecar, and a genuine GPU wedge all produced the same 0.16 t/s — three different bugs, one symptom. Distrust the performance framing of a hang; chase the fingerprint, not the speed.
3. Your speed intuitions are hypotheses. F16, hipBLASLt, fused prefill — three "obviously faster" ideas, three losses on measurement.
4. The cheap eval is the one you skip and the one that catches the disaster. Byte-perfect decode and a not-even-scary perplexity both passed while the model said "own own own."
5. The environment has state. On WSL2/ROCm a kill can outlive its process.

None of this is about exponents or palettes. It's the tax on shipping a format into a mature inference engine on consumer hardware — and it dwarfed the algorithm that started the whole thing. SCLP does what it set out to do: on the models we tuned for, 4-bit weights that match or beat the standard integer quant on quality per byte for decode-bound workloads, running coherently on a sub-$1,000 GPU. Getting there was mostly the work in this post, not the work in the first two. That's usually how it goes.

Postscript: trying it on Gemma 4 12B

While we were writing this post, Google released Gemma 4 12B, so we ran SCLP on it. The hypothesis was modest: perhaps we could compress it a little further than the standard quants, but because a 12B model isn't memory-constrained on a 24 GB card — it fits comfortably even at 8-bit — the bandwidth advantage SCLP relies on probably wouldn't materialize.

It didn't, and the result illustrates this post's running theme cleanly enough to be worth showing in full. Every configuration below was built with an imatrix from in-domain traces; perplexity is measured on a held-out out-of-domain set (lower is better, and as Post 2 noted these OOD numbers run roughly 50× inflated — read the rankings, not the absolutes). Throughput is llama-bench on the RX 7900 XTX, fully offloaded.

Two head-to-heads tell the story. At 8 bits, Q8_0 beats SCLP8 on every axis: it is smaller (it quantizes the embeddings too, which SCLP keeps at BF16), matches perplexity, and generates 1.7× faster. At 4 bits, Q4_K_M is both smaller than SCLP4 and far more accurate — pure SCLP4 puts 4-bit precision on the attention projections, which collapses quality (the same failure mode from Post 2; the MIXED recipe, which protects attention with SCLP6, recovers most of it). SCLP did not even win on size: the imatrix sidecar plus native embeddings push every SCLP build above its standard-quant counterpart.

The reason is the subject of this entire post — measure, don't assume. SCLP's advantage is bandwidth: fewer bytes read per weight at generation time, which converts to speed only when bandwidth is the bottleneck. On a model that fits in VRAM with room to spare, the standard quants' INT8 dp4a GEMV simply runs faster than SCLP's decode-and-multiply kernel, and the extra arithmetic SCLP spends on palette lookups and sidecar corrections is pure overhead rather than a trade against saved memory traffic. The wins in Posts 1 and 2 came from models pushed against the VRAM ceiling — and, for the 4-bit quality-per-byte result, from a mixture-of-experts architecture. Neither condition holds here, and the numbers reflect that precisely.

That is not a disappointing result so much as a sharp one: it marks where SCLP belongs. For a model too large for your card, every byte trimmed from the weight read is bandwidth returned. When the model already fits, the standard quant is the better tool.

SCLP is open source. The reference implementation (Python + HIP kernels) is at github.com/KerchumA222/sclp. The llama.cpp fork with GPU inference is at github.com/KerchumA222/llama.cpp, branch sclp. Both target AMD RDNA3 (ROCm/HIP) — CUDA porting is straightforward but not yet done.

8 bits per weight is the easy version. Half the bytes, a comfortable palette, a sidecar so small (0.01%) you can ignore it. The real question — the one that decides whether this is a clever trick or an actual competitor to INT4 — is what happens when you keep going. 6 bits. 5. 4.

Everything that was comfortable at 8 bits breaks at 4. Here's the order in which it broke, and what we did about each failure.

The palette runs out of room

The byte budget at each tier:

At 8 bits you spend 4 on the palette index and 4 on sign+mantissa. At 4 bits you have to fit everything — index, sign, mantissa — into a single nibble: 2 bits of palette index, 1 sign, 1 mantissa bit. The palette drops from 16 entries to 4.

Four exponent values cannot cover 99.9% of a matrix the way sixteen did. Recall from Post 1 that a typical weight matrix uses 10–16 distinct exponents. Force that down to 4 and you've either thrown away the tails (huge sidecar) or quantized the magnitude itself. Both of those bills come due below.

Per-block scaling, and exactly where it breaks

SCLP6 and SCLP8 carry a per-block scale: one BF16 multiplier per 32 weights. Before encoding, each block is normalized by its max-abs value; the decoder multiplies it back. This costs +6.25% (SCLP8) / +8.3% (SCLP6) and it works — it pulls the weights in a block into a tighter magnitude range so the palette spends its entries where they matter. Gemma4-31B dense and 26B MoE are both coherent with per-block scaling on.

We assumed the same trick would help SCLP4 most of all, since its palette is the most starved. It did the opposite.

Normalizing a block by its max-abs concentrates every weight's exponent up near 126–127. For a 16-entry palette that's fine — you still resolve the spread. For a 4-entry palette it's fatal: all four entries collapse onto nearly the same exponent, and SCLP4 degenerates into roughly 2-bit scalar quantization. On instruction-tuned models the result is unmistakable. Gemma4-26B-IT, pure SCLP4 with per-block scaling, asked anything:

own own own own own own own own own own...

Mode collapse. And here's the part that cost us time: the GPU decode was byte-perfect. We verified it against the CPU reference across all 508 million weights — identical. The garbage wasn't a bug. It was the honest output of a 4-entry palette that scaling had crushed into 2 bits of magnitude resolution. (Llama-3-8B base, same config, doesn't collapse — it stays incoherent-but-diverse. Collapse is an IT-model failure mode, which is its own lesson for the smoke-testing post.)

Per-block palettes

The fix is to stop fighting the local exponent variation and start exploiting it. Instead of one global palette plus a normalizing scale, give each 256-weight block its own 4-entry k-means palette. No scale multiply at all. A block whose weights cluster around exponent 120 gets a palette centered there; its neighbor at 124 gets its own. The same 4 entries that were useless globally become well-placed locally.

The overhead is tiny — 4 palette bytes per 256 weights is 1.56% — and the quality difference is not subtle. Llama-3-8B SCLP4, wikitext perplexity:

Per-block palette is now the only SCLP4 mode. Per-block scaling, the thing that helps every other tier, is the thing you must not do here. SCLP5 (an extra mantissa bit over SCLP4) uses the same per-block-palette structure.

MoE: don't decode what you don't route

Gemma4-26B is a mixture of experts — 128 experts per layer, of which a handful are active per token. The naive integration decoded the entire expert tensor to BF16, then let the router pick. For 128 experts that's a 16× waste of decode work and about 1 GB of scratch VRAM per layer, every token.

So we wrote a fused MoE GEMV: one thread block per (row tile, active expert). It decodes only the routed experts' weights, inline, straight into the dot product — the full BF16 expert buffer never exists. With it, the mixed-precision MoE model runs at the same 55 t/s as the dense path; without it, you pay for 120 experts you never use. Prefill (many tokens) still uses the two-pass decode, since there the GEMM dominates anyway.

The imatrix trick that worked (and the one that didn't)

At 4 bits the sidecar — the verbatim-BF16 escape hatch for weights the palette can't represent — grows from 0.01% to several percent. That's now a real fraction of the file, which means which weights you rescue matters. An importance matrix (imatrix) tells you which weights see the most activation. The obvious move is to weight the palette's k-means by importance so the clustering favors important weights.

We tried that first. It regressed perplexity 5×. Importance-weighting drags palette entries toward high-activation weights and starves everything else, and "everything else" is still most of the matrix.

What worked was applying imatrix to sidecar selection only, never to the palette. Two tiers: a mandatory tier (any weight whose palette distance exceeds a threshold) plus a discretionary tier (the top budget fraction ranked by importance × distance). The palette stays a clean unweighted k-means; the imatrix only decides who gets promoted to lossless storage.

Sweeping the budget on Gemma4 mixed-precision shows a sharp diminishing-returns point — perplexity falls off a cliff as you go from 0 to 1%, then flattens out:

1% is the recommended default; 2% is within noise of it — a quality floor, not an improvement. One more trap, and here I'll separate what we measured from what we inherited: ideally you build the imatrix from the BF16 model, not from a Q5_K_M intermediate. An already-quantized model's own rounding noise pollutes the activation statistics the imatrix records, so the importance signal you extract is partly measuring the other quant's errors — which is why llama.cpp quant-makers calibrate against full precision. Where a published BF16-calibrated imatrix was available off the shelf we used one. But the Gemma4 numbers in this section came from a Q5_K_M-sourced imatrix — Gemma4's BF16 is 48 GB and won't fit a 24 GB card — so we never isolated the contamination cost in a controlled same-model run. Treat the size of that effect as community wisdom, not a number from this project. What we did measure is the calibration domain: building the imatrix from in-domain agentic traces instead of wikitext cut SCLP4 sidecar PPL by 32%.

Where 4 bits lands

The headline comparison is against Q4_K, llama.cpp's standard 4-bit integer quant. "MIXED" throughout is one recipe: SCLP6 on the attention projections and ffn_down, SCLP4 on the bulk ffn_gate/ffn_up, embeddings and output kept native — the precision-where-it-matters policy from earlier in this post. The rows differ only in how the SCLP4 portion builds its palette (global vs per-block) and whether imatrix sidecar is applied; the SCLP6 attention/ffn_down half is the same in all three. On held-out agentic traces (OOD perplexity, the number we actually trust):

Per-block-palette SCLP4 beats the Q4_K hybrid by 2.2×, at 0.9 GiB smaller. Global-palette SCLP4 with a generous imatrix budget wins on quality outright — but most of its extra 2 GiB is sidecar, so it's the choice only when size is unconstrained. Q4_K still wins one thing: prefill throughput, because it goes straight through rocBLAS's integer path while SCLP pays the two-pass decode tax (Post 3 has the gory details). For prefill-bound work — long-context RAG — Q4_K gate/up is the right call. For chat and agents, which are decode-bound, SCLP4 is smaller and more accurate.

One thing that surprised us: SCLP4 is the smallest tier but not the fastest at token generation. SCLP4 lands at 34 t/s, behind SCLP6's 38. Fewer bits per weight should mean less to read and faster decode — but an apples-to-apples per-byte comparison of the two decode kernels showed SCLP4's is ~1.7× slower per byte than SCLP6's. The 1-bit mantissa and per-block palette pack more logical work into each byte (more unpacking, more palette indexing per weight), and at these sizes the model already fits comfortably in VRAM, so the bandwidth saving doesn't dominate. Smaller isn't automatically faster — you have to measure the kernel, not count the bits.

One caveat on all of these numbers: wikitext perplexity is inflated roughly 50× out of domain. A healthy Gemma4-IT scores ~50–200 on OOD wikitext that would read as catastrophic if you assumed base-model scales. Use the rankings, not the absolutes — which is exactly the kind of measurement discipline the next post is about.

Where this leaves us

The compression story is, by this point, basically done: SCLP4 at 4 bits/weight, per-block palettes to keep the starved palette useful, a fused MoE GEMV so you only decode what you route, and imatrix-targeted sidecar to spend your lossless budget where it counts. It beats the standard 4-bit integer quant on quality per byte for decode-bound workloads.

What's left is everything underneath. The algorithm was about a week of prototyping. Making it run — correctly, fast, without wedging the GPU — took the next two, and almost none of it was about compression. That's the last post: the bugs, the dead ends, and the lessons from the bottom of the stack.

SCLP is open source. The reference implementation (Python + HIP kernels) is at github.com/KerchumA222/sclp. The llama.cpp fork with GPU inference is at github.com/KerchumA222/llama.cpp, branch sclp.

We spent some time looking at the other field instead.

The exponent is barely random

A BF16 float has three fields: 1 sign bit, 8 exponent bits, 7 mantissa bits. The sign is basically a coin flip, so there's nothing to compress there. The mantissa encodes fine-grained precision within a magnitude range, and it's close to random too. The exponent is the odd one out — it determines the order of magnitude, and that's where the structure hides.

So we dumped the exponent distribution of every weight matrix in Llama-3-8B. A typical attention projection has 4096 × 4096 = 16 million weights, each carrying an 8-bit exponent that could in principle take any of 256 values. In practice, 10 to 16 exponent values cover 99.9%+ of every matrix — 14 on OPT-125m's MLP layers, as few as 10 on some attention projections. Calling the distribution "skewed" undersells it; it behaves more like a lookup table.

It's not a quirk of one model either. We checked OPT-125M, Llama-3-8B, and Gemma4-26B and saw the same pattern each time. Training concentrates weights into a narrow magnitude band, and the exponent is what encodes magnitude. The remaining ~0.01% of weights with rare exponents fall into two camps: near-zero (harmless to approximate) or extreme outliers (critical to preserve exactly).

The palette idea

If only 16 exponents matter, you don't need 8 bits to store them — you need 4. Build a palette of the dominant exponent values, store a 4-bit index per weight, and you've cut the exponent from 8 bits to 4 with zero loss for 99.9% of weights.

That leaves 4 bits for everything else, so we pack the sign (1 bit) plus the top 3 mantissa bits (3 bits) into the remaining nibble. One byte per weight.

BF16 weight (16 bits):  [sign:1][exponent:8][mantissa:7]
SCLP8 code  (8 bits):   [palette_idx:4][sign:1][mantissa_top3:3]

Dropping the bottom 4 mantissa bits sounds like it ought to hurt quality, but the effect turned out to be surprisingly mild. It behaves like a form of weight regularization, gently pulling values toward their magnitude's midpoint. On Llama-3-8B the truncation actually lowered perplexity, from 10.59 (full BF16) to 9.87 — which wasn't the result we'd bet on going in.

What about the outliers?

The 0.01% of weights whose exponents fall outside the top-16 palette could just be mapped to the nearest palette entry, but that introduces a magnitude error — potentially 2x or 4x off for a 1-2 step exponent mismatch. Most of those weights are tiny enough that it wouldn't matter, but a handful carry disproportionate weight for model quality.

So we don't approximate them at all. Any weight whose exponent isn't in the palette gets stored verbatim — full 16-bit BF16 — in a sidecar section appended to the compressed blob, and a fixup kernel scatter-writes them back into the output at decode time. The net effect is functionally lossless for every weight that matters.

The sidecar typically adds 0.01-0.03% overhead, and you can measure it per tensor to decide whether it's worth the bytes.

The Python prototype

The first implementation was pure NumPy. Encode takes a uint16 array of raw BF16 bit patterns — we never convert to float — builds the palette with k-means on the exponent histogram, and packs each weight into one byte. Decode runs the same steps in reverse. The whole thing is about 200 lines and finishes in a few seconds on a CPU.

# The core encoding loop (vectorized)
palette_idx = exp_to_idx[exponents]     # 4-bit palette lookup
smn = (signs << 3) | mantissa_top3      # 4-bit sign+mantissa
ws_stream = (palette_idx << 4) | smn    # pack into one byte

On top of that we wrote a .sclp file format, a round-trip test suite, and a converter that patches individual tensors into existing GGUF files — all before touching a single GPU kernel. Getting the algorithm right on CPU first, where you can print every intermediate value, saved weeks of debugging down the line.

Getting it onto the GPU

A compression scheme that can't keep up at inference speed is really just a file format. So the target was llama.cpp on an RX 7900 XTX (RDNA3, 24 GB VRAM, HIP/ROCm).

Step 1: Register the type

llama.cpp's type system is a set of enums and tables. We added GGML_TYPE_SCLP8 = 47 to ggml.h, registered block size and type size in ggml.c, told the CUDA/HIP backend it can handle SCLP in supports_op, and taught the GGUF loader to infer compressed blob sizes from tensor offsets (since SCLP blobs are smaller than ggml_nbytes would predict).

This plumbing took longer than the compression algorithm did. Every path that touches tensor metadata — allocation, loading, mmap, graph planning — has to agree on what "size" means for a compact blob.

Step 2: Two-pass decode

The safe first approach is to decode the entire compressed tensor to BF16 in a GPU kernel, then hand the BF16 buffer to rocBLAS for the matrix multiply. That's two passes over the data, but the decode kernel is simple and the GEMM is battle-tested.

The decode kernel is self-contained: thread 0 reads the blob header (palette size, palette bytes) and broadcasts the palette to shared memory, then all threads decode 8 weights each via coalesced uint64 loads. A second kernel scatter-writes the sidecar values. Both are HIP-graph-safe, with no host-device reads.

This worked on the first try — Llama-3-8B loaded and generated coherent text. But two-pass leaves the headline win on the table: at token-generation time you still read 16 bits per weight (the decoded BF16 buffer) on top of the compressed blob. The bandwidth savings only show up if you never write that BF16 intermediate in the first place.

Step 3: The fused GEMV

Two-pass is fine for prefill, where a large matrix multiply means the GEMM dominates. But token generation computes one output row at a time, so the matrix multiply collapses to a dot product — GEMV, not GEMM — and the bottleneck becomes memory bandwidth: how fast can you read the weight matrix?

BF16 reads 16 bits per weight; SCLP reads 8. Decode inline — compute the float value from the palette index and mantissa bits during the dot-product accumulation, without ever writing a BF16 intermediate — and you halve the memory traffic.

That's the fused GEMV kernel. One warp per output row. Each thread reads 8 bytes of compressed weights, decodes them to floats using the palette in shared memory, multiplies by the activation vector (also broadcast to shared memory), and accumulates. The sidecar correction is folded in: because the encoder sorts sidecar entries by weight index, each row can binary-search its contiguous range and apply corrections inline — no atomics, no second kernel.

The result is 43 t/s at 8 bits/weight vs 52 t/s at 16 bits/weight on Llama-3-8B. You read half the bytes but spend more ALU per byte on palette lookup, bit manipulation, and sidecar correction. The trade is a ~7 GB footprint reduction at a modest speed cost. Later optimization — K-tiling for occupancy, compilation unit splitting for register pressure — pushed throughput further on the smaller SCLP types.

Note: an earlier SCLP8 build without sidecar correction showed 66 t/s. Adding folded sidecar (binary search + correction per row) was necessary for quality but cost roughly a third of the throughput. The 43 t/s number reflects the current kernel.

The prefill gap

Prefill — processing the entire prompt in one batch — is where SCLP pays its tax. The two-pass path (decode blob → BF16 → rocBLAS GEMM) adds a full weight-matrix read before the GEMM, whereas Q8_0 goes directly through rocBLAS's INT8 path. SCLP8 prefill lands at ~2,650 t/s, against ~3,430 for Q8_0 and ~12,000 for BF16.

We did try fusing the decode into the GEMM, a combined decode+matmul kernel for M > 1. It worked for dense models, but on MoE models the F32 accumulation order between tensor-core tiles and scalar sequential math diverges at ~1e-3 per multiplication. Through 26 MoE layers those differences compound into catastrophic perplexity. Closing the gap would mean exactly replicating rocBLAS's tile accumulation order — doable, but not worth the engineering for a 20% prefill win.

For now the two-pass path is the right default, since prefill isn't the bottleneck for chat and agentic workloads anyway.

Where this leaves us

SCLP8 on Llama-3-8B comes in at 7.9 GB (vs 15.0 GB BF16), PPL 9.87 (vs 10.59), and 43 t/s token generation on an RX 7900 XTX. That's 2x compression on the weight streams, and the model scores slightly better than the original by perplexity.

But 8 bits per weight is the easy version. The more interesting question is what happens when you push to 6 bits, then 4 — when the palette shrinks from 16 entries to 4, per-block scaling degenerates, and the sidecar population jumps from 0.01% to 5%. That's where mixed precision, per-block palettes, and the imatrix trick that actually worked come in. Next time.

SCLP is open source. The reference implementation (Python + HIP kernels) is at github.com/KerchumA222/sclp. The llama.cpp fork with GPU inference is at github.com/KerchumA222/llama.cpp, branch sclp. Both target AMD RDNA3 (ROCm/HIP) — CUDA porting is straightforward but not yet done.