shoehorn fit unsloth/Qwen3-4B-GGUF --serve downloads a source model, measures the memory available on your machine, and builds a GGUF that fits that budget. Shoehorn LLM quantization differs from a preset such as Q4_K_M because it can assign a different precision to each tensor. The result aims to use nearly every available byte while preserving more important weights at higher precision.
Shoehorn reached Hacker News this week after releasing version 0.3.0 on August 19. The Rust project is young, but the problem it addresses is familiar: a downloaded quantization leaves VRAM unused, or misses the limit once context and inference buffers are included.
Our position is that Shoehorn is valuable as a packaging tool for known hardware. It is not a shortcut around model evaluation.
What is Shoehorn LLM quantization?
Quantization stores model weights with fewer bits. A BF16 model uses 16 bits for each weight. Common GGUF releases reduce that to roughly four, five, or six bits so the model needs less memory and runs on consumer hardware.
GGUF is a file format used by llama.cpp and compatible runtimes. It stores the weights plus metadata such as layer counts, attention dimensions, and tokenizer information. Shoehorn reads a BF16, F16, or F32 GGUF and writes a standard GGUF v3 file that llama.cpp can serve.
Most published GGUF files use one preset recipe. Q4_K_M, for example, applies a familiar mix chosen for a general size and quality target. The file size is fixed before your context length or GPU is considered.
Shoehorn reverses that decision. You provide the available memory and desired context. It subtracts the estimated inference overhead, then chooses one quantization format for each eligible tensor so the total stays under the remaining weight budget.
This is mixed-precision quantization. Important tensors may receive more bits while less sensitive tensors receive fewer. Shoehorn supports formats ranging from IQ2_XXS through Q8_0 and F16, subject to tensor shape and guardrails.
Install Shoehorn and inspect the budget
The simplest installation on Apple Silicon uses Homebrew:
brew install notactuallytreyanastasio/shoehorn/shoehornThe package installs llama.cpp as the inference backend. Shoehorn also publishes Linux and Windows binaries, while its automatic Metal memory probe is specifically designed for Apple Silicon. On other targets, set an explicit budget instead of assuming system RAM equals usable GPU memory.
Check what Shoehorn sees:
shoehorn vramOn a 24 GB M4 Pro, the project shows a 17.76 GiB recommended Metal working set. macOS reserves part of unified memory, so a model sized for all 24 GB would not be a valid plan.
Before writing a large output file, preview the allocation:
shoehorn plan \
-m Qwen3-4B-BF16.gguf \
-i qwen3.imatrix \
--ctx 8192 \
--budget 16GiBplan prints the selected type for every tensor, the size by type, and the remaining slack. It runs the solver without creating the final GGUF, which makes it the right first command for comparing context and memory choices.
Then create the file:
shoehorn quantize \
-m Qwen3-4B-BF16.gguf \
-i qwen3.imatrix \
--ctx 8192 \
--budget 16GiB \
-o qwen3-4b-fit.ggufServe it through llama.cpp:
shoehorn run -m qwen3-4b-fit.gguf --ctx 8192Everything after -- is passed to llama-server, so you can set its port or API key there. Shoehorn also offers shoehorn ui for a local browser interface.
If your product team needs to benchmark a local model against a hosted API on the hardware you will actually deploy, our generative AI development work can turn that comparison into a reproducible evaluation rather than a one-machine demo.
How Shoehorn calculates the fit
The model file is only one part of inference memory. The key-value cache grows with context length. Compute buffers depend on batch shape, vocabulary size, embedding size, and the llama.cpp graph. The operating system and runtime also need headroom.
Shoehorn expresses the budget as:
weight budget = usable VRAM - KV cache - compute estimate - reserveThe KV cache stores attention keys and values from previous tokens. Shoehorn derives its size from GGUF metadata, including the number of layers, requested context, key-value heads, and head dimensions. This term is deterministic for the selected cache type.
The compute buffer is an estimate. llama.cpp allocation can change with flash attention, batch configuration, and version. Shoehorn therefore subtracts a safety reserve, 512 MiB by default. Reducing that reserve without measuring real peak usage can produce a file that fits on paper and fails at load time.
You can choose a smaller KV representation:
shoehorn plan \
-m model-bf16.gguf \
--ctx 16384 \
--kv q8_0 \
--budget 12GiBAn q8_0 KV cache uses roughly half the memory of the default F16 cache, leaving more room for weights. That trade does not come for free. Cache quantization can affect output quality, and the effect depends on the model and workload.
Why the importance matrix matters
An importance matrix, usually called an imatrix, measures how strongly each input column activates across calibration text. A weight attached to frequently strong activations can cause more output error when rounded, so the solver should spend more bits there.
Shoehorn tests candidate formats by encoding and decoding sampled tensor rows. It scores the measured distortion with imatrix-derived weights. The solver then treats format selection as a multiple-choice knapsack problem: choose exactly one format per tensor while minimizing total weighted error under the byte limit.
It first uses Lagrangian relaxation, which places a price on each byte and lets tensors choose the best error-versus-size option. A greedy pass spends the remaining slack on the upgrades with the best error reduction per byte. The project reports budget utilization above 99.9 percent in practice.
Without an imatrix, Shoehorn falls back to activation-agnostic weighting and prints a warning. That output may still load, but the solver has less evidence about which weights deserve precision. We would not compare it with an imatrix-guided build as if the two had the same quality target.
If a model repository does not provide an imatrix, llama.cpp can generate one from calibration text:
llama-imatrix \
-m model-bf16.gguf \
-f calibration.txt \
-o model.imatrix \
-ngl 99Calibration data should resemble the prompts your application will send. A generic text corpus may underweight behavior that matters for code, multilingual output, or structured extraction.
What exact-fit quantization does not guarantee
Filling 99.9 percent of a byte budget is not the same as preserving 99.9 percent of model quality. The optimizer minimizes its measured distortion objective. Your product cares about task accuracy, latency, generation quality, and failure rate.
Run perplexity as a quick regression signal, then test application examples. For a coding assistant, measure patch acceptance and test pass rate. For extraction, compare field-level accuracy. For a local agent, include tool selection and long-context recall.
Shoehorn also needs the source GGUF and an output destination. A large BF16 model consumes substantial disk space even though the program memory-maps it instead of loading the whole file at once. The one-command fit path currently refuses split GGUF sources. Generating an imatrix locally may also require the unquantized model to fit well enough for calibration.
The automatic compute estimate is intentionally rough. A different llama.cpp build, driver, batch size, or flash-attention setting can move the real boundary. Keep the reserve until a load test proves you can safely reduce it.
Our related Muse Glimmer local-agent guide makes the same broader point: advertised parameter count does not determine whether a model is practical on a target device. Runtime, precision, context, and workload must be measured together.
When Shoehorn is worth using
Use Shoehorn when you control the deployment hardware and a preset leaves meaningful memory unused. It is particularly useful for a local appliance, developer workstation, or fixed GPU fleet where the same GGUF will be loaded repeatedly at a known context length.
It is not worth adding when a standard quant already fits with comfortable headroom and passes your evaluation. The custom artifact creates another build to store, identify, reproduce, and test. A few hundred megabytes of theoretical efficiency may not repay that operational cost.
Shoehorn is also the wrong answer when the model is fundamentally too large. The CLI reports when even the smallest supported mix exceeds the weight budget. At that point, choose a smaller model, reduce context, or use a hosted endpoint. Extreme two-bit quantization can make a model load while making its answers unusable.
FAQ
Does Shoehorn make any LLM fit in VRAM?
No. Shoehorn searches supported quantization formats and stays within the requested memory budget, but it cannot compress without limit. If the smallest eligible tensor mix plus the KV cache, compute estimate, and reserve exceeds available memory, use a smaller model or lower the context length.
Is Shoehorn better than Q4_K_M?
Shoehorn can use memory more precisely because it assigns formats per tensor for one hardware budget. That does not guarantee better results for every task. Compare the fitted model with a standard Q4_K_M file using the same prompts, context, sampling settings, and application-level evaluation.
Can Shoehorn quantize models for NVIDIA GPUs?
Shoehorn publishes Linux and Windows builds, and explicit --budget values let the solver target non-Apple hardware. Its automatic VRAM detection is documented around Apple Silicon, so verify the real free GPU memory yourself. The resulting GGUF is served through a compatible llama.cpp installation.
Build for the hardware you will ship
Shoehorn turns model packaging into an explicit optimization problem. That is useful. The production decision still requires a representative imatrix, a conservative runtime budget, and evidence that the fitted model handles your workload.
We build AI agents, generative AI applications, and full-stack SaaS products around measured model behavior. If you are choosing between a local GGUF, a dedicated GPU deployment, and a hosted API, book a call with Axentia. We can benchmark the options on your prompts before you commit the product architecture.