Inkling-2-Mini in 5.4 GB Memory
github.comThe GGUF route is in the repo (gguf_compiler.py emits plans over llama.cpp's own tensor types and the docs say the result runs in llama.cpp as-is), so porting these plans is clearly intended, but right now a 4090 gets you the Qwen files to read about rather than run.
Please [review the commenting guidelines](https://news.ycombinator.com/newsguidelines.html)
> The model is distributed as a single checkpoint of about 20GB, in mixed NVFP4 and MXFP8 precisions.
What's the point in quantizing a QAT model? Just for social media karma?
Quantization to a non-corresponding format promises none of those accommodations.
> Most of Inkling 2 Mini’s weights were trained and are deployed natively in NVFP4, with the last four layers in MXFP8.
I didn't find any non-sloppy factual report, but this channel has a good video on it: https://www.youtube.com/watch?v=OdMpNiByntM
There's some things like knowledge breadth that really need a huge model but those things can be compensated for with web lookups (which is preferable for source validation and currency anyway)
I'm not sure for. It might just be you can get these nice local models because of distillation from a much more powerful model.
If that holds, I'll be able to do 99% of my work with local models around March 2027.
My goal is not to be 100% local but to have all my personal data on local models only. Claude I just use to build stuff so it doesn't contain much data about me.
/igotthatreference.jpg
----
Gave it a quick browse to verify what my local clanker told me, it seemed to check out
I believe it is Dynamically allocating weights based on fisher information of conversation traces & For some K-quant look up table stuff it can store in an importance matrix to get better compression.
The results are complete nonsense and really don't back up any of the README claims. For Inkling-2-Mini; at 5.4 GB its KL is worse than Unsloth's UD-IQ1_S: 0.0895 vs 0.0808 on English/chat/code, and 0.287 vs 0.124 across 13 languages (e.g.: horrible). It then achieved better results by post-training the qaunt'd model on sample inputs (AFAIT).
According to anthropomorphic-clud their results for GSM8K & HumanEval are within the noise regime and vastly over stated.
Also, although not necessarily for this case, there are some inference engines made for just one model/model family, like antirez’ DS4.
This repo's README is a mess of different histories as the author bounces from one popular model to the next. It refers to vague things like "the 1.2GB file" that probably made sense in an early vibecoding pass, but are now lost after the repo has been pushed around to different popular models.
The truth is that random GitHub repos with messy READMEs almost never hold a real secret that the mainstream quantizations have missed. I would recommend sticking to mainstream quantizations like the official GGML release, Unsloth's quants, or the more well known community quantizers like bartowski or AesSedai. They typically follow an optimized size-quality tradeoff so you can find the one that fits your needs.
When you see a repo like this where everything is so obviously vibecoded that the README is a rambling mix of projects and claims, you should probably go looking for a more authoritative source. The local LLM communities get spammed with 10 of these types of projects per day but I have yet to see a real breakthrough come from any of the vibecoded ones. The real work is coming from people who study this and have compute to test at scale or work on implementing advanced techniques which require inference engine support, like trellis quants.