Inkling-2-Mini in 5.4 GB Memory
An article about a quantization compiler that achieves 12x lower KL divergence than Q4_K_M at the same file size.
Key points
- •It delivers 12 times lower KL divergence compared to Q4_K_M while maintaining the same file size.
- •The Inkling-2-Mini model can run within a 5.4 GB memory environment.
- •It is a quantization compiler related to the ShinMegamiBoson/lookatmyquant project.