Hacker NewsHN submitter: k__
Original published:

Inkling-2-Mini in 5.4 GB Memory

Inkling-2-Mini in 5.4 GB Memory

An article about a quantization compiler that achieves 12x lower KL divergence than Q4_K_M at the same file size.

Key points

  • •It delivers 12 times lower KL divergence compared to Q4_K_M while maintaining the same file size.
  • •The Inkling-2-Mini model can run within a 5.4 GB memory environment.
  • •It is a quantization compiler related to the ShinMegamiBoson/lookatmyquant project.
#LLM / Model Trends#Semiconductors & Infrastructure#Inkling 2 Mini#ShinMegamiBoson/lookatmyquant
See more