Janas: Run LLMs larger than RAM by streaming experts from SSD
AI summary
Editorial policyOverview of Janas, a tool that executes large LLMs exceeding RAM capacity via SSD streaming, achieving faster inference than llama.cpp.
Key points
- •Qwen3-Next-80B (48GB) was processed at 37 tok/s on a 32GB laptop, significantly outperforming llama.cpp's 2.8 tok/s.
- •All 13 models compared in Q4_K_M quantization format recorded faster read speeds than llama.cpp.
- •Specific performance figures were presented: 24 tok/s without drafting and 62 tok/s for reading.