ExLlamaV3

https://github.com/turboderp-org/exllamav3turboderp-org/exllamav3

EXL3フォーマットで動作

https://www.reddit.com/r/LocalLLaMA/comments/1uwylut/exllamav3_v100_major_performance_upgrades/ ExLlamaV3 v1.0.0 - Major Performance Upgrades

Removed flash-attention-2 and xformers dependencies

Extended tensor-parallel support to most models, including Gemma4

New attention kernel with online cache quantization, dual input for SWA layers and attention sinks; no more slowdown for KV quantization (can even speed up inference now)

Graph path for all attn/GDN modules

New conv1d kernel (removes support/need for causal_conv1d)

Greatly improved GEMM/GEMV performance on Ampere

New INT8 GEMV kernel

New MoE kernel ticket scheduler

Added GptOssForCausalLM

Added NemotronHForCausalLM

Many minor optimizations

Many more bugfixes

Many QoL improvements

Faster extension build with more compilation units