TL;DR
-
Infrastructure beat model launches: The week’s most useful work came from engine support for Kimi-K3, DeepSeek V4 Flash, Qwen3.5, GLM-5.2, Gemma4, and Ling 3.0 across vLLM, SGLang, llama.cpp, LMDeploy, and OpenVINO.1
-
Apple local inference moved fast: Ollama, Apple MLX, LocalAI, oMLX, Osaurus, and mlx-vlm all pushed MLX, Metal, KV-cache, or speculative-decode work aimed at making Mac inference less of a fallback path and more of a first-class serving target.
-
Blackwell kernel work spread through the stack: NVIDIA, FlashInfer, FlashAttention, Triton, TileLang, ROCm AITER, and SGLang all worked on FP8, NVFP4, MXFP4, MLA, MoE, and attention kernels for newer GPU families, with FlashInfer showing the most concentrated kernel velocity.2
-
Edge stacks converged on LLM workflows: Google LiteRT, OpenVINO, Qualcomm AI Hub, ExecuTorch, sherpa-onnx, MNN, and RunanywhereAI all shipped or merged work that treats speech, vision, text, and tool use as one mobile runtime problem.
-
Serving control planes got sharper: LiteLLM, ai-dynamo, Ray Serve, LocalAI, Triton Server, and Open WebUI focused on routing, compatibility, metrics, auth, Kubernetes behavior, and OpenAI-compatible edge cases, with LiteLLM turning Auto Router into a product surface.3
This Week in Inference
No confirmed major public open-source model launch drove the week. The repo activity tells a clearer story: inference teams spent the week catching up to the current model mix, especially Kimi-K3, DeepSeek V4 Flash, Qwen3.5, GLM-5.2, Ling 3.0, Gemma4, MiniMax-M3, and Qwen3-Omni, with SGLang, vLLM, llama.cpp, and LMDeploy doing much of the visible serving work.1
The optimization center of gravity stayed fixed on the same three levers: quantization, speculative decoding, and KV-cache management. Ollama used MLX MTP speculative decoding to speed Qwen3.5 on Apple GPUs, vLLM matured secondary KV-cache tiers, SGLang hardened session-aware radix caching, and FlashAttention advanced dynamic persistent scheduling for variable-length attention.7
Hardware work split across NVIDIA Blackwell, AMD MI-series GPUs, Apple Silicon, Intel GPU/NPU paths, Qualcomm Snapdragon NPUs, and browser runtimes. TensorRT shipped a major OSS release, OpenVINO shipped a coordinated release wave, Google LiteRT-LM added more GenAI runtime coverage, and Qualcomm GenieX expanded Snapdragon HTP support.11
Top Stories
vLLM and SGLang race to own Kimi and DeepSeek serving
vLLM made Kimi-K3 a headline model target while adding sequence-parallel speedups, shared-expert memory savings, and ROCm parity work around Kimi and Inkling paths through vLLM.1 SGLang pushed Kimi-K3, DeepSeek-V4, DSpark, XPU, Blackwell, NPU, and multimodal serving in the same week, which made SGLang one of the clearest indicators of how much production serving now depends on model-specific kernel and cache paths.4
The important shift sits below model loading. Both projects worked on KV behavior, CUDA graph safety, speculative decoding, MoE routing, and hardware-specific attention paths, because long-context MoE models now break naive serving stacks before they hit model quality limits.
NVIDIA, FlashInfer, FlashAttention, and Triton push the Blackwell kernel layer forward
NVIDIA shipped TensorRT OSS updates while TensorRT-LLM and CUTLASS advanced CuTe DSL, MLA decode, Kimi kernels, and Blackwell-era infrastructure through TensorRT-LLM.15 FlashInfer had one of the week’s densest kernel runs, adding DeepSeek sparse MLA, CAKE/KDA recurrent kernels, MoE fixes, FP8 and NVFP4 work, and SM100-family support through FlashInfer.2
FlashAttention also moved FA4 toward production use with SM100 variable-length scheduling and learnable-sink backward support through FlashAttention.10 Triton reinforced the compiler side with Cluster Launch Control, warp-specialization, AMD CDNA5 naming, sanitizer improvements, and Blackwell FP4 fixes through Triton.16
Apple local inference stopped looking like a side path
Ollama made Qwen3.5 faster on Apple GPUs with MLX MTP speculative decoding and supporting Metal kernels through Ollama.7 Apple’s own MLX stack worked on quantized matmul, Metal correctness, distributed JACCL, and Swift KV-cache behavior through MLX.17
LocalAI added Darwin and Metal work around vllm-cpp and MLX provider paths through LocalAI.18 oMLX and Osaurus pushed DeepSeek V4 Flash, Ling 3.0, LFM2.5, disk-prefix cache, and long-context safety for native macOS agent workflows through oMLX and Osaurus.19
Edge runtimes are turning into full GenAI platforms
Google’s edge stack linked LiteRT, LiteRT-LM, litert-torch, Model Explorer, MediaPipe, and samples around LLM, ASR, TTS, NPU, and compiled-model workflows through LiteRT-LM.13 OpenVINO shipped a release train across runtime, GenAI, and NNCF with Qwen3, Gemma, LFM, MiniCPM, Kokoro, low-bit compression, and GPU/NPU backend fixes through OpenVINO.12
Qualcomm expanded AI Hub apps and GenieX around Snapdragon NPU deployment, model metadata, QAIRT, VLM fixes, speculative decoding, and raw logits through GenieX.21 ExecuTorch added quantized MoE, MLX KV-cache work, Qualcomm QNN, Arm, Vulkan, XNNPACK, and wheel-distributed runtime planning through ExecuTorch.22
The serving layer got more productized
LiteLLM made Auto Router a first-class UI and analytics surface with create, edit, test, savings, and benchmark flows through LiteLLM.3 ai-dynamo tightened OpenAI and Anthropic compatibility, Kubernetes deployment APIs, multi-engine routing, TLS, tracing, and vLLM, TensorRT-LLM, and SGLang backend behavior through Dynamo.23
Ray Serve added KV-cache offload and atomic KV-aware routing behavior through Ray.24 Open WebUI focused on backend overhead, async RAG, Redis reads, access-control indexes, and JSON serialization hot paths through Open WebUI.25
Deeper Dive
Everything below is for readers who want the full picture. Feel free to scroll.
Code Changes by Category
Cloud & Datacenter Serving
vLLM, SGLang, TensorRT-LLM, LMDeploy, LightLLM, Dynamo, and Ray all worked on the same production problem: long-context MoE serving under mixed hardware. vLLM added secondary KV-cache tier work, SGLang hardened radix and session cache behavior, and TensorRT-LLM added disaggregated-serving security and runtime fixes.8
LMDeploy split its OpenAI-compatible server into endpoint modules and added GLM-5.2 plus Interns2 Mobius support through LMDeploy.27 LightLLM focused on PD serving, cache accounting, CPU KV cache NUMA interleaving, and FP8 W8A8 aliases through LightLLM.28
Dynamo pushed compatibility across OpenAI, Anthropic, vLLM, TensorRT-LLM, SGLang, Kubernetes DGD APIs, routing, scheduling, TLS, and tracing through Dynamo.29 Ray added Apple MPS resource support, Serve auth fixes, and KV-aware routing changes through Ray.30
Local LLM Runtimes
llama.cpp had a major DeepSeek V4, DSpark, MTP, backend, and multimodal week through llama.cpp.5 whisper.cpp absorbed a large ggml sync and shipped audio, Parakeet, VAD, Android, and backend fixes through whisper.cpp.31
Ollama added MLX MTP speculative decoding for Qwen3.5, OpenAI-compatible streaming fixes, MLX cache rewind fixes, and cloud-tag suggestions through Ollama.32 LocalAI shipped a large backend refresh around vllm.cpp, llama.cpp, whisper.cpp, stable-diffusion.cpp, audio.cpp, ds4, UI, gallery, signing, and endpoint expansion through LocalAI.33
llamafile shipped upstream-sync improvements and model support refreshes, then immediately received a ZIP bounds-validation security report and fix proposal through llamafile.34 exllamav2 and text-generation-webui had no landed code, but both saw community or docs activity through exllamav2 and text-generation-webui.35
Apple Silicon & MLX Ecosystem
Apple’s MLX core worked on quantized matmul, Metal attention and convolution behavior, distributed JACCL, API errors, and numerical cleanup through MLX.17 mlx-lm and mlx-swift-lm focused on long-prompt Metal resource limits, seed behavior, typed KV-cache configuration, MTP sliding windows, Qwen VLM support, and Linux build hardening through mlx-swift-lm.37
Blaizzy’s mlx-vlm added DFlash fixes, Inkling multimodal support, embeddings APIs, speculative acceptance stats, Kimi, Qwen, Gemma, LFM, and Molmo fixes through mlx-vlm.38 mlx-audio added ArkTTS, Chatterbox updates, loudness fixes, resampling fixes, and Whisper compatibility through mlx-audio.39
oMLX ran a six-release stabilization cycle around DeepSeek V4 Flash, Ling 3.0 Flash, cache integrity, SpecPrefill, Inkling, MTP, and long-context safety through oMLX.40 Osaurus added DeepSeek V4 Flash, LFM2.5, vMLX disk-cache diagnostics, WhatsApp integration, agent execution fixes, and macOS UX polish through Osaurus.41
Mobile & Edge Frameworks
Google’s LiteRT stack moved across runtime, LiteRT-LM, litert-torch, MediaPipe, samples, and Model Explorer through LiteRT-LM.13 OpenVINO shipped GenAI coverage, NNCF sub-4-bit compression, GPU MXFP4 and FP4 dynamic quantization, Qwen3 Omni and ASR work, and backend correctness fixes through OpenVINO.12
Qualcomm’s AI Hub apps, AI Hub models, and GenieX moved Snapdragon app catalog, QAIRT, model metadata, quantization, speculative decoding, VLM tokenization, and HTP support through GenieX.14 RunanywhereAI added NeuRT for Apple Neural Engine, Hexagon NPU coverage for LFM2.5, v4 SDK conformance, RAG ABI changes, and hosted macOS Connect workflows through runanywhere-sdks.42
MNN improved Metal LLM acceleration, 2-bit and 3-bit Metal decode, QNN, RISC-V, Hexagon, OpenCL, and crash fixes through MNN.43 sherpa-onnx restored Swift Package Manager support, moved Apple packaging to current ONNX Runtime artifacts, expanded Flutter, and fixed a Dart diarization leak through sherpa-onnx.44
Compilers, Runtimes & Graph Engines
TVM pushed TIRx typed-buffer migration, CUDA tcgen05, FlashMLA lowering, Blackwell work, Relax ONNX fixes, and tvm-ffi ABI repair through TVM.45 OpenXLA added CUDA and PTX updates, cuDNN fusion fixes, FFI Record APIs, IFRT serialization, PjRt GPU transfer work, and HLO correctness fixes through XLA.46
Triton advanced NVIDIA CLC, warp specialization, AMD CDNA5, sanitizer tooling, MLIR fixes, Blackwell FP4, and Gluon changes through Triton.16 TileLang shipped multi-backend compiler work, SM120 NVF4 block-scale MMA, Metal cooperative-tensor GEMM, TMEM layouts, ROCm CI, and analyzer fixes through TileLang.47
ONNX and ONNX Runtime tightened model packaging, WebGPU, WebNN, CUDA LLM kernels, quantized KV-cache attention, QMoE, external-data safety, and Olive package generation through ONNX Runtime.48 TensorFlow and JAX both advanced XLA, Pallas, Mosaic, IFRT, PJRT, GPU codegen, and edge runtime performance through JAX and TensorFlow.49
Models, Quantization & Optimization
Hugging Face Transformers added attention dispatch, cache/export fixes, Qwen Omni batching, tokenizer fixes, torchao, compressed-tensors, AMD Quark, XPU, and DeepGEMM/FP8 updates through Transformers.51 Diffusers added MiniMax-H3 modular text-to-video and audio support plus SDNQ quantization through Diffusers.52
ROCm AITER and ATOM coordinated on DeepSeek-V4, Kimi-K3, GLM-5.2, Qwen3, Gemma, MiniMax-M3, MLA, MoE, FP4 KV cache, and long-context decode through AITER.53 FlashAttention added FA4 SM100 scheduling, learnable-sink backward, sparse backward correctness fixes, and ROCm CK varlen binding repair through FlashAttention.54
Candle tightened tensor semantics, zero-sized matmul, where_cond, and Qwen3 batched masks through Candle.55 Neural Compressor advanced JAX per-channel quantization and dependency hardening through neural-compressor.56
Other Notable Changes
LiteLLM built Auto Router into a UI and analytics feature, then added provider, MCP, budget, spend, and signed Docker image fixes through LiteLLM.3 Open WebUI improved logging, async RAG, Redis model-pool reads, access-control indexes, JSON handling, and chat UI polish through Open WebUI.57
BentoML saw no merged code, but contributors proposed self-managed Kubernetes deployment skills and S3 observability through BentoML.58 Exo saw no merged code, but contributors proposed MLX Ring Attention, topology-aware placement, Docker packaging, and manual master promotion through exo.59
Community Pulse
Maintenance concern surfaced in projects with low merge activity but high user interest. Exo users asked whether maintainers still watch the repo through exo, and text-generation-webui users asked whether the project remains active through text-generation-webui.60
AI-assisted review became normal in several repos. PyTorch and ExecuTorch used Claude and Codex-style review on MLX, Qualcomm, Arm, SDPA, Dynamo, and Inductor work through ExecuTorch, while Triton, ncnn, Uzu, and AMDMIGraphX also showed automated review flows through Triton.61
User reports clustered around the same failure modes: KV corruption, long-context hangs, model-specific quantization breaks, tool-call formatting, hardware-specific kernel gaps, and installation or package ABI problems. SGLang tracked DSpark illegal memory and long-context hangs through SGLang, and vLLM tracked DeepSeek, Kimi, Gemma, XPU, ROCm, and FlashInfer regressions through vLLM.63
Community Debates
TensorFlow and tflite-micro had the sharpest security-scope debate. Contributors proposed overflow, index, parser, and model-validation hardening, while maintainers pushed some work back based on trusted-model assumptions or LiteRT sequencing through TensorFlow and tflite-micro.65
OpenVINO saw a strategic GPU runtime debate. One proposal argued for replacing Intel GPU plugin runtimes with Vulkan, while maintainers challenged performance evidence, deployment impact, and whether Vulkan should live as a separate plugin through OpenVINO.67
oMLX rejected a faster DSpark path after target-only and DSpark greedy outputs diverged, which showed the project’s bias toward correctness over speed for local agent use through oMLX.68 AITER maintainers rejected a DeepSeek-R1 FP8 exact-shape GEMM proposal despite claimed speedups because they wanted stronger kernel-development discipline and alignment with existing Triton patterns through AITER.69
Open WebUI debated OAuth account merge safety. A contributor proposed rejecting account merges when an identity provider reports an unverified email, and maintainers pushed toward an opt-in flag to avoid breaking documented identity-provider behavior through Open WebUI.70
Worth Watching
Kimi-K3 support now spans vLLM, SGLang, FlashInfer, TensorRT-LLM, ROCm ATOM, mlx-vlm, vllm-mlx, and LocalAI, so next week’s useful signal will come from correctness and benchmark reports rather than support checkboxes through vLLM.1 DeepSeek V4 Flash remains the same kind of stress test for local runtimes, serving engines, and GPU kernels through llama.cpp.5
Apple Silicon inference is moving from “runs locally” to “has its own stack”: MLX, Ollama, LocalAI, oMLX, Osaurus, RunanywhereAI, and Apple Core AI all worked on cache, Metal, ANE, MLX, and Swift runtime behavior through MLX.17 Edge runtimes now compete on complete app workflows, since LiteRT, OpenVINO, Qualcomm, ExecuTorch, MNN, sherpa-onnx, and RunanywhereAI all added speech, vision, model-package, and NPU support through LiteRT-LM.13
MoE caching is becoming a local-runtime problem. llama.cpp, LightLLM, vLLM, SGLang, TensorRT-LLM, ROCm AITER, and FlashInfer all touched MoE routing, hot experts, expert parallelism, or quantized MoE kernels through llama.cpp.71
Major Releases
Version numbers and release-note links live here as the reference list.
Ollama shipped v0.32.6, focused on Qwen3.5 MLX acceleration, OpenAI-compatible chat streaming, correct finish_reason handling for truncation, and cloud-model discovery. The most impactful change was MLX MTP speculative decoding for Apple GPUs..32
ggml shipped a dense llama.cpp build run plus whisper.cpp v1.9.2. llama.cpp releases focused on DeepSeek V4, DSpark, GLM-4.7-Flash MTP, Qwen3-TTS, metrics, and Windows hardening, while whisper.cpp packaged a large ggml sync and audio/runtime fixes. whisper.cpp v1.9.2.31
Google AI Edge shipped AI Edge Gallery 1.0.17, LiteRT-LM v0.15.0, and litert-torch v0.9.3. The release theme was GenAI runtime coverage across Apple backend integration, Gemma4 text and multimodal execution, ASR export, NPU tooling, and app-facing UX. LiteRT-LM v0.15.0.13
BerriAI shipped a fast LiteLLM release train from v1.93.1 through v1.97.0-dev.1. The releases centered on signed Docker image verification while the codebase pushed Auto Router UI, savings analytics, provider fixes, budgets, MCP, and CI hardening. Latest prerelease.72
LocalAI shipped v4.8.0 after a large release cycle. The release highlighted vllm.cpp development builds, 3D generation, audio.cpp, variant-aware gallery installs, backend refreshes, distributed-mode reliability, signing, and UI work..33
NVIDIA shipped TensorRT v11.2 and TensorRT-LLM v1.3.0rc23. TensorRT added plugin migration samples, FFTPlugin, ONNX parser support, and build packaging refreshes, while TensorRT-LLM documented known issues around DeepSeek, GB300, Qwen MoE, and Gemma FP8 scenarios. TensorRT v11.2.11
Microsoft shipped ONNX Runtime WebGPU Plugin EP v0.2.1. The release focused on attention-heavy browser LLM performance, including FlashAttention decode fusion, generalized prefill shared-memory paths, dynamic NVIDIA max_k_step, and QKV bias support..73
OpenVINO Toolkit shipped OpenVINO 2026.3.0, OpenVINO GenAI 2026.3.0.0, and NNCF v3.3.0. The combined release wave focused on GenAI model coverage, CPU/GPU/NPU backend hardening, INT2/INT3 and sub-4-bit compression, Qwen3-family workflows, and low-bit GPU quantization. OpenVINO 2026.3.0.12
Apache TVM shipped TVM v0.26.0.rc0 plus tvm-ffi v0.1.13 and two post-releases. The release train centered on TIRx typed buffers, CUDA tcgen05 and FlashMLA, Relax fixes, tvm-ffi ABI stability, and WebGPU/WASM work. TVM v0.26.0.rc0.45
Hugging Face shipped optimum-intel v2.1.0. The release added OpenVINO model support for Gemma 3n, Qwen3-Omni-MOE, SmolLM3, Qwen3-VL-Embedding, and Gemma 4 Unified..74
FlashInfer shipped v0.6.16, v0.6.16.post1, v0.6.17rc1, and nightlies. The release theme was MegaMoE, Blackwell sparse attention, FP8/NVFP4/W4A16 MoE, confidential distributed serving, and ABI fixes around tvm-ffi..75
FlashAttention shipped fa4-v4.0.0.beta25. The prerelease packaged SM100 varlen dynamic persistent scheduling, ROCm CK varlen binding repair, FA4 cleanup, and AI debugging methodology docs..54
LMDeploy shipped v0.15.0. The release emphasized long-context and MTP prefix-cache hits, guided decoding for speculative decoding, and TurboMind scheduler, allocator, and object-cache integration..76
LightLLM shipped v1.2.0. The release packaged RL serving with verl, disaggregated ViT serving, hybrid cache for linear-attention models, cache-aware PD, broader MTP speculative decoding, multimodal updates, and distributed inference fixes..77
Qualcomm shipped ai-hub-apps v0.34.0 and GenieX v0.3.18. The app release added Snapdragon-focused samples such as SAM3 and YamNet, while GenieX added Hexagon HTP support for Gemma4 QAIRT inference and a signed HTP catalog for llama.cpp on Windows on Snapdragon. GenieX v0.3.18.14
AMD Ryzen AI shipped Lemonade SDK v11.5.2. The release focused on HTTPS tray support, repeatable llama.cpp arguments, benchmark hardware metadata, device/backend error clarity, and Windows MSI fixes..78
ROCm shipped AITER v0.1.19.post1 and v0.1.19.post2. Both releases targeted decode MLA v3 long-context support, while the repo’s code activity covered DeepSeek, Kimi, GLM, Qwen, attention, MoE, GEMM, and quantization kernels..79
Blaizzy shipped mlx-audio v0.4.7 plus mlx-vlm v0.6.9 and v0.6.10. The releases focused on audio server stability, model-kind registry support, Kimi K3, Gemma 4, LFM2, agent skills, BPE flush fixes, and VLM prompt media handling. mlx-vlm v0.6.10.80
jundot shipped oMLX v0.5.4rc1 through v0.5.7. The cycle focused on DeepSeek V4 Flash, Inkling, Step-3.7-Flash MTP, Ling 3.0 Flash, SpecPrefill, cache integrity, long-context fallback safety, and macOS idle behavior..40
Osaurus shipped 0.22.13 through 0.22.17 plus a WhatsApp helper prerelease. The release run focused on DeepSeek V4 Flash, vMLX cache work, LFM2.5, WhatsApp channel support, custom agents, macOS UX, chat import, and cloud image/video generation..41
k2-fsa shipped the sherpa-onnx xcframework release. The release provided prebuilt Apple XCFramework artifacts that pair with the week’s Swift Package Manager repair, ONNX Runtime alignment, Flutter expansion, and Apple CI work. Release page.81
Mozilla AI shipped llamafile v0.10.5. The release focused on cleaner llama.cpp upstream syncs, agent/plugin improvements, Ternary Bonsai and Laguna-S support, transcribefile artifacts, CLI docs, and Vulkan documentation fixes..82
TileLang shipped v0.1.13. The release covered multi-backend dialects, SM120 NVF4 block-scale MMA, Metal cooperative-tensor GEMM, arbitrary TMEM layouts, source-span injection, and correctness fixes..47