TL;DR
- Long-context serving moved from feature to constraint: vLLM and SGLang pushed KV-cache, speculative decoding, and scheduler work aimed at DeepSeek, GLM, and million-token style workloads.1
- Local inference became more agentic: Ollama made its interactive agent the default CLI entry point, while LocalAI, Osaurus, and Open WebUI hardened tool, voice, and sandbox flows.2
- Kernel work centered on Blackwell and low precision: FlashInfer, CUTLASS, Triton, FlashAttention, and AITER all advanced FP8, NVFP4, MXFP4, sparse attention, or MoE paths.3
- Edge stacks caught up with model ambition: Google LiteRT, Qualcomm AI Hub, OpenVINO, ExecuTorch, ncnn, sherpa-onnx, and RunanywhereAI all moved on-device LLM, VLM, ASR, or NPU support forward.4
- The release firehose stayed open: llama.cpp, SGLang, vLLM, Ollama, LocalAI, CUTLASS, AITER, LiteLLM, Osaurus, and RunanywhereAI all shipped release trains aimed at real deployment pressure.5
This Week in Inference
OpenAI released GPT-5.6 as a tiered Sol, Terra, and Luna family with a roughly million-token context window, while Meta previewed Muse Spark with similar context scale and subagent-oriented use cases, according to the weekly model roundup shared in the briefing release roundup.6 xAI released Grok 4.5 with a reported trillion-plus parameter scale and lower per-token pricing claims, while OpenAI also pushed GPT-Live-1 and GPT-Live-1 mini for full-duplex voice inference weekly model summary.7 Thinking Machines’ Inkling also entered the week’s model conversation, and Hugging Face moved fast enough to land Inkling support in Transformers during the same cycle Transformers Inkling support.8
The technical response was clear: serving engines are being rebuilt around memory, not raw generation alone. SGLang made speculative decoding a central serving path, vLLM moved dense-model serving to a new default runner, and both projects spent the week on DeepSeek, GLM, MTP, MLA, KV-cache, and disaggregated-serving correctness SGLang release.9 The same pressure showed up lower in the stack: FlashInfer expanded CuTe DSL, NVFP4, sparse attention, and MoE kernels, while CUTLASS shipped new CuTe DSL and Blackwell/Hopper examples FlashInfer.3
The edge side did not sit out the week. Google pushed LiteRT and LiteRT-LM across GPU, NPU, WebGPU, structured output, and local LLM configuration paths LiteRT-LM.10 Qualcomm added Qwen3 and Qwen3-VL deployment support in AI Hub Models and GenieX, while OpenVINO improved quantized GQA KV-cache, GPU attention, NPUW, and Qwen3-VL embeddings Qualcomm AI Hub Models.11 Ollama also turned local inference into a product story by making its interactive agent the default CLI experience, while the market briefing’s funding note showed that local model distribution now has venture-scale backing Ollama release.12
Top Stories
vLLM turns the runner and KV path into the main event
vLLM made Model Runner V2 the default path for dense models and used the release cycle to push DeepSeek, GLM, MLA, ROCm, FlashInfer, KV-offload, speculative decoding, and Rust frontend work forward vLLM.13 The most important shift is architectural: vLLM is treating the scheduler, cache layout, connector layer, and tool parsing as core serving infrastructure rather than secondary plumbing vLLM roadmap issue.14
SGLang makes speculative decoding production machinery
SGLang shipped a major speculative-serving wave around DSpark, EAGLE cleanup, MTP verification, GLM production paths, and scheduler/KV-cache refactors SGLang.9 The project also expanded ROCm, MI355, MiniMax-M3, Inkling, diffusion, and multimodal support while keeping high-volume bug triage active across GLM, DeepSeek, PD disaggregation, and CUDA graph behavior SGLang issues.15
Ollama makes the local CLI an agent surface
Ollama changed its default command behavior so running ollama opens an interactive agent for chat, code, and work, backed by a new TUI, tools, sessions, approvals, file helpers, bash support, and web plumbing Ollama.12 That puts Ollama closer to Claude Code, Codex, and local agent shells than to a model downloader alone agent PR.16
Blackwell kernels became a cross-project race
FlashInfer pushed CuTe DSL, NVFP4, sparse attention, SM120 and SM121 support, MoE-EP, and serving-path fixes for vLLM, SGLang, and TensorRT-LLM integrations FlashInfer.3 CUTLASS shipped CuTe DSL runtime work, Blackwell grouped block-scaled GEMM examples, and Hopper FP8 GEMM with fused GELU epilogues CUTLASS.17 Triton, FlashAttention, AITER, and ATOM filled in the same story from the compiler and ROCm sides Triton.18
Edge inference got less fragmented
Google LiteRT, Qualcomm AI Hub, OpenVINO, ExecuTorch, ncnn, sherpa-onnx, MNN, and RunanywhereAI all moved model support, accelerator support, or SDK APIs forward for mobile and local devices LiteRT.4 The edge stack now looks less like a separate market and more like the downstream half of the same inference system: export, quantize, serve, stream, and measure across cloud, desktop, phone, and browser RunanywhereAI.19
Deeper Dive
Everything below is for readers who want the full picture. Feel free to scroll.
Code Changes by Category
Cloud & Datacenter Serving
vLLM shipped a dense-model runner change, DeepSeek and GLM serving work, ROCm kernels, FlashInfer integration, KV offload, and Rust frontend security fixes vLLM release.13 SGLang had one of the busiest weeks in the ecosystem, with DSpark, EAGLE, GLM, MiniMax-M3, ROCm, scheduler, cache, and multimodal changes landing in the same release window SGLang release.9
TensorRT-LLM hardened disaggregated KV transfer, cache manager plumbing, speculation, DeepSeek, Gemma, Mamba, MiniMax, Qwen, quantization, and IPC behavior TensorRT-LLM release.20 AI-Dynamo pushed routing, admission control, KV correctness, SGLang and vLLM backend integration, Kubernetes operator fixes, XPU groundwork, and observability Dynamo.21 Ray upgraded its LLM serving stack around vLLM, fixed Serve scalability paths, and moved embedded GCS fault tolerance forward Ray.22
LMDeploy rebuilt TurboMind cache scheduling around page, slab, and object allocators while improving generation defaults, Qwen3.5 MTP behavior, FP8 MoE-only quantization, and guided decoding LMDeploy.23 LightLLM fixed an abort-path crash for Qwen3.5-style linear-attention mixed models, improved visual-worker memory reservation, and refreshed DeepEP integration LightLLM.24 DeepSpeed added AutoEP plus AutoTP composition, Inference V2 EXAONE support, ZeRO race fixes, and rollout support DeepSpeed.25
Local LLM Runtimes
llama.cpp absorbed a huge portability week across CUDA, ROCm, SYCL, OpenCL, Vulkan, Metal, WebGPU, Hexagon, CPU, and new model paths such as DeepSeek and Tencent Hy3 llama.cpp.5 Ollama made the CLI agent default, tightened Gemma 4 templates, restored Metal NAX kernels, fixed MLX timeout behavior, and polished launch integrations Ollama release.12 LocalAI shipped new diarized transcription, voice cloning, streaming TTS, video/avatar generation, DFlash gallery entries, VRAM budgets, and distributed-node upgrade fixes LocalAI release.26
Cactus removed its CoreML and Apple NPU-specific path while adding Needle support, tool calling, CQ handling, Homebrew fixes, and post-release parser fixes Cactus.27 Mozilla llamafile stayed quiet in merged code but saw active sandboxing and tool-calling proposals around pledge, SECCOMP, Landlock, MiniCPM5, and upstream llama.cpp sync llamafile sandbox PR.28 GPT4All, text-generation-webui, FastChat, and exllamav2 had no landed code, but users reported project-momentum concerns, MiniMax-M3 loader gaps, model onboarding requests, and EXL2 metadata security issues exllamav2 issue.29
Apple Silicon & MLX Ecosystem
Apple’s MLX stack improved Core ML deformable convolution conversion, MLX Metal diagnostics, GGUF metadata loading, quantized SDPA, Swift NVFP4 global scale support, FoundationModels integration, Gemma 4, Qwen3, Qwen3VL, and generation cancellation behavior MLX Swift LM.30 oMLX shipped Lightning MTP, custom kernels, Qwen long-context fixes, TurboQuant KV-cache corrections, FA-256 routing, and native-kernel packaging oMLX release.31 Osaurus shipped rapid macOS local-agent releases with Bonsai 1-bit JANG support, MLX engine fixes, model residency controls, sandboxing, MCP tools, evals, and localization Osaurus release.32
Blaizzy’s MLX audio and VLM projects added KV-cache quantization, APC batching, MOSS transcription, IndexTTS, Qwen3-Omni fixes, Gemma 4 dtype fixes, and realtime Swift audio fixes mlx-vlm.33 FluidAudio consumed byte-exact NeMo text normalization from text-processing-rs, added LuxTTS CoreML, improved ASR and diarization paths, and cut redundant transcribe decoding work FluidAudio.34 try-mirai advanced Uzu Gated DeltaNet speculation, Metal kernels, Gemma4, Qwen3.6 Hanashi, DFlash conversion, and lalamo model-format support Uzu.35
Mobile & Edge Frameworks
Google LiteRT and LiteRT-LM moved across WebGPU, GPU, NPU, Gemma dispatch, YNNPACK integration, structured outputs, OpenAI-compatible usage fields, Swift APIs, prompt templates, streaming audio, and local server behavior LiteRT-LM.10 Qualcomm added Qwen3-1.7B QAIRT SpinQuant, Qwen3-VL-8B-Instruct, GenieX runtime updates, Docker Hub model pulls, sliding-window controls, and safer aarch64 runtime behavior Qualcomm AI Hub Models.11 ExecuTorch expanded GGUF, MLX, CUDA quantization, WebGPU, Vulkan, Arm, and NXP backend paths while improving pass/runtime correctness ExecuTorch.36
OpenVINO improved quantized GQA KV-cache, GPU SDPA, NPUW MoE, LoRA builder support, Qwen3-VL embeddings, Minja template caching, and benchmark tooling OpenVINO GenAI.37 sherpa-onnx added Qualcomm QNN ASR export and runtime support for Nemotron and Moonshine, hardened TTS silence scaling, and improved Android Rust bindings sherpa-onnx.38 ncnn landed GEMM block quantization infrastructure for LLM decoder-style workloads across CPU and Vulkan backends ncnn.39 MNN enabled Transformer C4 graphs across CPU, CUDA, Metal, and OpenCL-adjacent paths while tightening QNN export behavior MNN.40
Compilers, Runtimes & Graph Engines
OpenXLA worked on GPU VMM allocation, command-buffer VA remapping, PJRT and IFRT memory semantics, FP6 support, DotMerger, TopK, ROCm, SYCL, oneDNN, and HLO correctness OpenXLA.41 JAX reworked XLA metadata APIs, advanced Pallas and Mosaic GPU kernels, added FP6 paths, hardened ROCm device detection, and updated TPU and XLA dependencies JAX.42 Apache TVM focused on Relax frontend correctness for ONNX, PyTorch, and TFLite, plus IR identity semantics, ORCJIT, ABI-aware builds, CUDA, Metal, and Web fixes TVM.43
Triton advanced Blackwell multi-CTA, TMA, TMEM, AMD gfx1250, Gluon, sanitizer, JIT cache, and dtype correctness work Triton.44 TensorFlow mirrored much of the OpenXLA compiler wave into its core tree, while TensorFlow Serving rolled to latest passing TensorFlow builds and TFLite Micro fixed a profiler ordering regression TensorFlow.45 ONNX and ONNX Runtime tightened TopK, QDQ reference ops, malformed-model validation, pooling, Range, WebGPU, Web external data, sub-byte tensors, and GenAI evaluation workflows through Olive ONNX Runtime.46
Models, Quantization & Optimization
Hugging Face Transformers shipped Inkling support and vLLM compatibility fixes, Diffusers added Cosmos3 pipelines and Nunchaku Lite quantization, and Optimum Intel added FunAudioLLM plus Qwen3-Omni OpenVINO export work Transformers release.47 FlashInfer pushed dense, MXFP8, NVFP4, sparse attention, MoE, TRTLLM-gen, MegaMoE, and autotuning work FlashInfer.3 FlashAttention advanced FA4 performance for B300/SM100/SM103A, fixed backward JIT cache churn, and hardened Hopper archive extraction FlashAttention release.48
ROCm AITER and ATOM moved together on DeepSeek-V4, GLM-5.2, MLA, SWA, FP8, MXFP4, MoE, gfx1250, SGLang, and vLLM paths AITER release.49 Intel Neural Compressor added QDQ support, layer-wise quantization fixes, and JAX per-layer configuration work Neural Compressor.50 NNCF added OpenVINO GroupedMatMul support for data-free weight compression and Qwen3 MoE compression examples NNCF.51 KTransformers focused on DeepSeek-V4-Flash hybrid CPU-GPU deployment docs, SGLang backend compatibility, GPU-resident expert memory, and AMX BF16 training fixes KTransformers.52
Other Notable Changes
LiteLLM had a security-heavy week around MCP auth, OAuth delegation, Dynamic Client Registration bridge flows, guardrails, provider routing, GPT-5.6 pricing, Anthropic streaming, Claude Code passthrough, and enterprise observability LiteLLM release.53 Open WebUI fixed streaming response corruption, WebSocket recovery, iframe prompt confirmation, terminal HTML sandboxing, Milvus Lite multitenancy, and pipe base-model resolution Open WebUI.54 BentoML had no merged code, but contributors opened security, ASGI middleware, observability, proxy, and test-coverage PRs BentoML security issue.55 Triton Inference Server added security guidance for executable model code and gRPC response-pool sizing, plus QA fixes for OpenVINO SBSA and Dynamo layout changes Triton Server.56
Community Pulse
The busiest issue queues belonged to vLLM, SGLang, LiteLLM, Ray, Google AI Edge, Open WebUI, and ggml’s llama.cpp, each of which saw high user pressure around serving correctness, model support, runtime regressions, or release adoption vLLM issues.57 Security reports were a visible pattern: exllamav2 received an EXL2/GPTQ quant metadata out-of-bounds report, KTransformers received an unauthenticated pickle deserialization RCE report, BentoML received a Dockerfile instruction-injection report, and Diffusers received a sharded-checkpoint path traversal report KTransformers issue.58
Apple Silicon communities stayed active across MLX, oMLX, Osaurus, Blaizzy, FluidAudio, and Uzu, with most threads focused on long context, speculative decoding, audio latency, KV-cache memory, and local-agent UX MLX discussions.59 Edge communities pushed real-device bugs into the open, including Qualcomm HTP failures, OpenVINO GPU/NPU issues, sherpa-onnx Android crashes, ncnn Mali Vulkan correctness, and Google LiteRT GPU miscomputes LiteRT issues.60
Community Debates
vLLM debated specialized cache features against maintainability
A DFlash sliding-window attention proposal closed after maintainers moved toward a KV-spec approach instead vLLM DFlash SWA PR.61 A proposed global redundancy buffer also closed because maintainers did not want to surprise users who already tune memory tightly vLLM redundancy buffer PR.62
SGLang kept replacing broad experiments with narrower scheduler and cache paths
An alternate DSpark implementation closed after native architecture work landed elsewhere SGLang DSpark PR.63 A proposal to preserve SWA sliding-window suffixes during eviction closed in favor of a simpler construction-side approach and future checkpoint-interval work SGLang SWA eviction PR.64
llama.cpp showed the tension between backend-specific wins and portable design
An AMD APU UMA detection PR closed after debate over HIP memory APIs, Linux memory reporting, cgroups, and safe loading budgets llama.cpp UMA PR.65 A CDNA-specific chunked MFMA prefill kernel also closed because maintainers preferred a generic MMA path and lacked an owner llama.cpp CDNA PR.66
Open WebUI enforced contribution process even on plausible fixes
A concrete chat-corruption fix for deleting a message while a reply streamed closed over CLA, template, and title issues Open WebUI PR.67 A proposed WebSocket compression change stayed contested because maintainers warned that disabling compression could make long streams huge under the current full-response-per-chunk behavior Open WebUI compression PR.68
TensorRT-LLM and CUTLASS kept narrowing broad kernel proposals
CUTLASS closed an initial FP8 GELU implementation in favor of a shared-kernel version that merged CUTLASS closed PR.69 TensorRT-LLM redirected multiple older branches around coverage tooling, MLA KV sizing, Mamba warmup, RMSNorm, and CUDA mapping tests into narrower successor PRs TensorRT-LLM PR.70
Worth Watching
Million-token context is turning KV-cache compression, paged cache ownership, cache offload, and prefix reuse into first-order system features, as seen in vLLM, SGLang, LMDeploy, Dynamo, OpenVINO, and KTransformers work this week vLLM KV work.71 Speculative decoding is spreading across NVIDIA, AMD, Apple Silicon, and local runtimes through SGLang, vLLM, oMLX, Uzu, LocalAI, FlashInfer, AITER, and Ollama-adjacent MLX work SGLang DSpark.72
Blackwell and ROCm are both forcing kernel projects to expose sharper hardware paths for FP8, NVFP4, MXFP4, sparse attention, and MoE, with FlashInfer, CUTLASS, Triton, FlashAttention, AITER, ATOM, and TensorRT-LLM all active FlashInfer NVFP4.73 Edge inference is shifting from demo apps to full SDK distribution, with RunanywhereAI, Qualcomm, Google LiteRT, OpenVINO, ExecuTorch, ncnn, MNN, and sherpa-onnx all handling real packaging, accelerator, and runtime-failure reports RunanywhereAI release.19
Major Releases
Ollama shipped v0.32.0, centered on making the interactive agent the default CLI entry point, renaming the ChatGPT launch integration, simplifying launch selection, and tightening the model picker experience. The most impactful change is the product pivot from local model runner to local agent shell..12
vLLM shipped v0.25.0 and v0.25.1 across the core project, plus vLLM-Gaudi v0.24.0. The core release made Model Runner V2 the default for dense models and added EVS, realtime embeddings, hybrid prefix caching, multimodal prefix attention, and speculative decoding compatibility, while the patch fixed TorchCodec import behavior and the Gaudi release tracked Intel Gaudi Software v1.24.1..13
SGLang shipped v0.5.15 and v0.5.15.post1, focused on GLM-5.2 production serving, speculative decoding, FlashInfer dependency fixes, long-input NaN fixes, and IndexShare repairs for PD disaggregation and context parallelism. The headline was GLM-5.2 NVFP4 serving with Spec V2 enabled by default..9
ggml shipped a heavy llama.cpp rolling-release run from b9934 through b10034, with backend work across WebGPU, HIP, ET, DeepSeek, Hy3, SYCL, CUDA, server tooling, CORS, reasoning budgets, and prompt-cache state ownership. whisper.cpp did not release, but it absorbed a large ggml sync during the week. Representative release.74
LocalAI shipped v4.7.0 and v4.7.1, centered on multimodal expansion, managed voice cloning, longcat video, moss-transcribe-cpp, F5-TTS, streaming vibevoice-cpp, DFlash gallery support, GPU selection, and a llama.cpp serving-option patch. The week made LocalAI look more like a full local multimodal platform than an OpenAI-compatible text server..26
NVIDIA shipped CUTLASS v4.6.0, CUTLASS v4.6.1, and TensorRT-LLM v1.3.0rc21. CUTLASS focused on CuTe DSL, compiler/runtime tooling, JAX FFI, GEMM docs, and bug fixes, while TensorRT-LLM highlighted AutoDeploy backend deprecation, Minimax M3, and PyTorch-backend model-support direction. CUTLASS release notes.75
BerriAI shipped nine LiteLLM releases across stable, release-candidate, development, and prerelease channels. The week’s theme was MCP auth, OAuth delegation, DCR bridge flows, guardrails, GPT-5.6 cost maps, provider compatibility, Claude Code passthrough, and enterprise observability. Representative release.53
Hugging Face shipped Transformers v5.13.1 and v5.14.0. The patch focused on vLLM compatibility and custom-model behavior, while the headline release added Inkling support for Thinking Machines’ large multimodal MoE model..47
AMD Ryzen AI shipped FastFlowLM v0.9.45 and Lemonade v11.0.0. FastFlowLM added Qwen3.6-35B-A3B MoE support and refreshed NPU/backend behavior, while Lemonade shipped Trellis.2 image-to-3D, OpenMOSS TTS, and router metadata. FastFlowLM release notes.76
FlashAttention shipped fa4-v4.0.0.beta22. The release focused on SM103A tcgen.ld.red, FA4 backward compile-key churn, Hopper archive-extraction hardening, and SM100 block-sparse backward support..48
FlashInfer shipped four nightly builds for v0.6.15 development. The nightlies track a week of CuTe DSL, Blackwell, NVFP4, sparse attention, MoE, TRTLLM-gen, autotuning, and serving correctness work. Representative nightly.77
ROCm shipped AITER v0.1.17 and v0.1.16.post4. The releases focused on OPUS gating, gfx950 hardening, RadeonFlow MXFP4 a4w4 MoE, MLA decode/reduce, FlyDSL FP8 MQA logits, MLA metadata, and top-k optimization..49
Blaizzy shipped mlx-audio v0.4.5 and mlx-audio-swift v0.1.3. The Python release highlighted ZONOS2 TTS, Higgs v3 caching, thread-safety, MOSS-TTS, and batch generation, while the Swift release expanded codecs, ASR, AER, LID, and Turbo voice paths. mlx-audio release notes.78
Cactus shipped v2.0 and v2.0.1. The release cycle covered LFM fixes, Gemma4 multimodal multi-turn work, converter logic, graph work, CoreML removal, and Homebrew distribution fixes..27
FluidInference shipped text-processing-rs v0.3.0. The release added byte-exact NeMo text normalization for seven languages behind the fst-engine feature and exposed nemo_tn_fst through the xcframework..79
jundot shipped five oMLX releases across v0.5.0 development, release-candidate, feature, and hotfix builds. The release train centered on Lightning MTP, custom-kernel acceleration, oQe quantization, TurboQuant KV-cache fixes, Qwen long-context stabilization, and native-kernel packaging. Latest release.31
k2-fsa shipped sherpa-onnx QNN model releases as asr-models-qnn-3 and asr-models-qnn-binary-3. The releases published Qualcomm QNN ASR model assets and companion context-binary model artifacts..80
Microsoft shipped ONNX Runtime v1.27.1. The patch release included CUDA QMoE decode-path optimization, MLAS KleidiAI igemm, QMoE CPU livelock fixes, and CI/build fixes..46
Osaurus shipped seven releases from 0.21.12 through 0.22.4. The cadence focused on Bonsai 1-bit JANG support, MLX runtime fixes, model residency, sandboxing, MCP tools, plugin hardening, evals, UI polish, and localization. Latest release.32
Qualcomm shipped ai-hub-models v0.57.3 and v0.58.0. The releases added GR00T-N1.5, Qwen3-VL-8B-Instruct, Qwen3-1.7B QAIRT SpinQuant deployment, GenieX runtime updates, and refreshed performance data. Latest release.11
RunanywhereAI shipped runanywhere-sdks v0.20.9 and v0.20.10. The release train aligned Swift, Kotlin, Flutter, React Native, Web, rcli, C/C++ core, llama.cpp, Sherpa, ONNX, Apple MLX, QHexRT, and CoreML diffusion across a broad on-device SDK surface. Latest release.19
try-mirai shipped lalamo v0.14.0. The release added runner CI, JAX flags, IBM Granite support, DeltaNet intra-chunk scan replacement, and related fixes while Uzu continued runtime work without a tagged release..81