Gemini And DeepSeek Swamp KV Caches

TL;DR

Long-context serving became the main fight: vLLM, SGLang, TensorRT-LLM, LMDeploy, and Dynamo all pushed cache, scheduler, disaggregation, or multimodal serving work for larger agentic workloads.1
Qwen3.8 spread through local and Apple stacks: Ollama, Apple MLX, oMLX, mlx-vlm, uzu, and Runanywhere all added or hardened Qwen3.8 paths, with Apple Silicon taking a clear role as a first-class inference target.2
Kernel projects chased the memory wall: FlashInfer, CUTLASS, FlashAttention, Triton, ROCm AITER, and llama.cpp all worked on low-bit, MoE, attention, or KV-cache paths that reduce bandwidth pressure.
Edge inference looked more like production software: Google LiteRT, ExecuTorch, Qualcomm AI Hub, OpenVINO, Cactus, and sherpa-onnx all shipped runtime, packaging, and model-coverage work for mobile and device deployments.3
Security and operational hardening moved up the stack: Triton Inference Server, LocalAI, LiteLLM, Open WebUI, ONNX Runtime, and ExLlamaV3 all surfaced or fixed issues around auth, malformed inputs, task state, streaming, or vulnerability reporting.

This Week in Inference

Google made Gemini 3.7 Flash generally available with multimodal inputs, a million-token input window, code execution, function calling, and grounding hooks for agent workflows Gemini changelog.4 DeepSeek moved V4 Pro 0813 into general availability with a million-token context window, a much larger output budget, and stronger agent and coding benchmark claims DeepSeek release details.5 xAI pushed Grok 4.6 with a large context window, a new high reasoning level, and flat reported pricing versus the prior model Grok 4.6 summary.6

The model pattern is now clear: providers want agentic models to keep more state, call more tools, and run longer workflows, while serving teams pay the bill in prefill, KV-cache, routing, and memory bandwidth Baseten DeepSeek analysis.7 Quantization research followed the same pressure point, with WUSH, ParoQuant, AdaMX, and CubicQuant all targeting lower-bit inference that protects reasoning quality or kernel efficiency WUSH.8 Local inference showed the same bottleneck in practical form, as llama.cpp users pointed to large gains from quantized KV-cache decode work at long context llama.cpp.9

Hardware and infrastructure news reinforced that inference has become its own capital market. Etched raised a large new round and delivered its first rack to Jane Street, which puts custom inference silicon in the same strategic category as model labs and cloud clusters HPCWire.10 NVIDIA’s Blackwell MLPerf backdrop still shaped open-source work, as FlashInfer, CUTLASS, TensorRT-LLM, Triton, and vLLM all chased newer GPU kernels, FP4 and FP8 paths, or continuous batching stability TensorRT-LLM.11 The same pressure reached edge stacks, where LiteRT, ExecuTorch, OpenVINO, Qualcomm AI Hub, Runanywhere, Cactus, and sherpa-onnx all improved packaging, backend selection, model catalogs, or mobile runtime behavior LiteRT.3

Deeper Dive

Everything below is for readers who want the full picture. Feel free to scroll.

Code Changes by Category

Cloud & Datacenter Serving

vLLM spent the week on production hardening rather than a clean headline release: core work touched ROCm CI, Gaudi fixes, NPU/XPU paths, ModelRunner pooling, compressed-tensors FP8, FlashMLA, DeepSeek MLA, Kimi-K3 disaggregated serving, and OpenAI frontend behavior vLLM.1 The Gaudi plugin fixed INC quantization, Qwen cache restore, MoE gate handling, paged attention layout, decode bucket bounds, Mamba prefill batching, and shared-expert synchronization vLLM Gaudi.22

SGLang had one of the densest serving weeks in the ecosystem, with DFlash2, fmha_v2, FP8 DSA tuning, DeepSeek-V4 metadata parallelization, quantized lm_head loading, Kimi-K3 mixed quantization, HiCache host memory mode, DCP work, and sglang-omni audio serving all moving in one window SGLang.12 sglang-omni added Breakable Prefill CUDA Graph support for Qwen3-Omni, speech prefill paths, MOSS-TTS, Fun-ASR, MiniMax Music, Qwen3-TTS fixes, and WebSocket/audio operational features sglang-omni.23

TensorRT-LLM added VisualGen image editing, GlmImage support, DFlash for Qwen3.6, speculative decoding for Kimi K3, KV cache V2 recompute-pause paths, Prefill Breakable CUDA Graph support, Python disaggregation safety, and a Torch/Triton upgrade TensorRT-LLM.11 Triton Inference Server hardened malformed HTTP inference headers so oversized header-size values now return a client error instead of crashing the server Triton Server.24

Dynamo shipped a production-scale operator and serving release with elastic EP scheduling, DRA-aware planning, KV routing, frontend validation, NATS TLS, variable-length prefill benchmarking, and stronger shutdown/downscale design work Dynamo.25 LMDeploy shipped a release centered on Interns2 Mobius, GLM-5.2, TurboMind ViT support, preprocessing refactors, GLM-5.2 FP8 throughput gains, SSM prefix-cache correctness, int4 KV-cache fixes, and Anthropic thinking-block compatibility LMDeploy.26

LiteLLM merged a large proxy-control wave across batch-token reservations, refunds, spend accounting, project budgets, auto-router UI, shadow evals, model deprecation governance, Vertex regional pricing, OTEL spans, signed image guidance, and dashboard migration work LiteLLM.27 Ray improved distributed runtime reliability, Data shuffle execution, Serve LLM streaming error surfacing, Mobilint NPU scheduling, and KubeRay docs while preparing larger storage and shuffle changes Ray.28

Local LLM Runtimes

llama.cpp added Kimi-K3 support, BailingMoE3 and Ling 3.0 flash support, Granite SWA/MoE model classes, draft GGUF speculative-mode detection, DSpark and DFlash sampling paths, Vulkan long-context fixes, CUDA/HIP/SYCL/Metal/WebGPU updates, server auth and metrics fixes, and signed release artifacts llama.cpp.9 whisper.cpp pulled ggml backend syncs and continued CUDA, HIP, SYCL, OpenVINO, Vulkan, Metal, OpenCL, WebGPU, CPU, and packaging updates whisper.cpp.29

Ollama’s Qwen3.8 work spanned renderer selection, MLX import, reasoning templates, shard validation, tool routing, non-leading system message tolerance, model metadata caching, and parser error cleanup Ollama.2 LocalAI hardened default HTTP auth, added end-to-end context compression, improved KNN routing and global admission, moved cold distributed loads to durable jobs, refreshed many backend pins, and expanded its model gallery LocalAI.30

ExLlamaV3 fixed autosplit reserve accounting for multi-model inference and saw active Qwen3.8, DFlash2, ROCm, tensor parallel, and vulnerability-coordination threads ExLlamaV3.31 CTranslate2 reduced inference-only Python import cost by lazy-loading converter/spec modules, then updated cxxopts for GCC compatibility CTranslate2.32

vllm-mlx hardened streaming tool-call parser state, Llama tool parsing, MLX stream ownership, prefix-cache reuse, scheduler parity, and embedding length controls vllm-mlx.33 Open WebUI merged authentication event parity, SSO event fixes, JWT revocation, Responses routing cleanup, integration UI polish, background-task lifetime fixes, and SQL-backed access filtering work Open WebUI.34

Apple Silicon & MLX Ecosystem

Apple’s MLX release covered GGUF metadata fixes, RMSNorm CUDA speedups, attention and quantization cleanup, Metal kernels, 3D convolution, FFTs, distributed transport, packaging, and JIT work MLX.15 Apple coreai-models added Muse Glimmer support, VLM video inputs, GPU constrained generation, export contract updates, and smaller release-mode exports coreai-models.35

mlx-lm aligned with the new MLX runtime, fixed InternLM RoPE, Qwen RMSNorm conversion, XTC defaults, exact size parsing, and model save behavior mlx-lm.36 mlx-swift-lm added Qwen3.5 and Qwen3.6 MTP speculative decoding, thinking budgets, tool-call parser hardening, rejected-tool-call events, video VLM options, TranslateGemma, and Qwen VLM fixes mlx-swift-lm.16

Blaizzy’s mlx-vlm added Qwen3.8-27B, Nemotron-Parse, GOT-OCR, PLaMo VL, LFM tiling gains, SmolVLM/Idefics3 fixes, RoPE fixes, APC/cache hardening, server settings updates, and MLX compatibility fixes mlx-vlm.37 mlx-audio added MiniMax Music 3 as a first-class music pipeline and documented Irodori-TTS models mlx-audio.38

oMLX moved fast on Qwen ANE/GPU split prefill, Q5/Q6/Q8 ANE paths, tuner utilities, DFlash 2, distributed serving, runtime preflights, DeepSeek-V4 memory estimation, and MCP compatibility oMLX.39 uzu and lalamo added Qwen3.8, GPT-OSS MXFP4 layout preservation, QKVG fusion, DFlash export fixes, small-M Metal GEMM selection, RHT fusion, shared-chat examples, and tokenizer ownership cleanup uzu.40

Mobile & Edge Frameworks

Google LiteRT shipped runtime packaging, Rust integration, ABI tracking, GPU/compiler modules, Qualcomm KV-cache transforms, Hexagon builder fixes, Arm plugin skeleton work, and LiteRT-LM embedding, constrained decoding, audio preprocessing, and OpenAI-compatible embeddings LiteRT.20 MediaPipe added LiteRT-LM embedder executor work and Holistic Landmarker sample coverage across Python, Android, iOS, and Web MediaPipe.41

ExecuTorch shipped a patch release and merged Arm/Cortex-M quantized KV-cache export, sliding-window cache policy, Ethos-U/VGF coverage, CMSIS transforms, Qualcomm QNN QAT/QAD work, ROCm AOTI support, Vulkan fixes, and portable-kernel corrections ExecuTorch.42 Qualcomm AI Hub Models added GenieX QAIRT plugin support for Gemma and Qwen, new catalog entries, GenAI quantization recipes, mixed-int coverage, and scorecard collection improvements AI Hub Models.43

Cactus shipped a release with wider calibrated-quant support, Needle support, tool calling in run environments, Parakeet audio bucket exports, Metal CQ4 fixes, and Raspberry Pi portability Cactus.44 sherpa-onnx shipped Flutter VAD/ASR examples, unified Dart/Flutter initialization, Apple packaging fixes, pyannote window-shift bindings, Android/Java/Web fixes, and release metadata updates sherpa-onnx.45

FluidAudio expanded ASR with Canary, Nemotron hotword biasing, Parakeet vocabulary boosting, diarization fixes, Inflect TTS, NeuTTS, KokoroAne fixes, and packaging cleanup FluidAudio.46 Runanywhere pushed a cross-platform SDK and app train across Windows, Apple Neural Engine, WebGPU, QHexRT/NPU, Flutter, React Native, Android, Electron, iOS, macOS, and Web Runanywhere SDKs.47

ncnn merged a unified KV-cache allocator for MultiHeadAttention and SDPA across CPU architectures and Vulkan, added BF16 modelbin storage, and improved normalization accuracy for LLM paths ncnn.48 mllm added MiniCPM5-1B CPU inference plus a native KV-head GQA cache and decode operator that cuts attention-cache memory for grouped-query models mllm.49

Compilers, Runtimes & Graph Engines

Triton advanced Blackwell/Rubin codegen with warp reductions, masked memory improvements, MMA descriptor cleanup, tensor-map simplification, cluster barrier fixes, packed BF16/FP8 conversions, TCGen5 reuse, FP4/NVFP4/MXFP matmul metadata, and AMD backend corrections Triton.50 CUTLASS shipped CuTe DSL primitives, a task scheduling framework, better diagnostics for spills/local memory/source lines, and custom epilogue fusion improvements CUTLASS.18

FlashInfer added next-gen GPU GEMM kernels, TRT-LLM Gen MoE support, MegaMoE fixes, attention/FMHA/MLA coverage, video sparse attention, ragged prefill fixes, Cake KDA/GDN serving paths, sampling safety, and CI ownership improvements FlashInfer.17 FlashAttention relaxed CUTLASS DSL requirements to unblock FA4/CuTe dependency combinations and kept Blackwell HD256, Ampere FA4, deterministic, and ROCm install work active FlashAttention.51

OpenVINO added SSMFusion, a PagedAttention spec, transformer encoder frontend lowering, ONNX GQA fixes, GGUF Q2_0/stateful conversion, Qwen3-Omni benchmarking, Qwen3.5/3.6 VLM PagedAttention benchmarking, Node.js ASR, GPU BF16/SwiGLU/RoPE kernels, and NPU updates OpenVINO.52 ONNX Runtime shipped a CUDA plugin execution provider, device-free WebGPU compilation, sandboxed Windows compatibility, Qwen CPU/WebGPU ops, WebGPU Gelu/BiasGelu fusion, CUDA FP4/QMoE work, and broad validation hardening ONNX Runtime.53

TVM improved typed PTX helpers, tensor-map validation, WebGPU stack allocations, ONNX scalar quantization import, StableHLO shape ops through TFLite frontend, memory aliasing, and DLight GEMV correctness TVM.54 OpenXLA improved GPU collective scheduling, ROCm latency-estimator data, CPU complex lowering, SVD edge cases, PJRT/IFRT cleanup, Mosaic dynamic slicing, autotune caches, and compiler diagnostics OpenXLA.55

JAX shipped a release and modernized Bazel, Python, XLA rolls, PJRT export, Pallas/Mosaic/fuser paths, TPU/GPU/ROCm/Intel portability, NumPy compatibility, typing, and tracing performance JAX.56 TensorFlow moved PjRt/IFRT, XLA CPU/GPU/ROCm, TPU/Mosaic, TFLite conversion, tflite-micro wheels, compression tests, and TensorFlow Serving request logging forward TensorFlow.57

Models, Quantization & Optimization

Hugging Face added Stable Audio 3 Medium and MiniMax Music 3 in Diffusers, Step 3.7 in Transformers, LTX2 image-to-video and DeepSeek-OCR-2 OpenVINO support in Optimum Intel, plus tensor-parallel inference, Krea memory fixes, NVFP4, FlashAttention fixes, and Candle backend correctness work Transformers.58 Microsoft Olive added Qwen3.5/3.6 MoE VL quantization, K-last MoE GPTQ, PyTorch KQuant MoE support, MobiusBuilder filtering, QNN GPU weight sharing, and newer Transformers support Olive.59

KTransformers shipped full fine-tuning support, AMD CPU AVX512 fine-tuning, native block-FP8 routed-expert LoRA SFT, Qwen VLM MoE fine-tuning, and a consolidated SFT release workflow KTransformers.60 Intel Neural Compressor added a JAX dot-product-attention quantization switch, Transformers compatibility fixes for PyTorch examples, Qwen3-235B-A22B evaluation handling, and deprecation markers for TensorFlow examples Neural Compressor.61

AMD FastFlowLM shipped Qwen3.5 and Qwen3.6-MoE NPU optimizations, FLM-quant updates, OMNI/dequant speedups, server robustness, packaging, and driver docs FastFlowLM.62 Lemonade added ROCm support, server model/options APIs, GUI updates, Prometheus metrics, Qwen/Nemotron catalog additions, TheNoise image generation work, and local/server fixes Lemonade.63

Other Notable Changes

Modular open-sourced Mojo and advanced MAX graph/runtime observability, MLIR text IO, profiler spans, shared graph memory pools, Kimi-K2.5 vision cache routing, Qwen3.5 GDN fusion, Metal diagnostics, RISC-V detection, and Mojo stdlib cleanup Modular.64 DeepSpeed added HybridEngine CUDA graph capture, an AutoTP compiler pass, native pinned-memory backend support, activation-offload controls, ZeRO dtype and reduction fixes, MoE grouped-GEMM safety, and DeepCompile alias handling DeepSpeed.65

Mooncake improved Store HA, snapshots, ordered oplog states, LocalSSD extraction, DFS and object-storage backends, transport-aware segment splitting, MUSA IPC, RDMA credit/control codecs, EFA fixes, native EP packaging, and resharding manifests Mooncake.66 Osaurus added projects for group chats, follow-up suggestions, reasoning-effort constraints, OpenAI answer-room reservation, vmlx repins, stop/cancel persistence, subagent spawning, knowledge fixes, and silent-restart attribution Osaurus.67

Community Pulse

The busiest communities showed the same stress pattern: long-context correctness, multi-accelerator support, and production error handling created more issue traffic than classic model-loader bugs. vLLM users reported compressed-tensors FP8 regressions, ROCm DeepSeek-V4 corruption, Kimi-K3 disaggregated serving issues, prefix-cache misses, and ModelOpt hangs vLLM issues.68 SGLang users pushed on AMD roadmaps, DeepSeek-V4 long-context crashes, Kimi-K3 P/D latency, XPU GDN mismatches, DSpark drift, and HiCache zero-hit sessions SGLang issues.69

Apple-adjacent communities were unusually active because Qwen3.8, MLX runtime changes, ANE split prefill, and VLM/audio projects all moved at once. MLX users discussed Python shutdown segfaults, JACCL tensor-parallel corruption, MoE/grouped GEMM throughput, reduction fusion, very-large local Qwen/DeepSeek inference, and low-bit quantization MLX issues.70 oMLX users produced concrete Qwen3.8 performance reports, ANE tuner eligibility findings, and fleet A/B/A validation for the release train oMLX discussions.71

Security and operational maturity also stood out. Triton fixed an HTTP crash path from oversized inference headers Triton PR.24 LocalAI enabled protected HTTP routes by default LocalAI PR.72 ncnn received several ASan-driven malformed param/model reports ncnn issues.73 ExLlamaV3 received a CERT/CC vulnerability coordination issue around insufficient bounds checking in a kernel dispatch path ExLlamaV3 issue.74

Community Debates

Open WebUI enforced feature intake discipline

An Olostep web search integration closed because maintainers said new providers need demonstrated community demand before merge Open WebUI PR.75 That decision matches several other process-gated closures for OpenAPI tool results, offline assets, Qdrant embedding config, and provider-specific fixes Open WebUI debates.76 The project appears to be optimizing for maintainer throughput and product coherence over accepting every integration PR.

LocalAI accepted context compression only after end-to-end wiring

LocalAI’s context-compression PR first drew maintainer pushback as incomplete, then landed after the implementation covered model metadata, summarization, preserved prompt/tool units, metrics, validation, and docs LocalAI PR.77 The same project reverted a broad test-isolation experiment after maintainers found it too invasive and brittle LocalAI revert.78 The lesson was direct: production features can land quickly, but half-wired infrastructure changes get cut.

Modular opened Mojo, but not the compiler floodgates

Modular merged the Mojo open-source milestone and contributor docs, but closed an external compiler fix because the project is not yet accepting compiler contributions Modular compiler PR.79 Several other PR closures clarified the internal-upstream workflow, where work may land internally or close as superseded rather than merge through GitHub Modular PRs.80 That creates momentum and friction at the same time: the code is open, but compiler governance is still staged.

SGLang and vLLM both pruned risky kernel and parser work

SGLang closed several architecture or performance proposals around sparse attention, speculative suffix decoding, token-slot accounting, Blackwell overlap, and MoE replay metadata because maintainers wanted stronger evidence, cleaner scope, or safer semantics SGLang PRs.81 vLLM closed parser and multimodal cache proposals when template compatibility, remote code, field clobbering, or OpenAI API behavior made the change risky vLLM PRs.82 Both projects are learning that serving-engine correctness now spans kernels, schemas, templates, and user API contracts.

CUTLASS and Triton pushed back on broad performance heuristics

CUTLASS maintainers questioned pruning SM90 generator configs based on register spills, asking for broader profiling rather than a simple generator rule CUTLASS PR.83 Triton maintainers pushed back on Blackwell 256-bit global access changes, repro-only PRs, plugin-key optimization docs, and loader complexity Triton PRs.84 Kernel projects are willing to move fast, but they still demand proof that micro-optimizations survive across shapes and architectures.

Worth Watching

DFlash2 is the thread to track next week because llama.cpp, vLLM, oMLX, ExLlamaV3, SGLang, TensorRT-LLM, and uzu all have active or recent work around draft, MTP, DFlash, or speculative decode behavior vLLM DFlash2.85 The open question is whether engines can make draft acceleration reliable across quantized models, long context, and mixed hardware without giving back the latency gains in scheduler complexity.

Apple ANE/GPU split inference is becoming a real subsystem. oMLX, Runanywhere, FluidInference, Apple coreai-models, and MLX-adjacent projects are testing when to push prefill, audio, or vision work to ANE/Core ML versus MLX/Metal oMLX.86 Maintainer resistance in mlx-audio-swift and mlx-vlm to CoreML/ANE offload PRs shows that the split is still a product and architecture decision, not a default mlx-audio-swift PR.87

AMD inference is moving from “supported” to “competitive but sharp-edged.” ROCm AITER, ATOM, vLLM ROCm, SGLang AMD, LMDeploy, llama.cpp HIP, Lemonade ROCm, and FastFlowLM all added or debugged AMD paths this week AITER.88 The remaining watch items are CustomAllreduce correctness, gfx950/gfx1250 kernel coverage, ROCm packaging, and whether MI300/MI350 class systems get parity in long-context MoE workloads.

Edge SDKs are converging on model catalogs plus backend-specific packaging. Google LiteRT, Qualcomm AI Hub, Runanywhere, Cactus, sherpa-onnx, FluidAudio, ExecuTorch, and OpenVINO all improved not only runtimes, but install, sample, metadata, and app integration paths Runanywhere SDKs.89 The next bottleneck is less likely to be “can this op run” and more likely to be “can this model be downloaded, selected, benchmarked, sandboxed, and updated safely on the user’s device.”

Major Releases

Version numbers and release-note links live here as the canonical reference.

Dynamo shipped v1.4.0 as its 17th feature release, covering router multi-datacenter work, sequenced KV relay, peer reservation replay, endpoint-scoped event transport, an experimental vLLM-compatible generate endpoint, and tokenizer prefix-cache work.25 The release theme was production distributed serving, with operator scheduling, KV routing, API validation, and release infrastructure moving together.

AMD Ryzen AI shipped FastFlowLM v1.0.2, GAIA v0.23.0, and Lemonade v11.6.0 and v11.7.0 across the local Ryzen AI stack FastFlowLM release.62 The dominant theme was NPU and ROCm local inference, with Qwen optimization, server APIs, GUI fixes, catalog updates, signed skills, and safer agent defaults.

Apple MLX shipped MLX v0.32.1 with GGUF metadata int64 cast fixes, normalization docstring corrections, configure-time NAX warnings, and CUDA RMSNorm forward speedups.15 The release sat on top of a much broader week of runtime, attention, quantization, Metal, distributed, and downstream MLX updates.

BerriAI LiteLLM shipped v1.97.0 plus v1.98 and v1.99 prerelease trains, with release notes centered on signed Docker image verification via cosign Stable release.90 The code behind the train focused on proxy spend controls, batch-job quotas, shadow evals, routing UX, provider accounting, model deprecation governance, and OTEL cleanup.

Blaizzy shipped mlx-audio v0.5.0 plus mlx-vlm v0.6.14 and v0.6.15 mlx-audio release.38 The theme was MLX multimodal expansion, with MiniMax Music 3, Irodori-TTS docs, VLM server fixes, batched padding, and MLX compatibility updates.

Cactus Compute shipped cactus v2.1.0 with wider calibrated-quant support, Needle support, tool-calling support in the run environment, benchmarks, and README fixes.44 The broader week hardened Parakeet audio buckets, Metal quantization, Needle tool schemas, native engine lifecycle, fine-tuning docs, and offline deployment.

Dao-AILab FlashAttention shipped fa4-v4.0.0.beta27 as a focused prerelease that relaxed CUTLASS DSL requirements for FA4/CuTe compatibility.51 That small packaging change mattered because B300 and QuACK dependency combinations were already exposing hangs and resolver conflicts.

FlashInfer shipped six nightly builds in the v0.6.18 development line Latest nightly.91 The release cadence matched a high-velocity week around Blackwell, Rubin, MoE, TRT-LLM Gen, MLA, ragged attention, Cake KDA/GDN, and CI hardening.

FluidInference shipped FluidAudio v0.15.6 with cleanup of download migration scaffolding, KokoroAne normalization fixes, English TTS normalization, and native NeMo text-normalization routing where available.46 The larger week expanded ASR, TTS, VAD, diarization, and CoreML/ANE conversion work.

ggml shipped a dense llama.cpp nightly train plus a v0.1.2 prerelease, while whisper.cpp consumed ggml 0.20.x syncs without a separate weekly release in the supplied notes llama.cpp latest weekly tag.92 The dominant theme was formalized versioning, backend breadth, new model support, long-context fixes, speculative decode metadata, and signed artifacts.

Google AI Edge shipped LiteRT v2.2.0 and LiteRT-LM v0.16.1 LiteRT release.20 LiteRT focused on Rust integration, ABI tracking, GPU accelerator, Dispatch, Compiler modules, and LiteRT.js/Tensor API updates, while LiteRT-LM fixed a Windows JVM crash path.

Hugging Face shipped Transformers v5.15.1 as a patch release for DFlash/MTP candidate-generator fixes and accelerator image-processing failures with Lanczos filtering.58 The wider Hugging Face week added audio/music generation, Step 3.7, OpenVINO model support, tensor parallel inference, NVFP4, and Candle backend fixes.

InternLM LMDeploy shipped v0.16.0 with Interns2 Mobius, GLM-5.2, Intern-S2-Mobius MoE/meta-MoE paths, CP attention fixes, and TurboMind ViT support for InternVL and Qwen VL models.26 The release also wrapped preprocessing refactors, GLM-5.2 FP8 performance, SSM prefix-cache work, and protocol compatibility updates.

JAX shipped v0.11.1 with export deserialization checks for old artifacts, a temporary bypass flag for expired versions, and the beginning of a jax.numpy.top_k entry.56 The week also modernized build tooling, XLA rolls, PJRT export, Pallas/Mosaic, TPU/GPU/ROCm/Intel portability, and NumPy compatibility.

jundot oMLX shipped v0.6.0 through v0.6.3rc1 in a rapid Apple Silicon train Latest release.39 The dominant theme was distributed serving and Qwen ANE/GPU acceleration, with DFlash 2, split tuning, cluster reliability, MCP fixes, cache reuse, and long-context improvements.

k2-fsa sherpa-onnx shipped v1.13.6 with Flutter VAD/ASR examples, Dart/Flutter initialization, pyannote window-shift bindings, iOS SPM fixes, Android/Java fixes, and Web fixes.45 The release strengthened cross-platform speech deployment and packaging behavior.

kvcache-ai KTransformers shipped v0.7.0 with full fine-tuning and AMD CPU AVX512 fine-tuning support.60 The week’s broader work added block-FP8 routed-expert LoRA SFT, Qwen VLM MoE fine-tuning, and a consolidated SFT release workflow, while Mooncake focused on Store HA and transport infrastructure.

Meta ExecuTorch shipped v1.4.1 to fix incorrect baked-in wheel paths and XNNPACK PreLU in quantized graphs.42 The surrounding week expanded Cortex-M, Arm, Qualcomm QNN, ROCm AOTI, Vulkan, and portable-kernel coverage.

Microsoft ONNX Runtime shipped v1.28.1 and plugin-ep-cuda/v0.1.0 ONNX Runtime release.53 The patch release focused on device-free WebGPU compilation, sandboxed Windows compatibility, and graph-validation fixes, while the plugin release packaged CUDA execution as a separate provider.

NVIDIA CUTLASS shipped v4.7.0 with experimental CuTe DSL primitives, task scheduling, improved compiler diagnostics, and Operator API custom epilogue fusion improvements.18 The release aligned with heavy TensorRT-LLM and Triton kernel work around FP4, FP8, Blackwell, and serving diagnostics.

Ollama shipped v0.32.11 through v0.32.15 in a fast local-inference release train Latest release.93 The dominant theme was Qwen3.8 support, Apple Silicon optimization, agent launch integrations, WebP transcoding, non-leading system-message fixes, and model metadata caching.

Osaurus shipped 0.22.21, 0.22.22, and 0.23.0 Latest release.67 The train added group-chat projects, silent-restart attribution, API answer-room reservation, ZAI-GLM reasoning controls, Qwen reasoning constraints, vmlx repins, and custom-agent subagent fixes.

Qualcomm shipped ai-hub-models v0.60.0, ai-hub-apps v0.35.0, GenieX v0.3.20, and GenieX v0.4.0 AI Hub Models release.43 The theme was on-device GenAI delivery, with QAIRT plugin support, signed Hexagon HTP catalog bits, new sample apps, Windows on Snapdragon improvements, and expanded quantization metadata.

ROCm AITER shipped v0.1.20 with manylinux wheels for gfx942 and gfx950.19 The release sat inside a broad AMD inference week that included attention, MoE, GEMM, ROCm 10 sampling gates, ATOM serving integrations, and MIGraphX compiler controls.

RunanywhereAI shipped SDK v0.20.18, v0.20.19, and v0.20.24, plus Windows Electron and macOS iOS-app releases SDK latest release.47 The release train focused on on-device runtime coverage across WebGPU, Windows, Apple Neural Engine, QHexRT/NPU, MLX, ONNX, Sherpa-ONNX, and cross-platform starter apps.

SGLang shipped sglang-omni v0.1.2 with CI calibration precheck, docs, Qwen3-Omni multimodal embedding merge optimization, and Qwen3-ASR language detection.94 Core sglang did not publish a release in this slice, but it merged a large serving, quantization, cache, and hardware-support wave.

try-mirai shipped uzu 0.5.16 and 0.5.18, both mainly package/version refreshes across Cargo, Swift, README, Python/Swift/TypeScript docs, and generated metadata Latest release.95 The code work behind the releases focused on Qwen3.8, GPT-OSS, MXFP4, DFlash, Metal GEMM, shared chat, and reasoning controls.

zetic-ai shipped ZeticMLangeiOS 1.10.0 with a public Swift Package metadata update pointing at a checksum-pinned XCFramework archive.96 The release was automation-driven and did not include visible community discussion.