The deployment of Qwen3.8-Flash-Next on Apple Silicon has reached a new milestone with the integration of support for MLX-serve. A new engine optimization allows the model to handle a 1 million token context window while maintaining high generation speeds and stability, specifically tuned for the M5 Max with 128GB of unified memory.
Performance and Quantization
The implementation leverages an 8-bit KV cache to manage the massive memory requirements of a million-token context. To balance performance and quality, this specific quantization uses 8 bits for dense layers and 4 bits for expert layers.
According to the developers, this configuration sustains generation speeds of approximately 40 tok/s for prose and up to 75 tok/s for coding tasks, even when operating deep within the context window. The stability of the model at extreme lengths was demonstrated through a snapshot at roughly 760k tokens, where it successfully built an MLX Serve Monitor plugin.
Technical Requirements for Local Execution
To run the model at this scale on Mac hardware, a specific system configuration is required. Users must set iogpu.wired_limit_mb=1200 to ensure the GPU has sufficient wired memory for the operation.
This update expands on previous efforts to bring Alibaba's MoE architecture to local hardware. While Qwen3.8-Flash-Next was previously available via GGUF weights, the MLX-serve integration optimizes it specifically for Apple's unified memory architecture. This follows a trend of accelerating Qwen models on Mac, as seen with previous MLX-based optimizations that significantly boosted token throughput.
No comments yet. Be the first!