Tencent has released WeMM-Video, an open family of models that understand video, audio, images and text in a single reasoning process. There are four variants: 2B, 4B, 9B and 35B-A3B, where the suffix indicates 35 billion total parameters with about 3 billion active per token. The release came on November 19, with weights on Hugging Face and a technical report on arXiv.

Psst!Add AlexTech.ai asPreferred Source on Google

The distinctive point is the joint reading of images and sound. The model can combine a video's frames with its soundtrack to describe what happens, transcribe speech and reason about what is seen and heard. It also accepts silent video, standalone audio, images and text, and responds in Chinese or English.

official image of WeMM-Video

Image: collections — https://huggingface.co/collections/tencent/wemm-video

Separate vision and audio encoders turn media into tokens that the language model processes alongside text. Video frames carry timestamps, so the model knows when each event occurs.

Training used about 1.2 trillion multimodal tokens, with progressive pre-training and a multi-stage post-training pipeline combining multi-teacher on-policy distillation (OPD), an anti-repetition technique called MrDPO and specialist reinforcement learning. According to the report, the models were evaluated across more than 120 benchmarks.

Per the report, the flagship 35B-A3B establishes a new open-source state of the art in omni-modal video evaluations. The report also claims highly competitive results on general audio, silent video, image and text tasks. These are figures reported by Tencent; no independent evaluation of them is available yet.

The full technical report is on arXiv, the model pages are on Hugging Face and the code is on GitHub. For developers and researchers, the signal is that a Chinese company is releasing open weights for a multimodal family that covers video with sound, including in its smaller versions.