An embedding model with 577 million parameters, designed to run on devices such as smartphones and laptops, would have moved from 26th to first place on the audio leaderboard of MAEB, the reference benchmark for audio and text-to-audio embeddings, currently dominated by multimodal models of 8 and 9 billion parameters. This is what an author on Reddit claims, who published the weights on Hugging Face: the difference would come from an addition of just 2.37 MB of frozen PCA matrices, with no training of the model weights.
According to the post, EmbeddingGemma-2's score would rise from 49.3% to 54.4%, a gain of +5.1 points. On the audio-only leaderboard the model would go from 26th to first, while on the full MAEB leaderboard from 16th to first. These are figures stated by the post's author and not yet verified by third parties.
Image: Google — https://blog.google/innovation-and-ai/technology/developers-tools/embeddinggemma-2/
The base model is Google's EmbeddingGemma-2, presented in early October as an open multimodal model that brings text, images, audio and video into a single 768-dimensional vector space. The model has 740 million parameters in total, of which 577 million are in the part the Reddit post uses as its base. Google designs it for search and classification on local hardware, without sending files to a cloud service, and in an earlier article we covered its cross-modal search capabilities.
For anyone working locally, the key point is size: a five-point gain obtained with a few hundred kilobytes of addition, rather than with a larger model, changes the calculation of what to run on one's own hardware. The post argues that the method requires no training, but independent verification of the scores remains the decisive step: until others reproduce the leaderboard, the first place stands as one author's claim.
No comments yet. Be the first!