A language model trained from scratch exclusively on philosophy texts has been released by Gustave Cortal under the name PhiloLM. It is a 112-million-parameter base model, a neural network far smaller than large commercial systems, built to show that training on a small specialized corpus can yield a usable model.

Psst!Add AlexTech.ai asPreferred Source on Google

The core of the project is the PhiloText dataset, made of about 0.5 billion tokens from open sources: the Stanford Encyclopedia of Philosophy and the Internet Encyclopedia of Philosophy (IEP), public-domain books, reviews, textbooks and educational web pages. The model was then trained on 1.5 billion tokens in total, a very modest volume by current standards.

Training was possible on very cheap hardware. According to the documentation published on Hugging Face, the model was trained in about 15 hours, over 88,777 updates, on a single Intel Arc A770 16 GB card, which the author lists at $350. The process used the Muon optimizer, bigram embeddings, sliding-window attention and Grouped Query Attention (GQA), techniques that reduce computation and memory costs.

The most concrete point for users concerns inference speed. The model is optimized to run locally and reaches about 500 tokens per second on an Intel Core i5-14400F, listed as a $150 CPU. Expensive graphics cards are not required for fluid generation.

The project's value lies mainly in its method. It shows that a focused corpus of modest size, combined with efficiency techniques, can produce a trainable model in limited time and on a small budget. Weights and dataset are available on Hugging Face, and the author also offers a demo that runs in the browser.