Meta’s FAIR research lab has added e-means to faiss, its open-source library for similarity search, as a replacement for k-means, the standard method for sorting large piles of data into groups. K-means picks a set number of group centers, assigns every data point to its nearest center, moves each center to the middle of its points, and repeats. Every repeat reads the whole dataset.
E-means updates the centers from small batches using running averages, so it learns while it reads. According to the faiss repository on GitHub, that cuts the passes needed by about a factor of 10, delivers a similar gain in clock time, and ends with lower error than the classic method, with no added settings to tune. Those figures are Meta’s own, and the README points to a paper rather than an outside test.
It runs on CPUs through faiss and on NVIDIA GPUs through PyTorch, with custom kernels for H100 and H200 chips. Five passes is the default.
A team that re-clusters embeddings on a schedule can run the five-pass default beside its current k-means job and compare error and run time on its own data.
Reported from the facebookresearch/faiss repository on GitHub, which carries no publication date.
