One Encoder, Two Jobs: UltraBERT and the End of Scaling Your Embedder
UltraBERT tops 9 of 10 encoder benchmarks at 150M and 380M parameters with a 32K context, and the research behind it proves more pretraining compute stopped helping encoders long ago. Here is what that changes for your RAG pipeline.
