Unlock the full potential of state-of-the-art multilingual LLMs with FineWeb2, a groundbreaking 20 terabyte (5 billion document) dataset. This new pre-training data is generated by a revolutionary curation pipeline that automatically adapts to support any language. Overcoming the inherent difficulty of tailoring filtering and deduplication for a large number of languages, FineWeb2 has been scaled to over 1000 languages using Common Crawl snapshots. It produces more performant models than prior datasets for non-English corpora and includes a principled approach to rebalance datasets for additional performance uplift. Access the released dataset, pipeline, training, and evaluation codebases today!

Examine the rapid progress of LLMs, the rise of open-source models, and implications for research evaluation. Understand the evolving role of economists in the age of ever-advancing AI