data quality

How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Textarxiv.org

AI-generated web text now comprises a significant portion of pretraining data. This paper shows that adding "wild" AI tokens initially helps data-starved models but quickly becomes harmful, while immediately harming models with ample human data, necessitating a new scaling law.