>We use synthetic data heavily to create Nemotron-4-340B-Instruct: over 98% of our training data has been synthetically generated throughout our alignment process.
Very interesting to see synthetic data used so heavily during alignment.
Are there any known models that make heavy use of synthetic data during pretraining?