Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theory

doi:10.48550/arXiv.2410.08942

Paper

Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theory

Published Oct 11, 2024 · Aymane El Firdoussi, M. Seddik, Soufiane Hayou

ArXiv

UNKNOWN SJR score

0

Citations

0

Influential Citations

Full textSemantic Scholar

Abstract

Synthetic data has gained attention for training large language models, but poor-quality data can harm performance (see, e.g., Shumailov et al. (2023); Seddik et al. (2024)). A potential solution is data pruning, which retains only high-quality data based on a score function (human or machine feedback). Previous work Feng et al. (2024) analyzed models trained on synthetic data as sample size increases. We extend this by using random matrix theory to derive the performance of a binary classifier trained on a mix of real and pruned synthetic data in a high dimensional setting. Our findings identify conditions where synthetic data could improve performance, focusing on the quality of the generative model and verification strategy. We also show a smooth phase transition in synthetic label noise, contrasting with prior sharp behavior in infinite sample limits. Experiments with toy models and large language models validate our theoretical results.

Preprint

Study Snapshot

Pruning synthetic data can improve performance in large language models, with smooth transitions in synthetic label noise.

PopulationOlder adults (50-71 years)

Sample size24

MethodsObservational

OutcomesBody Mass Index projections

ResultsSocial networks mitigate obesity in older groups.

Sign up to use Study Snapshot

Consensus is limited without an account. Create an account or sign in to get more searches and use the Study Snapshot.

Create an account

Paper

Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theory

Sign up to use Study Snapshot

References

Alignment with Preference Optimization Is All You Need for LLM Safety

Alignment techniques, such as noise contrastive alignment (Safe-NCA), can effectively enhance LLM safety and performance, achieving competitive results with state-of-the-art models.

The Llama 3 Herd of Models

Llama 3 is a set of language models that supports multilinguality, coding, reasoning, and tool usage, delivering comparable quality to leading models like GPT-4 on various tasks.

Gemma 2: Improving Open Language Models at a Practical Size

Gemma 2 improves open language models by incorporating local-global attentions, group-query attention, and knowledge distillation, delivering best performance for their size and offering competitive alternatives to models 2-3 times bigger.

Falcon2-11B Technical Report

Falcon2-11B is a strong foundation model with strong generalization across tasks, and its multimodal counterpart, Falcon2-11B-vlm, achieves higher average scores compared to open-source models of similar size.

Qwen2 Technical Report

Qwen2 is a comprehensive language model series that surpasses prior open-weight models and exhibits competitive performance against proprietary models in various benchmarks.

Citations