Solving Empirical Bayes via Transformers

doi:10.48550/arXiv.2502.09844

Paper

Solving Empirical Bayes via Transformers

Published Feb 14, 2025 · Anzo Teh, Mark Jabbour, Yury Polyanskiy

ArXiv

1

Citations

0

Influential Citations

PDF

Abstract

This work applies modern AI tools (transformers) to solving one of the oldest statistical problems: Poisson means under empirical Bayes (Poisson-EB) setting. In Poisson-EB a high-dimensional mean vector $\theta$ (with iid coordinates sampled from an unknown prior $\pi$) is estimated on the basis of $X=\mathrm{Poisson}(\theta)$. A transformer model is pre-trained on a set of synthetically generated pairs $(X,\theta)$ and learns to do in-context learning (ICL) by adapting to unknown $\pi$. Theoretically, we show that a sufficiently wide transformer can achieve vanishing regret with respect to an oracle estimator who knows $\pi$ as dimension grows to infinity. Practically, we discover that already very small models (100k parameters) are able to outperform the best classical algorithm (non-parametric maximum likelihood, or NPMLE) both in runtime and validation loss, which we compute on out-of-distribution synthetic data as well as real-world datasets (NHL hockey, MLB baseball, BookCorpusOpen). Finally, by using linear probes, we confirm that the transformer's EB estimator appears to internally work differently from either NPMLE or Robbins' estimators.

Preprint

Study Snapshot

Transformer models can outperform classical algorithms like NPMLE and Robbins' estimators in solving Poisson-EB problems, with small models (100k parameters) outperforming the best classical algorithm.

PopulationOlder adults (50-71 years)

Sample size24

MethodsObservational

OutcomesBody Mass Index projections

ResultsSocial networks mitigate obesity in older groups.

Solving Empirical Bayes via Transformers

References

Citations