Anthony Caterini
Layer 6
Room: M3 3127
What are Tabular Foundation Models and How Do They Work?
Tabular data is central to statistical practice, but its diversity has made it challenging to develop models that transfer across datasets. Tabular foundation models (TFMs) address this challenge by pretraining neural networks across many prediction tasks. Most use in-context learning: a transformer receives labelled observations from a new dataset as input and predicts responses for new observations without updating its parameters. On many small and medium-sized datasets, these models rival or outperform gradient-boosted trees such as XGBoost, often with substantially less task-specific tuning. I will introduce how TFMs are trained and used, highlighting design choices from our work on TabDPT at Layer 6. I will then turn to a less settled question: why do these models generalize so effectively? An influential framework interprets synthetic pretraining as learning to approximate Bayesian posterior predictive distributions under a chosen prior. This interpretation does not by itself explain strong performance on real datasets that differ from the synthetic distributions used for pretraining. Our recent work explores a complementary, retrieval-based perspective: in a controlled setting, predictive performance is closely associated with a model’s ability to retrieve relevant examples from its context. I will discuss the evidence for this perspective, its limitations, and the open questions it raises about what TFMs learn during pretraining and how they adapt to new tasks.