Gradient-boosted trees still set the baseline on tables, and the baseline is moving

In 2022 Léo Grinsztajn, Edouard Oyallon and Gaël Varoquaux assembled 45 tabular datasets from varied domains and pitted standard and newly proposed deep learning methods against tree-based models such as XGBoost and random forests. They ran hyperparameter searches on a scale that is unusual in this literature, releasing every point from a 20,000 compute-hour search, so that nobody could say the trees had won because the networks were under-tuned. Their conclusion in the abstract was plain: tree-based models remain state-of-the-art on medium-sized data, roughly ten thousand rows, even before counting their speed advantage.

That paper became the standard citation for a piece of practitioner folklore: when the data is a table, start with gradient-boosted trees. Four years later the folklore is still a good default, with an asterisk that grows each quarter. The asterisk comes from the field's own results, so this essay takes two passes at the question. The first asks why trees do well on tables, because the answer tells you when the advice holds. The second asks what has changed since 2022, because the first author of the "trees win" paper is now the first author of a report on a tabular foundation model, and that fact summarizes the situation better than any argument could.

Tables are not images

Deep networks earned their reputation on data with a spatial or sequential structure that a network can exploit by design: pixels near each other belong together, words earlier in a sentence affect words later on. A table has none of that. Its columns are heterogeneous, each with its own meaning and units: age in years, income in reais, a postal code, a count of late payments, a yes-or-no flag. Reordering the columns changes nothing about the problem. The structure that matters is the identity of each column and a handful of interactions among them.

Grinsztajn's team traced the gap to three inductive biases, and they read like a description of the typical business table. The first is tolerance for uninformative features. Real tables carry columns that are noise or nearly so, such as identifiers, free-text codes and fields that were never filled in properly. A tree ignores a column that never helps a split. A neural network mixes every input into every hidden unit and has to learn to ignore the junk, which costs data.

The second is that the orientation of the data matters. A tree splits on one column at a time, so it treats each column as its own meaningful axis. A standard multilayer perceptron is nearly indifferent to a rotation of the input space, and in the authors' experiments randomly rotating the features degraded tree-based models far more than it degraded neural networks. That asymmetry is evidence that the column identities carry information which trees use and networks throw away.

The third is the ability to learn irregular functions. Business rules and real-world processes are full of thresholds. A credit policy treats an applicant with 49,000 in income differently from one with 51,000. A tax bracket, a pricing tier, an eligibility cutoff and a regulatory limit are all steps, and the target as a function of the inputs can jump. A tree represents a step with a single split. A network built from smooth activations approximates it with a steep ramp, which takes more units and more data. The tendency has a name in the literature. Nasim Rahaman and colleagues showed in 2019 that networks trained by gradient descent learn low-frequency components of a target function before high-frequency ones, a spectral bias toward smooth solutions that persists even when the truth is not smooth. A sharp step is a high-frequency feature, and the network gets to it last.

The unglamorous advantages

Tables come with practical irritations that the libraries around boosted trees have spent a decade absorbing. Missing values are routine, and XGBoost and LightGBM learn at each split which branch a missing value should take, so no imputation step is needed. High-cardinality categorical columns such as postal codes, merchant identifiers and product SKUs are routine as well. CatBoost, described by Liudmila Prokhorenkova and colleagues at NeurIPS 2018, handles them with an ordered scheme for target statistics that avoids the leakage a naive version would introduce. The same table fed to a neural network needs missing-value handling, scaling of skewed numeric columns and an embedding layer for each categorical column, each of which is a place for a mistake.

Operations favor trees too. A boosted model trains in minutes on a CPU, scores a row in a tiny fraction of the time a transformer needs, occupies megabytes, and can be exported to run inside a database or on a device. Retraining weekly is cheap enough to be boring. Cost per prediction matters little at a thousand predictions a day and a great deal at a hundred million, which is the scale some fraud scoring and ad ranking systems operate at.

What the larger studies add

A fair objection is that 45 datasets is a narrow sample. In 2023 Duncan McElfresh and colleagues compared 19 algorithms on 176 datasets, a much larger study, and their title posed a question: when do neural nets outperform boosted trees on tabular data? Their answer was that the debate is overemphasized. On many datasets the difference between the two families is negligible, or light hyperparameter tuning of a GBDT matters more than the choice between families. They found that gradient-boosted trees handle skewed and heavy-tailed feature distributions and other irregularities better than networks do.

The practical content of that result is about where effort goes. A team that spends a month searching for the right network architecture on a tabular problem is, on average, spending effort in the place where it makes the least difference. An afternoon spent cleaning the data, fixing leakage in the features and tuning a boosted model with early stopping usually returns more.

The baseline is moving

The 2023 study contained a finding that has turned into a trend. Among the algorithms it tested was TabPFN, a transformer pretrained on a large number of synthetic tabular datasets generated from a prior over data-generating processes. At inference time, instead of training a model, TabPFN is given the labeled training rows as context and predicts the test rows in a single forward pass, in the way a language model learns from examples in its prompt. It was limited to training sets of about three thousand rows, and even so it came out best on average.

In January 2025 a Freiburg-led team published a successor in Nature. The paper reports that the model outperforms all previous methods on datasets of up to ten thousand samples by a wide margin, with substantially less training time, and the headline comparison was that in 2.8 seconds it beat an ensemble of the strongest baselines tuned for four hours on classification. In November 2025 the team released TabPFN-2.5, revised in February 2026, for up to 50,000 rows and 2,000 features, with a claim of leading the TabArena benchmark and matching a four-hour tuned AutoGluon ensemble. Against default XGBoost it reports win rates of 100% on small and medium classification datasets and 87% on larger ones. It also describes a distillation step that converts the model into a compact network or tree ensemble, with much lower latency.

Why would a pretrained model beat a tuned ensemble on small data in particular? A reasonable account is that when there are only a few hundred rows, the main problem is deciding how much structure the data can support, and a model that has seen a very large number of synthetic tables has a better prior than a tree ensemble starting from nothing. A boosted model has to learn from those few rows which interactions matter and how smooth the relationships are, so it overfits or underfits depending on its tuning. As the number of rows grows, the prior matters less, which matches the pattern of the claims: the advantage is largest on small tables and shrinks as they get bigger. Note also what the 2.8-second comparison counts. The model does no gradient training on your table; the time is inference over the whole training set, which moves the cost to prediction time.

Independent evidence is more cautious. TabArena, a living benchmark presented by Nick Erickson, Frank Hutter and colleagues at NeurIPS 2025, reports that GBDTs remain strong on practical tabular datasets, that deep learning methods have caught up when given larger time budgets and ensembling, and that foundation models do well on smaller datasets. Ensembles across model families set the state of the art. The authors also warn that validation procedure strongly affects how well models reach their potential, and that some deep learning models are over-represented in the best ensembles because of overfitting to the validation set.

Several limits apply. Most of the headline numbers come from the developers' own benchmarks, even when the benchmark itself is independent, and replication by other groups is the proper test. Training-set size is capped, so large tables remain the territory of boosted trees. A model that conditions on the training set at prediction time carries that set into every inference call, which affects memory and latency, and is the reason for the distillation step. And the comparison that matters in production is rarely accuracy alone.

What to do on Monday

Evaluation deserves more thought than the model. A random train-test split flatters any model on data that drifts over time, because the test rows come from the same period as the training rows and share its quirks. For anything with a time dimension, such as churn, demand, default or fraud, hold out the most recent months as the test set and train on what came before. Differences between model families are often smaller than the difference between a random split and a time-based split on the same model, and a comparison that uses the wrong split can crown the wrong winner.

The two passes combine into a procedure that is cheap to follow.

1. Build the baseline first: a boosted-tree model (XGBoost, LightGBM or CatBoost) with early stopping and a sensible validation split. Budget an afternoon.
2. If the table has under roughly 50,000 rows, run a tabular foundation model on the same split, which takes minutes. If it wins by more than the noise in your validation estimate, treat that as a real signal.
3. Ensemble whatever is best. The TabArena result that cross-family ensembles set the state of the art means a blend is rarely worse than its best member.
4. Only then consider a custom neural architecture, and only for a reason the data suggests, such as very high-cardinality categorical columns or a need to share representations across tasks.

Where the table has a text column, an LLM can earn a place in the pipeline as a feature extractor: embed the text and feed the vectors, or a few derived labels, into the boosted model alongside the other columns. Asking an LLM to predict directly from serialized rows is a different matter. TabLLM, from a team around David Sontag, found a fine-tuned model competitive with strong baselines including gradient-boosted trees in the very-few-shot regime, where labels are scarce and the model's prior knowledge helps. The paper's case is made for that regime, and it does not extend to tables with thousands of labeled rows.

Where interpretability enters

Regulated decisions add a constraint that favors the older tools. Boosted-tree libraries support monotonic constraints, which force the prediction to move only upward as a chosen input increases. That lets a credit model guarantee that a higher income never lowers the score, which credit model reviewers often require. They also work with attribution methods such as SHAP that produce a per-decision explanation. In Brazil, Article 20 of the LGPD gives data subjects the right to request review of decisions made solely by automated processing that affect their interests, including credit profiles, and to ask for clear information on the criteria used. A model whose behavior can be constrained and explained makes that obligation easier to meet, and any replacement should be judged on those terms as well as on accuracy.

The question to ask of a new tabular method is the one that the 2022 paper implicitly asked, and the answer has to come from your data: on my table, at my size, against a well-tuned boosted model, by how much does it win, and what does the win cost in latency, memory and explainability? A method that cannot answer that has not earned a place in the pipeline yet.

Sources

1. Grinsztajn, L., Oyallon, E., Varoquaux, G. (2022). Why do tree-based models still outperform deep learning on tabular data? NeurIPS 2022 Datasets and Benchmarks. arXiv:2207.08815.
2. McElfresh, D., Khandagale, S., Valverde, J., et al. (2023). When Do Neural Nets Outperform Boosted Trees on Tabular Data? NeurIPS 2023 Datasets and Benchmarks. arXiv:2305.02997.
3. Hollmann, N., Müller, S., Purucker, L., et al. (2025). Accurate predictions on small data with a tabular foundation model. Nature 637, 319-326.
4. Grinsztajn, L., et al. (Prior Labs) (2025, revised 2026). TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models. arXiv:2511.08667.
5. Erickson, N., Purucker, L., Tschalzev, A., et al. (2025). TabArena: A Living Benchmark for Machine Learning on Tabular Data. NeurIPS 2025 Datasets and Benchmarks. arXiv:2506.16791.
6. Hegselmann, S., Buendia, A., Lang, H., Agrawal, M., Jiang, X., Sontag, D. (2023). TabLLM: Few-shot Classification of Tabular Data with Large Language Models. arXiv:2210.10723.
7. Chen, T., Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. KDD 2016. arXiv:1603.02754.
8. Brasil. Lei nº 13.709/2018 (Lei Geral de Proteção de Dados Pessoais), art. 20.
9. Rahaman, N., Baratin, A., Arpit, D., et al. (2019). On the Spectral Bias of Neural Networks. ICML 2019. arXiv:1806.08734.
10. Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., Gulin, A. (2018). CatBoost: unbiased boosting with categorical features. NeurIPS 2018. arXiv:1706.09516.