Deterministic Tabular Learning with TabM (Huber) and OOF Blending: A Leakage-Safe, Math-First Protocol

Abstract

Tabular prediction in operational settings hinges on three issues: (1) determinism for credible comparisons; (2) leakage safety across preprocessing, validation, and ensembling; (3) robustness to moderate outliers. We revisit the multi-head MLP architecture TabM and show that, trained with a Huber objective and embedded in a strictly out-of-fold (OOF) pipeline, it forms a strong, fully deterministic baseline for regression and classification. Methodologically, our protocol (1) fits all transformers on the training split only; (2) trains TabM foldwise with early stopping under Huber (SmoothL1) loss; (3) performs OOF-only linear blending with a closed-form residual correction to capture diversity without search noise. Empirically, on the Student Habits & Performance regression task TabM(Huber) outperforms XGBoost, histogram GBDT, SVR, Random Forest, and KNN under an identical seeded split; on the UCI Wine and Wisconsin Breast Cancer benchmarks the pipeline yields near-ceiling held-out performance with transparent diagnostics (confusion matrices, ROC/PR curves). We release reproducibility metadata and advocate OOF-only, train-only preprocessing as a minimal contract for leakage-safe tabular evaluation.