Predicting soil NPK from spectra

A pre-registered comparison of 11 models and 6 preprocessing chains on the Open Soil Spectral Library (OSSL v1.2) · computed on the UKY workstation (Intel Core Ultra 9 285, 24 cores, NVIDIA RTX PRO 2000 Blackwell)

1. Framework

The design was fixed before any result was seen. Nothing was tuned on the test set.

1. Split. One stratified 70/30 split per target, seeded once and shared by every model and preprocessing. Identical train rows, identical test rows.
2. Experiment A — models. 11 algorithms, all on snv_sg input.
3. Experiment B — preprocessing. 6 chains × 4 models, same split.
4. Experiment C — cross-library transfer. Train on one library, test on another. This is the honest test of generalisation, and it is where everything falls apart.
5. Metrics. R², RMSE, MAE, bias, plus RPD and RPIQ because RPD assumes a normal target and extractable P and exchangeable K are both strongly right-skewed.
6. Interpretability. SHAP and LIME — reported separately, labelled for what they are. They explain a fitted model; they are not competitors in accuracy.

Preprocessing chains tested

raw snv snv_sg = SNV + Savitzky-Golay (w=11, p=2) snv_d1 = SNV + first derivative msc = multiplicative scatter correction msc_sg

SNV removes multiplicative light-scatter effects. Derivatives remove additive baseline drift and sharpen overlapping absorption bands. MSC rescales each spectrum against a reference.

2. The data — and one finding that changed the design

Targets were joined from OSSL soillab to spectra on id.layer_uuid_txt. Three things had to be established before modelling, and each one is a real property of the release rather than a nuisance to work around:

3. Results — Experiment A: models

All models on snv_sg, identical split. Sorted by R².

Total N (vis-NIR) — n_test = 3,600

TabPFN reaches R² = 0.911 (RPD 3.34). SVR and Cubist follow at 0.859, then the boosted trees cluster at 0.79–0.81. PLS — still the chemometrics default — lands last at 0.719.

Exchangeable K (vis-NIR) — n_test = 3,610

Much harder. TabPFN gets 0.509; everything else is at or below 0.381. No model clears RPD 1.5, i.e. all are in the "poor" band for practical use.

Extractable P (MIR) — n_test = 3,600

TabPFN 0.493, then LightGBM 0.338 and XGBoost 0.315. Note CNN1D is worst on P (0.106) despite MIR having the clearest physical signal.

4. Results — Experiment B: preprocessing

Same split, four models, six chains. Cell shading marks the best chain within each row.

Total N — R² by preprocessing and model

Exchangeable K

Extractable P (MIR)

5. Results — Experiment C: cross-library transfer

This is the test that matters for deployment and it is the one that fails. A model is trained on one spectral library and applied to another. Negative R² means worse than predicting the test set's own mean.

6. Interpretability — SHAP and LIME

Reading the profiles against known assignments:

• N concentrates on ~1408–1414 nm and 1882 nm, plus visible bands near 400 and 670 nm. The 1400 nm region is an OH/H₂O overtone and 1880 nm a water band; the visible bands are the humus/iron colour region that carries organic-matter information. Soil total N is largely bound in organic matter, so this is physically sensible.
• K lands on 1654, 1696, and 2496–2500 nm. Reported K absorption features in vis-NIR sit near 1400 and 1450 nm, and much of the 2450–2500 nm signal is clay (Al-OH) — clay is a major exchange-site host, so the model is plausibly picking up the exchange complex rather than K itself. That distinction is a hypothesis, not a claim.
• P (MIR) is dominated by 3628–3698 cm⁻¹ and 1554–1558 cm⁻¹, with 600 cm⁻¹ also salient. Note the model did not centre on the ~1030–1100 cm⁻¹ P-O region that would be the textbook expectation — consistent with the low accuracy and with P being largely present as occluded/sorbed species rather than free phosphate.

Do the predictions actually track the measurements?

SHAP says where a model looks; parity says whether the numbers are right. Below, TabPFN's predicted values against measured values on the held-out test set. Points on the dashed 1:1 line are perfect predictions.

7. Pros, cons and when to use each model

8. What actually follows from this

1. Use TabPFN first. It won every target without hyperparameter tuning, and its margin grew as the task got harder (N: +0.05 R² over the runner-up; K: +0.13; P: +0.16). This agrees with Barkov et al. (2026), who found TabPFN best across 85 pedometric regression tasks, and with Huang et al. (2026) on MIR.
2. The marginal returns are in preprocessing, not architecture. Switching from raw to snv_d1 bought XGBoost +0.09 R² on N and +0.15 on K — comparable to the gain from switching algorithms entirely.
3. Report cross-library results. Within-library accuracy overstates field performance badly. The transfer experiment is the honest one.
4. Manage expectations per element. N is predictable (RPD 3.4), P and K are not (RPD 1.4–1.5). That ordering matches the literature: total N is bound to organic matter and has real spectral expression; extractable P and K have weak or indirect responses.
5. If you need P, use MIR and expect a weak model. There is no vis-NIR route in OSSL, and MIR only reaches R² ≈ 0.5. The literature reports the same difficulty.
6. Treat SHAP as hypothesis generation. It tells you where to look, not what the chemistry is.

9. Caveats to state plainly

• Cap of 12,000 rows per target. Chosen so the comparison finished in one session and so the file could ship over the slow link. Absolute R² would likely rise with the full sets; the ranking is the result, and rankings are robust to this at these n.
• TabPFN was capped at 8,000 training rows (it is designed for "small data", and this keeps memory bounded). Every other model saw all 8,400. So TabPFN's win is not an artefact of easier data — it won on less.
• P mixes lab methods (Mehlich-3, ISO, Bray-1) because no single method has enough coverage with MIR. Method differences add noise to the target. The P number is therefore a lower bound on what a single-method dataset could achieve.
• CNN1D got one architecture and 40 epochs. A tuned CNN would likely do better; these numbers are not a verdict on deep learning for spectra, only on this untuned configuration.
• Hyperparameters were reasonable defaults, not tuned per model. Tree models in particular would gain from a search. TabPFN needs none, which is part of its appeal.
• Single split, not repeated CV. Split uncertainty is not quantified here. The earlier validation-methods work on the same library showed ±0.03 R² spread across repeated holdouts, so differences below ~0.03 should not be over-read.

10. Reproducing this

Scripts and data are on the workstation at ~/npk/. Everything ran there; Hermes drove it over the reverse SSH tunnel.

npk_compare.py       # the whole experiment (fixed split, 3 experiments)
make_figs.py         # the seven figures
npk_bench_small.npz  # N and K: 12,000 rows x 1051 vis-NIR bands
npk_p_mir.npz        # P: 12,000 rows x 1701 MIR bands
results_models.csv / results_preproc.csv / results_transfer.csv

11. References

1. Hollmann, N., Müller, S., Purucker, L., Krishnakumar, A., Körfer, M., Hoo, S.B., Schirrmeister, R.T., Hutter, F. (2025). Accurate predictions on small data with a tabular foundation model. Nature 637, 319–326. doi:10.1038/s41586-024-08328-6
2. Barkov, V., Schmidinger, J., Gebbers, R., Atzmueller, M. (2026). From field-scale to large-scale spectral libraries: Tabular foundation models in soil spectroscopy. arXiv:2608.00608. arxiv.org/abs/2608.00608
3. Huang, Y.-C., Minasny, B., et al. (2026). Zero-shot inference with Tabular Prior-data Fitted Network (TabPFN) for soil MIR spectral analysis. Geoderma 471, 117880. doi:10.1016/j.geoderma.2026.117880
4. Schmidinger, J., et al. (2026). A case for kriging-based spatial features with TabPFN in digital soil mapping. Computers and Electronics in Agriculture. S0168169925014589
5. Safanelli, J.L., Hengl, T., Parente, L.L., Minarik, R., Bloom, D.E., Todd-Brown, K., Gholizadeh, A., Mendes, W.S., Sanderman, J. (2025). Open Soil Spectral Library (OSSL): Building reproducible soil calibration models through open development and community engagement. PLoS ONE 20(1), e0296545. doi:10.1371/journal.pone.0296545
6. Chang, C.-W., Laird, D.A., Mausbach, M.J., Hurburgh, C.R. (2001). Near-infrared reflectance spectroscopy–principal components regression analyses of soil properties. Soil Science Society of America Journal 65, 480–490. (RPD interpretation bands.)
7. Gozukara, G., et al. (2025). Prediction accuracy of pXRF, MIR, and Vis-NIR spectra for soil properties — a review. Soil Science Society of America Journal 89, e0028.
8. Morellos, A., et al. (2016). Machine learning based prediction of soil total nitrogen, organic carbon and moisture content by using VIS-NIR spectroscopy. Biosystems Engineering 152, 104–116. doi:10.1016/j.biosystemseng.2016.04.018
9. Mokere, R., et al. (2026). Soil spectroscopy improves mid-infrared soil property prediction. Frontiers in Soil Science. doi:10.3389/fsoil.2026.1760011 (reports unreliable spectroscopic prediction of soil phosphorus.)
10. Grinsztajn, L., Oyallon, E., Varoquaux, G. (2022). Why do tree-based models still outperform deep learning on typical tabular data? NeurIPS Datasets and Benchmarks.
11. Lundberg, S.M., Lee, S.-I. (2017). A unified approach to interpreting model predictions. NeurIPS. (SHAP.)
12. Ribeiro, M.T., Singh, S., Guestrin, C. (2016). "Why should I trust you?" Explaining the predictions of any classifier. KDD. (LIME.)
13. Quinlan, J.R. (1992). Learning with continuous classes. 5th Australian Joint Conference on Artificial Intelligence. (Cubist.)

Generated on the UKY workstation via the Hermes reverse-SSH link · 111 configurations · 65 minutes · all numbers from results_*.csv, not hand-entered.