🧟 The Silent Bug: how one rename killed a whole pipeline

🔬 Run this autopsy on real soil data: download the exercise kit — 150 real soil samples (USDA-KSSL / ICRAF-ISRIC library via the Open Soil Spectral Library, MIT license), ASD-style spectra 350–2500 nm × 2151 bands + 7 lab properties (SOC, pH, texture, CEC, BD), and a guided exercise.R with guess-first checkpoints. Reproduces the crash on the documented input path and the silent contract violation on the accidental one.

0. The crime scene in 15 seconds

The MLSP package ships a helper that is supposed to join lab measurements (SOC, pH, clay…) with spectral reflectance columns (2151 wavelength bands):

merged <- merge_of_lab_and_spectrum(soil, spec)   # documented entry point

Follow the manual exactly → it crashes instantly:

And here is the twist that makes this bug worth studying: the crash is the good outcome. The scary versions of this bug class fail silently (Stage 1's recycling lesson: exact multiples make no warning). Here R at least screams. Our job: find the exact line that made it scream.

1. What the function receives

Two tables, per the package's own manual:

The intended join key on both sides is LAB_NUM (sample 1 ↔ sample 1, etc.). One sample id per row, no duplicates. So far, so innocent.

2. The murder weapon: aggregate()'s habit 🍌

The package then does this (this is the real line 50 of its source):

NS_Spectral <- aggregate(x = data, by = list(data$Wavelength), FUN = mean)
# columns now: Group.1 | LAB_NUM | Wavelength | X350 …   (3 id-like cols!)

colnames(NS_Spectral)[1] <- "LAB_NUM"   # renames the COVER PAGE, not what you think

Freeze right there. The rename didn't replace anything meaningful — it re-labelled the cover page Group.1 → LAB_NUM, while the real original LAB_NUM sat untouched in position 2. Consequence:

Watch it happen — column surgery, live

These are the actual column names before/after. Nothing is deleted; a wrong one is merely relabelled:

3. Autopsy — reproduce it line by line in R

agg <- aggregate(x = spec, by = list(spec$Wavelength), FUN = mean)
names(agg)[1:3]
[1] "Group.1" "LAB_NUM" "Wavelength"   # ← the guest column

colnames(agg)[1] <- "LAB_NUM"            # the fatal line (line 50)
sum(duplicated(colnames(agg)))
[1] 1                                     # ← one DUPLICATE name: LAB_NUM ×2

m2 <- try(merge(soil, agg, by = "LAB_NUM"), silent = TRUE)
class(m2)
[1] "try-error"                           # same error as the full pipeline ✔

agg_bersih <- aggregate(x = spec, by = list(spec$Wavelength), FUN = mean)
colnames(agg_bersih)[1] <- "Wavelength_fix"   # NOT "LAB_NUM"
mg <- merge(soil, agg_bersih, by = "LAB_NUM")  # ← JALAN! (runs!)
dim(mg)
[1] 150 2161     # 150 rows ✔ — merge works; 2161 cols = leftover junk to clean (Stage 3 PR)

4. The fix (minimal-diff, in your own copy)

Never edit a package. Copy the function, fix the rename logic, keep everything else identical:

merge_of_lab_and_spectrum_fix <- function(soil_data, data_NaturaSpec_cleaned) {
  NS_Spectral <- aggregate(x = data_NaturaSpec_cleaned,
                           by = list(data_NaturaSpec_cleaned$Wavelength), FUN = mean)
  colnames(NS_Spectral)[1] <- "Wavelength"                        # 1. honest name
  NS_Spectral <- NS_Spectral[, !duplicated(colnames(NS_Spectral))] # 2. drop dup LAB_NUM
  Soil_NS_Spectral <- merge(soil_data, NS_Spectral, by = "LAB_NUM") # 3. unambiguous key
  Soil_NS_Spectral <- Soil_NS_Spectral[, -9]        # 4. drop the stray Wavelength col
  soil <- Soil_NS_Spectral[, 1:8]
  vnir <- Soil_NS_Spectral[, -c(1:8)]
  vnir.matrix <- as.matrix(vnir[, -1])              # drop the LAB_NUM that came along
  list(soil = soil, vnir.matrix = vnir.matrix)      # … rest identical to the original
}

5. Build the bug yourself — interactive

The mini-spec built in Stage 1/2 uses n samples and a scaled-down band count, but the mechanism (and every number the package prints) is identical. Push the slider and create the exact data the package chokes on:

6. Loud vs silent — the two failure modes

📋 Quiz yourself (3 quick checks)