Metabolic Syndrome Risk Validation
An external, longitudinal validation of two metabolic-syndrome risk scores on the US NHANES Linked Mortality File, benchmarked against established clinical equations (Framingham, ACC/AHA PCE, FINDRISC) with survey-weighted competing-risks survival analysis.
OVERVIEW
An external, pre-registered, longitudinal validation of two metabolic-syndrome risk scores (the triangular-areal-similarity RMRS and a B9 decision tree) on the US NHANES Linked Mortality File, benchmarked against the ACC/AHA Pooled Cohort Equations, Framingham 2008, and FINDRISC. Built in R (with one Python step) as a 15-step reproducible pipeline over a 17,031-adult cohort, it uses survey-weighted Fine-Gray competing-risks and Cox models, IPCW time-dependent AUC, competing-risks decision-curve analysis, and a 500-replicate bootstrap. The finding is nuanced: the clinical equations dominate for cardiovascular and all-cause mortality, while RMRS competes with FINDRISC for diabetes-related mortality, and an XGBoost ceiling plus a held-out CART refit localize the B9 tree's weakness to transportability rather than method. The manuscript and OSF pre-registration are included as drafts.
ARRIVED AS
Two recently published metabolic-syndrome risk scores were developed and tuned on Korean cohorts. The open question was whether they hold up out of sample: do they predict long-term mortality in a large, survey-sampled US population, and do they add anything over the clinical risk equations doctors already use? Answering that honestly means a pre-registered external validation, not a re-fit on convenient data.
Risk scores are usually celebrated on the data they were built on and rarely tested cleanly somewhere else. This project is an external validation: take two metabolic-syndrome risk scores from recent Korean-cohort papers and ask whether they generalize to a large US survey population followed for mortality, judged against the clinical equations already in use. The work is deliberately pre-registered so the analysis plan is fixed before the results are seen, and the headline is a nuanced, partly negative finding, which for a validation study is the point, not a disappointment.
WHAT I BUILT
- 01Implements five risk scores from scratch in R, the metabolic-syndrome-derived RMRS (a triangular-areal-similarity method) and a B9 decision tree, plus the ACC/AHA Pooled Cohort Equations, Framingham 2008, and FINDRISC as comparators, each with unit tests.
- 02Runs them on a NHANES 1999 to 2018 cohort (17,031 adults) linked to the 2019 Mortality File, for all-cause, cardiovascular, and diabetes-related mortality.
- 03Uses survey-weighted Fine-Gray competing-risks and Cox models, IPCW time-dependent AUC, competing-risks decision-curve analysis, and a 500-replicate PSU-cluster bootstrap for confidence intervals, the inference a clinical journal would expect.
- 04Diagnoses why a score underperforms: an XGBoost ceiling on the same five inputs and a CART refit on a held-out NHANES split separate 'the signal is weak' from 'the imported splits do not transport'.
WHAT CHANGED
- An honest, mixed result rather than a flattering one: for cardiovascular and all-cause mortality the established clinical equations dominate (Framingham 2008 reaches AUC 0.858 at 9.5 years for cardiovascular mortality versus 0.660 for RMRS), while for diabetes-related mortality RMRS competes with FINDRISC (AUC 0.752 vs 0.770) and adds modest incremental value (NRI and IDI confidence intervals exclude zero).
- Localizes the B9 decision tree's weakness to transportability: a CART refit on held-out NHANES data recovers RMRS-level discrimination, so the Korean-calibrated splits, not the tree method, are the problem.
- Pre-registered and fully reproducible: an OSF analysis-plan draft, a 15-step R/Python pipeline, renv-pinned dependencies, and a Makefile that regenerates the cohort and every result table from raw NHANES.
Data flow
click a stage
Download NHANES 1999 to 2018, link the 2019 Mortality File, and apply inclusion criteria to a 17,031-adult cohort.
COMPONENT
No component mapped to this stage.
Decisions, with the cost of each.
A decision without its trade-off is marketing. Each row says what was chosen, why, and what it gave up.
Pre-register and validate externally, do not re-fit
A score re-fit on new data almost always looks good; that is not validation. Fixing the analysis plan in advance and testing the published scores as-is on an independent population is the only way to learn whether they actually transport.
Re-fit the scores on NHANES (inflated, circular); report only the favorable outcome (cherry-picking).
Survey-weighted competing-risks survival, not plain Cox on raw data
NHANES is a complex survey, and cause-specific mortality competes with other causes. Survey weights plus Fine-Gray competing-risks models (with IPCW time-dependent AUC and a PSU-cluster bootstrap) are what make the estimates valid for the US population rather than the raw sample.
Unweighted Cox (biased for the population, ignores competing risks); logistic regression on an arbitrary horizon (throws away time).
Add an XGBoost ceiling and a held-out CART refit to explain failure
Knowing a score underperforms is not enough; the useful question is why. An XGBoost ceiling shows whether the five MetS inputs carry signal at all, and a CART refit on held-out NHANES shows whether the tree method or its imported splits is the problem, separating a weak signal from poor transportability.
Report the AUC gap and stop (no mechanism, no actionable conclusion).
The part that mattered.
The numbers behind the work, and the code that produced them.
- NHANES cohort
- 17,031 adults
- 1999-2018 + 2019 Linked Mortality File
- head-to-head
- 5 risk scores
- RMRS · B9 · PCE · Framingham · FINDRISC
- competing risks
- Fine-Gray
- survey-weighted, IPCW AUC, 500-rep bootstrap
- and reproducible
- Pre-registered
- OSF draft · renv · Makefile rebuild
# Robust Metabolic Syndrome Risk Score (RMRS), Shin et al. 2024 PeerJ CS.
# Triangular Areal Similarity over 5 MetS factors.
# f(x) = 0.5 + 0.5 * x / (1 + |x|); f(0)=0.5, f(-1)=0.25, f(1)=0.75
elliot_sigmoid <- function(x) {
0.5 + 0.5 * x / (1 + abs(x))
}
# Each raw measurement is z-scored so its diagnostic threshold maps to 0,
# then passed through the Elliot sigmoid so the threshold maps to 0.5.
scale_factor <- function(x, factor) {
thresholds <- list(glucose = 100, triglycerides = 150, ...)
# ... normalize relative to threshold, then elliot_sigmoid(...)
}
RMRS is reimplemented from the source paper rather than approximated: each metabolic-syndrome factor is normalized so its clinical diagnostic threshold sits at 0.5 after an Elliot sigmoid, and the five scaled factors feed the triangular-areal-similarity score. Implementing it faithfully is what makes the external validation fair.
weighted_finegray_hr <- function(df, score, time_col, status_col, ...) {
# Geskus weights: survival::finegray() builds the weighted risk set
fg <- survival::finegray(
survival::Surv(.time, .status_f) ~ .score + .cluster, data = df
)
# subdistribution hazard via weighted Cox on the Fine-Gray data
survival::coxph(
survival::Surv(fgstart, fgstop, fgstatus) ~ .score,
weights = fgwt, data = fg, ...
)
}
Cause-specific mortality competes with other causes of death, so the model estimates a subdistribution hazard the Fine-Gray way: survival::finegray() builds the weighted risk set, then a weighted Cox fit gives the hazard ratio. This is what keeps the comparison honest for competing outcomes in a survey sample.
✓ LEARNED
A clean external validation usually produces a mixed result, and that is the value: here the metabolic-syndrome scores trail the clinical equations for cardiovascular and all-cause mortality but compete for diabetes-related mortality.
Diagnosing failure beats reporting it: the XGBoost ceiling and held-out CART refit turned 'the B9 tree is worse' into 'the imported splits do not transport', which is an actionable conclusion.
Pre-registration plus renv and a Makefile means the whole study rebuilds from raw NHANES, which is what makes a negative or mixed finding trustworthy rather than dismissible.