Background

Foundation models fine-tuned on rheumatological clinical sequences achieve impressive average predictive accuracy for disease activity scores (DAS28, CDAI, SLEDAI). However, point predictions without calibrated uncertainty estimates are clinically dangerous — a confident but wrong DAS28 prediction could trigger inappropriate biologic escalation or premature tapering. Current approaches (MC dropout, deep ensembles) require distributional assumptions that may not hold in the heavy-tailed, regime-switching dynamics characteristic of autoimmune disease trajectories.

Hypothesis

We hypothesize that split conformal prediction applied to the residuals of a rheumatology foundation model will produce prediction intervals with guaranteed finite-sample marginal coverage (1-α) under the sole assumption of exchangeability, and that these intervals will exhibit clinically informative heteroscedasticity: wider intervals will concentrate at disease state transitions (remission↔flare), polypharmacy windows, and patients with rare immunophenotypes — precisely the scenarios where clinical vigilance is most needed.

Proposed Methodology

1. Calibration set construction: Reserve 20% of longitudinal rheumatology visits (n≥2,000 patients, ≥10,000 visits) as a calibration holdout, ensuring temporal ordering is respected via time-stratified splitting to approximate exchangeability.
2. Nonconformity score: Use the locally-weighted conformal score s_i = |y_i - ŷ_i| / σ̂_i, where σ̂_i is estimated from a secondary model trained on absolute residuals, enabling adaptive interval widths that scale with local prediction difficulty.
3. Coverage guarantee: For any user-specified α (e.g., 0.10), compute the (1-α)(1+1/n)-quantile of calibration scores to obtain prediction intervals with P(Y_{n+1} ∈ Ĉ) ≥ 1-α.
4. Clinical stratification analysis: Partition prediction intervals by (a) proximity to LLDAS/flare transition boundaries, (b) concurrent DMARD count, (c) HLA-DRB1 shared epitope dosage, and (d) anti-CCP/RF serostatus to characterize where the model is least certain.
5. Decision-theoretic integration: Define a clinical abstention rule — when the conformal interval spans a treatment-relevant decision boundary (e.g., DAS28 interval crosses 3.2 or 5.1), flag the prediction as unreliable and recommend additional clinical assessment before acting.

Testable Predictions

• P1: Conformal intervals will achieve empirical coverage within ±2% of nominal (1-α) across all patient subgroups, including rare phenotypes (anti-MDA5+, anti-SRP+) that are underrepresented in training data.
• P2: Interval width will be ≥40% larger at visits within 8 weeks of a disease state transition compared to stable-state visits (p<0.001, Wilcoxon rank-sum).
• P3: The clinical abstention rule will reduce false treatment escalation decisions by ≥25% compared to acting on point predictions alone, as measured in retrospective decision simulation.
• P4: Patients flagged by the abstention rule will have 3× higher rate of subsequent unexpected clinical events (hospitalization, organ damage accrual) than non-flagged patients.

Limitations

• Marginal coverage guarantees do not ensure conditional coverage per subgroup; achieving approximate conditional coverage requires sufficient calibration samples per stratum, which may be infeasible for very rare phenotypes.
• Exchangeability assumption is approximate in longitudinal data — covariate shift from evolving treatment protocols or secular trends could degrade coverage over time, necessitating periodic recalibration.
• The method quantifies model uncertainty, not irreducible aleatoric uncertainty from disease biology — wide intervals may reflect data sparsity rather than genuine clinical unpredictability.
• Computational overhead of maintaining calibration sets and recomputing quantiles is modest but non-zero in real-time clinical deployment.

Clinical Significance

Conformal prediction offers a rare combination in clinical AI: finite-sample validity without distributional assumptions. In rheumatology, where disease trajectories are inherently nonlinear and patient heterogeneity is extreme, knowing when not to trust the model is as valuable as the predictions themselves. The abstention rule transforms a black-box foundation model into a decision-support system that explicitly communicates its epistemic boundaries, aligning with regulatory expectations (FDA guidance on clinical decision support, EU AI Act high-risk requirements) and the ethical imperative to maintain physician authority over treatment decisions in autoimmune disease management.

RheumaAI Research • rheumai.xyz • DeSci Rheumatology