Background

When a RA patient fails first-line methotrexate, the clinician must choose among TNF inhibitors (adalimumab, etanercept, infliximab), IL-6 inhibitors (tocilizumab), T-cell costimulation blockers (abatacept), B-cell depletion (rituximab), and JAK inhibitors (tofacitinib, baricitinib). Current guidelines recommend "any approved biologic" — the sequence is largely trial-and-error.

Hypothesis

We propose that RA biologic sequencing can be formulated as a Markov Decision Process (MDP) where:

• States: (DAS28 category × current drug × treatment line × anti-drug antibody status × comorbidity profile)
• Actions: {switch to TNFi, switch to IL-6i, switch to CTLAi, switch to anti-CD20, switch to JAKi, dose escalate, add MTX}
• Transition probabilities: P(next state | current state, action) estimated from registry data
• Rewards: R = -DAS28(t) - 0.5×(Sharp progression) - 0.3×(adverse events) + 2.0×(remission achieved)
• Discount factor: γ = 0.95 (valuing long-term outcomes)

The optimal policy π* (solved via value iteration) will outperform the empirical "standard of care" sequence by ≥23% in cumulative discounted reward over 5 years.

Why MDP Over Simpler Models

1. Sequential decisions: Each treatment choice affects future options (immunogenicity, resistance)
2. State dependence: A patient who failed TNFi due to anti-drug antibodies has different optimal next step than one who failed due to primary non-response
3. Long-term optimization: Greedy strategies (always pick highest short-term response rate) sacrifice long-term outcomes
4. Bellman optimality: MDP provides mathematically provable optimal sequential strategy

Testable Predictions

1. MDP policy recommends different first biologic than guidelines in >40% of cases (based on comorbidity/ADA profile)
2. Cumulative DAS28-days-above-target over 5 years is 23% lower under MDP policy vs standard care
3. MDP identifies IL-6i as optimal first biologic for patients with high CRP + hepatic steatosis (avoiding JAKi cardiac risk)
4. For anti-CCP high-titer patients, MDP favors rituximab earlier than current guidelines suggest
5. Total biologic costs are equivalent (±5%) — the optimization is clinical, not financial

Implementation

• State space: ~2,400 states (discretized)
• Action space: 7 treatment decisions
• Transition matrices: estimated from BIOBADAMEX + CORRONA + RABBIT registries
• Solution: value iteration (convergence in <100 iterations)
• Validation: counterfactual evaluation on held-out registry patients

References

• Sutton RS, Barto AG. Reinforcement Learning. MIT Press, 2018.
• Smolen JS, et al. EULAR RA management recommendations. Ann Rheum Dis. 2023.
• Puterman ML. Markov Decision Processes. Wiley, 2014.
• Gottenberg JE, et al. Sequential biologic therapy in RA. Ann Rheum Dis. 2019.