Evaluating the Performance of the Time DependentNet Reclassification Improvement Under Different Settings

Abstract

Objective: In recent years, new measures have been proposed to evaluate the improvement in classification performance by the addition of a new risk factor to a baseline risk model that includes a set of baseline risk factors. Therefore, net reclassification improvement (NRI) and integrated discrimination improvement (IDI) methods have been utilized in medicine and these metrics have been adapted to time-to-event data in recent years. The aim of this study is to evaluate the performance of the time dependent NRI (NRI(t)) under different scenarios. Material and Methods: Various datasets were composed according to the purpose of each different scenario which were censoring rates (20%, 40%, and 60%), sample sizes (30, 50, 250, 500, and 1000) and number of risk categories (2, 3, and 4). Also, follow-up time was generated from Weibull distribution. All analyses and data generation process were performed using R version 3.4.3. Results: When number of risk categories was specified as three or four, the performance of improved model was better than two-category version. As censoring rate increased, the performance of improved model was decreased. Also, as expected, the performance of the model improved as sample size increased. In general, NRI(t) values were stable for two-category version independently of sample size and censoring rate through follow-up times. But especially for large sample sizes, the performance was higher in early time for three or four risk categories. Conclusion: In this study, it was found that as censoring rate decreased and number of risk categories and sample size increased, the NRI(t) improved.