Development and Application of A Measure of Prediction Accuracy for Binary and Censored Time to Event Data
Clinical preventive care often uses risk scores to screen population for high risk patients for targeted intervention. Typically the prevalence is low, meaning extremely unbalanced classes. Positive predictive value and true positive fraction have been recognized as relevant metrics in this imbalanced setting. However, for commonly used continuous or ordinal risk scores, these measures require a subjective cut-off threshold value to dichotomize and predict class membership. In this talk, I describe a summary index of positive predictive value (AP) for binary and event time outcome data. Similar to the widely used AUC, AP is rank based and a semi-proper scoring rule. We also study the behavior of incremental values of AUC, AP and the strict proper scoring rule scaled Brier score (sBrier) when an additional risk factor Z is included. It is shown that the incremental values agreement between AP and sBrier increases as the class unbalance increases, while the agreement between AUC and sBrier decreases as class unbalance increases. Under certain configurations, the changes in AP and sBrier indicate worse prediction performance when Z is added to the risk profile, while the changes in AUC are almost always favor the addition of Z. Several real world examples are used throughout the talk to illustrate and contrast these metrics.
Nonparametric failure time with Bayesian Additive Regression Trees
Bayesian Additive Regression Trees (BART) is a nonparametric machine learning method for continuous, dichotomous, categorical and time-to-event outcomes. However, survival analysis with BART currently presents some challenges. Two current approaches each have their pros and cons. Our discrete time approach is free of precarious restrictive assumptions such as proportional hazards and Accelerated Failure Time (AFT), but it becomes increasingly computationally demanding as the sample size increases. Alternatively, a Dirichlet Process Mixture approach is computationally friendly, but it suffers from the AFT assumption. Therefore, we propose to further nonparametrically enhance this latter approach via heteroskedastic BART which will remove the restrictive AFT assumption while maintaining its desirable computational properties.
