Statistics and Data Science Seminar: "Mitigating multiple descents: a general framework for model-agnostic risk monotonization"

Abstract: Recent empirical and theoretical analyses of several statistical learning models and procedures have led to the discovery of a peculiar behavior in high dimensions, referred to as double descent, whereby the prediction risk of a predictor is non-monotonic as a function of the asymptotic aspect ratio of the number of parameters versus the sample size. We develop a simple methodology based on cross-validation and sample splitting that takes as input a generic prediction procedure and returns a modified predictor whose out-of-sample risk is, asymptotically, free of any double or multiple descent behaviors. We analyze the asymptotic risk profiles returned by our methodology in the proportional asymptotic setting in which the feature size grows proportionally with the number of observations. By carefully choosing the size of the training and test sets, we demonstrate that, under mild assumptions, the risk of the resulting predictor is asymptotically monotonic in the asymptotic aspect ratio. Our results apply to a variety of learning problems and procedures. Joint work with: Pratik Patil, Arun Kumar Kuchibhotla, Yuting Wei and Matey Neykov.