Senior Honors Thesis Presentation: "Parallel Penalized Quantile Regression Via ADMM"

Abstract: Penalized quantile regression is a robust regression analysis method for analyzing data with heterogeneity. Instead of estimating the conditional mean of the response variable, it estimates the conditional quantiles of the response variable. The QR-ADMM (quantile regression alternating direction method of multipliers) algorithm, proposed by Yu et al, is one amongst many methods that solve penalized quantile regression problems. The ADMM algorithm solves convex optimization problems by breaking them into smaller pieces and handling each separately in a distributed fashion. However, the computation bottleneck of QR-ADMM lies in its double-loop computation from numerical methods such as coordinate descent. By rewriting the original penalized quantile regression problem into an equivalent form, the QPADM algorithm, proposed by Yu et al, only requires single-loop computation. The QPADM algorithm utilizes parallelization to achieve a significant performance boost from the QR-ADMM algorithm. As suggested by Yu et al, implementing the QPADM algorithm in distributed frameworks such as Apache Spark would benefit from the parallelism across multiple working nodes. In this paper, we propose the Scala-Spark implementation of the QPADM algorithm. Compared to the Rcpp implementation of QPADM, the Scala implementation benefits from the parallel processing power of Spark, and therefore demonstrates more favorable performance.

Host: Nan Lin