Many factors limit the ability of resident selection committees to reliably determine which medical student applicants will be top performers in their internship and residency programs. Resident selection committees expend significant time and effort to read applications and then interview and score applicants, all with the goal of creating a rank order list (ROL) for the National Residency Matching Program. Although much of the information used in this process is outside the control of the selection committee, one factor they can control is how they use committee member scores to create the ROL. ROLs are typically generated using the average score assigned by committee members. With this approach, a single harsh faculty member can strongly influence the rank order of applicants if he or she scores only some of the applicants.The author of this commentary discusses the challenges inherent in the creation of ROLs and examines a new approach, described by Ross and Moore in this issue, that can eliminate the problem of biased scoring. ROLs created using this new ROSS-MOORE (Recruitment Outcomes Simulation System-Moore Optimized Ordinal Rank Estimator) approach will better represent the rank ordering of each faculty member of the selection committee. However, ROLs will remain poorly predictive of future performance in internship and residency programs until evaluative data supplied by medical schools are more accurate in predicting which students will become excellent physicians.