Jingyan Wang “Understanding and Improving Evaluation: People, Algorithms, and Design”

Abstract:
High-stakes evaluation problems – to estimate the quality of items or people – arise in many real-world applications such as admissions, grading, and hiring. My research focuses on understanding and improving evaluation along the dimensions of accuracy, fairness, efficiency, and reliability. In this talk, I describe bias mitigation in a sequential setting, where an evaluator rates candidates in an online, irrevocable fashion. Motivated by the psychology literature, I propose a natural model for the evaluator’s rating process that captures the lack of calibration inherent to this task. Various facets of the proposed model are supported by theoretical results and crowdsourcing experiments. Under this model, I then study bias correction by posing it as a statistical inference problem. I propose a near-linear online algorithm and show that it is information-theoretically optimal, outperforming the de facto approach of using the ranking induced by the raw scores. To conclude the talk, I briefly discuss tradeoffs involved in the design of reviewer allocation schemes in large-scale evaluation tasks, and my outreach efforts in mitigating biases in academic practices.

Bio:
Jingyan Wang is a Ronald J. and Carol T. Beerman President’s postdoctoral fellow in the H. Milton Stewart School of Industrial and Systems Engineering at Georgia Institute of Technology. She received her Ph.D. from the School of Computer Science at Carnegie Mellon University, advised by Nihar Shah, and her B.S. in Electrical Engineering and Computer Sciences with a minor in Mathematics from the University of California, Berkeley. Her research interests lie in understanding and improving high-stakes decision-making systems such as those involving hiring and admissions, using tools from statistics and machine learning. She is the recipient of the Best Student Paper Award at AAMAS 2019, and was selected as a Rising Star in Data Science at the University of Chicago in 2022.

Details

• Date: Tuesday, February 28
• Time: 10:00 am - 11:00 am
• Location: 32-G882 Hewlett

Host

• Guy Bresler
• Email: fern@csail.mit.edu