I am an HCI researcher in the Human-Computer Interaction Institute at Carnegie Mellon University. I am also a Siebel Scholar and a fellow of the PIER Program in Interdisciplinary Education Research. I have previously worked with the FATE (Fairness, Accountability, Transparency, and Ethics in AI) group at Microsoft Research.
I lead or co-lead several research projects focused on answering the broad question: 
"How can we empower people to meaningfully contest and shape the behavior of the algorithms that impact their lives?"
To explore this question, I partner with relevant stakeholders to co-design systems that draw upon complementary strengths (and mitigate complementary biases) of human and AI decision-makers
I create new design and prototyping methods to engage stakeholders throughout the development process
, and I conduct field experiments to understand the causal impacts of these human–AI systems in the real world. 
Much of my work focuses specifically on understanding and designing for the evolving roles of K-12 teachers as AI increasingly enters the classroom. In this line of research, I work closely with both teachers and students to explore how human and AI instruction might be most effectively combined.
My research has won five paper awards from the International Artificial Intelligence in Education Society (IAIED) and the International Society of the Learning Sciences (ISLS). This research is funded by the National Science Foundation (NSF), the Institute of Education Sciences (IES), and CMU's Metro21 Smart Cities Institute (Metro21).
You can find my publications on my Google Scholar page. If you’d like to know more about my work or explore opportunities for collaboration, please get in touch!
Representative papers

Holstein, K., Wortman Vaughan, J., Daumé III, H., Dudík, M., Wallach, H. (2019). Improving fairness in machine learning systems: What do industry practitioners need?  In Proceedings of the ACM CHI Conference on Human Factors in Computing Systems (CHI’19). ACM.  
[ACM]   [supp. materials]   [pdf]
Holstein, K., McLaren, B. M., & Aleven, V.  (2018). Student learning benefits of a mixed-reality teacher awareness tool in AI-enhanced classrooms. In Proceedings of the 19th International Conference on Artificial Intelligence in Education (AIED 2018). LNAI 10947 (pp. 154-168). Springer: Berlin.  
*Best Paper Award*   [pdf]

Holstein, K., McLaren, B. M. & Aleven, V.  (2019). Co-designing a real-time classroom orchestration tool to support teacher–AI complementarity. Journal of Learning Analytics (JLA). 
[link]   [pdf]
Holstein, K., McLaren, B. M. & Aleven, V.  (2019). Designing for complementarity: Teacher and student needs for orchestration support in AI-enhanced classrooms. In Proceedings of the 20th International Conference on Artificial Intelligence in Education (AIED 2019)  
*Best Paper and Best Student Paper Nominee*   [link]   [pdf]
Back to Top