I am an Assistant Professor in the Human-Computer Interaction Institute at Carnegie Mellon University, a Siebel Scholarand a fellow of the PIER Program in Interdisciplinary Education Research. I have previously worked with the FATE (Fairness, Accountability, Transparency, and Ethics in AI) group at Microsoft Research.
I am the PI or co-PI on several projects focused on the design, development, and evaluation of AI systems that complement and bring out the best of human ability in work that people find personally meaningful.
To do this, I partner with relevant stakeholders to co-design systems that draw upon complementary strengths (and mitigate biases) of human and AI decision-makersI create new design and prototyping methods to facilitate meaningful stakeholder involvement throughout the AI development lifecycle. Finally, I conduct field experiments to understand the causal impacts of these human–AI systems in deployment.
Much of my work has focused specifically on understanding and designing for the evolving roles of K-12 teachers as AI increasingly enters the classroom. In this line of research, I work closely with both teachers and students to explore how human and AI instruction might be most effectively combined.
My research has won five paper awards from the International Artificial Intelligence in Education Society (IAIED) and the International Society of the Learning Sciences (ISLS). This research is funded by the National Science Foundation (NSF), the Institute of Education Sciences (IES), and CMU's Metro21 Smart Cities Institute (Metro21).
You can find my publications on my Google Scholar page and my CV. If you’d like to know more about my work or explore opportunities for collaboration, please get in touch!
Representative papers

Holstein, K., Wortman Vaughan, J., Daumé III, H., Dudík, M., Wallach, H. (2019). Improving fairness in machine learning systems: What do industry practitioners need?  In Proceedings of the ACM CHI Conference on Human Factors in Computing Systems (CHI’19). (p. 600). ACM.  
[ACM]   [supp. materials]   [pdf]
Holstein, K., McLaren, B. M., & Aleven, V.  (2018). Student learning benefits of a mixed-reality teacher awareness tool in AI-enhanced classrooms. In Proceedings of the 19th International Conference on Artificial Intelligence in Education (AIED 2018). LNAI 10947 (pp. 154-168). Springer: Berlin.  
*Best Paper Award*   [pdf]

Holstein, K., McLaren, B. M. & Aleven, V.  (2019). Co-designing a real-time classroom orchestration tool to support teacher–AI complementarity. Journal of Learning Analytics (JLA), 6(2), 27-52. 
[link]   [pdf]
Holstein, K., McLaren, B. M. & Aleven, V.  (2019). Designing for complementarity: Teacher and student needs for orchestration support in AI-enhanced classrooms. In Proceedings of the 20th International Conference on Artificial Intelligence in Education (AIED 2019). (pp. 157-171). Springer, Cham.
*Best Paper and Best Student Paper Nominee*   [link]   [pdf]
Back to Top