Lilian de Greef

I am a researcher at Apple, applying machine learning to improve accessibility in technology.
I received my PhD at the University of Washington with an NSF fellowship and Microsoft Research PhD fellowship, advised by Shwetak Patel in the Ubiquitous Computing Lab. My PhD research focused on integrating HCI, ML, and computer vision to improve access to medical care with low-cost commodity hardware and mobile phones. Outcomes of my PhD work are under active commercial development and have already begun to change medical standards of care.
I graduated from Harvey Mudd College in 2012 as a President's Scholar with a bachelors degree in Computer Science. I conducted research with Zach Dodds in Computer Vision for Robotics and Christine Alvarado in HCI and Computer Vision.



Interdependent Variables: Remotely Designing Tactile Graphics for an Accessible Workflow

Publication Video Supplemental Material
Lilian de Greef, Dominik Moritz, Cynthia Bennett
My colleagues and I shared our experiences of working together, as a team of blind and sighted researchers, on creating an accessible workflow for remotely collaborating on data visualizations. In this experience report, we outline our process for developing and incorporating tactile graphics, share lessons and tips, and advocate the importance of treating and supporting access work as a collective responsibility.


Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels

Publication Video Supplemental Material
Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, Aaron Everitt, Jeffrey P. Bigham
Screen Recognition applies computer vision to automatically infer accessibility metadata for mobile apps from their pixels. Work from this project was published at CHI ’21 and released as an accessibility feature in iOS for VoiceOver.

During PhD


AI for CHAMP (Cardiac High Acuity Monitoring Program)

Lilian de Greef, Jessica Lundin, Naoto Usayama, Children's Mercy Hospital CHAMP Team
During an internship at Microsoft Research, I worked on improving CHAMP, a system for monitoring infants with single ventricle heart disease. I communicated with medical partners at Children’s Mercy Hospital, concretized technical goals, formulated methodology, wrangled and pre-processed data, developed prediction algorithms, and drafted the team’s future work. Insights from my work have already influenced clinical practice, with the potential for further impact as the project develops. Highlighted on MSR's Instagram. Published as part of dissertation.


Use of a Smartphone App to Assess Neonatal Jaundice

James A. Taylor, James W. Stout, Lilian de Greef, Mayank Goel, Shwetak Patel, Esther K. Chung, Aruna Koduri, Shawn McMahon, Jane Dickerson, Elizabeth A. Simpson, Eric C. Larson
This project is follow-up work to BiliCam, in which we made modifications to the BiliCam system and ran larger study to validate its efficacy at screening newborn jaundice. This study spanned across 7 different sites around the country to broaden the diversity of skin tones in our dataset. From our dataset of samples from 530 newborns, BiliCam demonstrated medically accurate estimates. Our findings are published as a journal paper in Pediatrics, August 2017.


BiliCam: Using Mobile Phones to Monitor Newborn Jaundice

Publication Patent #1 Patent #2 Poster Talk Slides Video
Lilian de Greef, Mayank Goel, Min Joon Seo, Eric Larson, James W. Stout, James A. Taylor, Shwetak Patel
This project investigates how to use smartphone cameras to screen newborns for dangerous levels of jaundice — a medical condition characterized by yellowing of the skin. I work in close collaboration with UW Medical Center to develop data collection procedures and software, apply computer vision to parse images, and use machine learning to estimate jaundice levels. Work from this project was published as a paper in UbiComp 2014 with an Honorable Mention Award, and resulted in two patents, and current commercial development.


TeleTourist: Immersive Telepresence Tourism for Mobility-Restricted Participants

Patent Poster Abstract Poster
Lilian de Greef, Meredith Ringel Morris, Kori Inkpen
During an internship at Microsoft Reserach, I conceived and developed a prototype of TeleTourist, a system that uses video calls with strangers to share experiences for people with mobility restrictions. I interviewed individuals with mobility restrictions as formative work, designed system features, and implemented a subset of them for a prototype. This project was published as a poster at CSCW '16 and resulted in a patent.


Perspective Aware Projected User Interfaces

Patent Talk Slides
Lilian de Greef, Alex Colburn, Jim Curlander
During an internship at Amazon I designed, developed, and evaluated eyes and head tracking related user interface elements to use in enhanced reality interfaces in fulfillment centers. I combined concepts from computer graphics with HCI to produce several prototypes and demonstrated the system in its intended environment. This work resulted in a patent.



Microsoft Surface for Stroke Rehabilitation

Publication Talk Slides
Cati Boulanger, Adam Boulanger, Lilian de Greef, Andy Kearney, Kiley Sobel, Russell Transue, Z Sweedyk, Paul H. Dietz, Steven Bathiche
As part of a four-person team, I designed and developed technology to motivate and assess rehabilitation for stroke patients affected in their upper extremities, using the Microsoft Surface. We interviewed stroke patients and physical therapists, designed a rehabilitative game played on the Microsoft Surface, produced a prototype, and ran user study with stroke patients. Our work was presented at CWIC '12 and published in a note at CHI '12.


Accessible Aerial Autonomy

Publication Poster Talk Slides Video
Nick Berezny, Lilian de Greef, Bradley Jensen, Kimberly Sheely, Malen Sok, David Lingenbrink, Zachary Dodds
I created and explored vision-based localization algorithms for aerial robots as part of a team of five students. We prototyped autonomous cooperation between ground-based and airborne robots and demonstrated localization for a quadrotor helicopter toy using only a built-in camera. We presented our work as demos at GCER '11 and AAAI '11, a talk at CWIC '12, and a publication in TePRA '12.


LogiSketch: A Free-Sketch Digital Circuit Design and Simulation System

Talk Slides
Kevin Chan, Lilian de Greef, Alexa Keizur, Alice Paul, Brian Liao, June Woo Suk, Christine Alvarado
I helped developed software for tablet PCs to recognize and simulate hand-drawn logic circuits as part of a siz-person team. I specifically focused on methods to refine sketch recognition and refactoring code. I helped the team with user interface design and circuit simulation. The software was then rolled out and used in Harvey Mudd College’s introductory computer science course.


The Automated Australian Chook Gate

Tech Report
Lilian de Greef, Mark Ellis, Nick Hill, Nicole Peck
I was part of a four-person team to design a device that automatically opens and closes a chicken coop for a client in Australia. With it, his chickens (a.k.a. chooks, in Australia) could go outside during the day and be safe from predatory foxes at night, regardless of his availability. We designed and developed a system, including a functional prototype, regularly communicated with our client, and delivered a final tech memorandum & presentation.

Featured Press

Gizmodo Engadget Reuters Med City News New Scientist Futurity ECN Mag

Teaching & Mentoring

Graduate Mentor

Undergraduate Research, University of Washington: 2014 - present
Over several years, I mentored four undergraduate students to participate in research projects. Through guidance and projects focused on real-world applications, I helped them understand research, learn and develop skills for applying computer vision or user-centered design, practice skills for working with iOS or OpenCV, and figure out future career plans.


UW CSE 373: Data Structures and Algorithms for Non-Majors, University of Washington: Summer 2017
As a predoctoral instructor, I taught an course on the fundamentals of data structures and algorithms. As part of my role, I prepared and delivered lectures with interactive activities for a class of 67 students ranging from high school to graduate school students with a diverse set of majors; managed a team of 5 teaching assistants; prepared homework assignments; wrote, proctored, and graded exams; hosted office hours; and mentored some students one-on-one outside of office hours. My teaching was rated 4.4/5 by my students in the final course review.

Teaching Assistant

UW CSE 373: Data Structures and Algorithms for Non-Majors, University of Washington: Summer 2016
In a team with two other teaching assistants, I prepared and ran review sections, hosted and tutored students in office hours, graded homework assignments, and helped proctor and grade exams for a class of 78 students.

Grader and Tutor (Grutor)

HMC CS 70: Data Structures and Program Development, Harvey Mudd College: Fall 2010, Spring 2012
CS 70 is one of the most intense courses in Harvey Mudd's core computer science curriculum; for their first time in the program, students design and implement programs from scratch, learn C++ in depth, and experience strict grading rubrics that emphasize coding style and rigor. I hosted and tutored students in office hours and graded homework assignments as part of a team of grutors. My tutoring hours became so popular that I usually kept a mental queue of who I was supposed to help next, often approached by people asking “enqueue?”

Grader and Tutor (Grutor)

HMC CSE 60: Principles of Computer Science, Harvey Mudd College: Spring 2010
I hosted and tutored students in office hours and graded homework assignments as part of a team of grutors for the second course in Harvey Mudd's computer science curriculum.

Teaching Assistant

HMC Winter Physics Workshop, Harvey Mudd College: Winter 2010
To help students who entered Harvey Mudd College with less physics experience, I helped assist the first iteration of a one-week intensive workshop that took place during winter break. Rather than a comprehensive review of background physics, it focused on developing the tools needed to do the work in the physics course of mechanics (Physics 24). With one other assistant, roles included attending all class sessions, and tutoring students through exercises during class sessions and homework assignments.