The University of Toronto Department of Computer Science is well-represented at the 2024 ACM Conference on Human Factors in Computing Systems (CHI), the premier international conference of Human-Computer Interaction (HCI).
CHI 2024 is taking place in Honolulu, Hawaii from May 11 to 16 bringing together researchers, practitioners and industry leaders to showcase some of the latest research in human-computer interaction.
The research being presented by our faculty and students covers a variety of topics such as:
augmented and virtual reality techniques and applications;
portable devices and systems for health and well-being tracking;
community voices and modern digital tools;
large language models for various creative tasks and assisting with academic procrastination; and
review of algorithmic decision-making in homelessness research and higher education.
“Computing and the Stigmatized: Trust, Surveillance, and Space Politics with the Sex Workers in Bangladesh” co-authored by Assistant Professor Ishtiaque Ahmed has been recognized with a Best Paper Award, and additional papers have received honourable mentions, as noted below.
Here is a closer look at the U of T papers accepted to the conference.
Browse papers by topic:
Interactive Technologies
Fengyuan Zhu, Mauricio Sousa, Ludwig Sidenmark, Tovi Grossman
With modern tracking techniques, we can now integrate the phone experience into VR. But this raises questions of how to represent and interact with the phone. In our work, we consider three spatial anchorings for representation and two techniques for interactions and evaluate them across common smartphone operations. Our findings lay the foundations of multidevice ecosystem for current in trending devices like Vision Pro.
Overview of the spatial anchoring and interaction techniques investigated in our study.
ProInterAR: A Visual Programming Platform for Creating Immersive AR Interactions
Hui Ye, Jiaye Leng, Pengfei Xu, Karan Singh, Hongbo Fu
Interactive AR experiences aim to bring the best of our physical and digital worlds together in a complementary immersive setting. Authoring such experiences, however, is disparate and tedious. This paper addresses the problem by presenting a user-friendly framework to create such experiences.
Writing out the Storm: Designing and Evaluating Tools for Weather Risk Messaging
Sophia Jit, Jennifer Spinney, Priyank Chandra, Lydia Chilton, Robert Soden
This paper quantitatively and qualitatively evaluates severe weather risk communication issued by Environment and Climate Change Canada (ECCC) and considers the design space of authoring support tools for risk communication professionals. This research finds that current risk messaging practice may omit some the themes associated with best practices in risk communication and proposes design recommendations for risk communication authoring support tools that can improve the efficacy and inclusivity of these messages.
Example of a proposed writing support tool which was developed from the quantitative analysis of ECCC risk messages.
Cohabitant: The Design, Implementation, and Evaluation of a Virtual Reality Application for Interfaith Learning and Empathy Building
Mohammad Rashidujjaman Rifat, Reem Ayad, Ashratuz Zavin Asha, Bingjian Huang, Selin Okman, Dina Sabie, Hasan Shahid Ferdous, Robert Soden, Syed Ishtiaque Ahmed
This paper presents a novel virtual reality application, Cohabitant, that can help people understand another person’s faith-based perspectives, and thus bring them closer. The application integrates elements such as attire, food, rituals, and games, accompanied by a pre-scripted audio guide narrating each element’s cultural and religious significance, with immersive role-playing and storytelling techniques to achieve its goal.
VR application Cohabitant allows the user to explore different clothes of people of different faiths and ideologies and understand their perspectives.
Snap, Pursuit and Gain: Virtual Reality Viewport Control by Gaze
Hock Siang Lee, Florian Weidner, Ludwig Sidenmark, Hans Gellersen
This study explores using eye movements, as an alternative way to control the view in virtual reality through head-mounted displays. We developed three techniques, Dwell Snap, Gaze Gain and Gaze Pursuit that allow users to navigate virtual worlds by moving their eyes instead of their heads. In our comparisons with traditional methods, we found that these eye-based techniques were just as fast and accurate, offering a promising avenue for more comfortable and accessible exploration of virtual environments. This has broad implications for improving user experience in various applications where users are limited in their range of motion of the head and body.
Pavel Manakhov, Ludwig Sidenmark, Ken Pfeuffer, Hans Gellersen
This study investigates how our ability to focus on visual targets in a VR display is affected when we move at walking and jogging speeds. We explore different ways of presenting targets, such as fixing them to the head or the environment, introducing delays, and keeping them at a constant distance in front of the user. Our experiment reveal that targets stable relative to the environment are easier to focus on with greater speed and precision. This has practical implications for head-mounted displays and improving user experience in interactive technologies especially when considering eye tracking as an input method during movement.
The different reference we compare in this study while walking and jogging. (a) Head: The user’s viewpoint, where the spatial UI instantly follows head movements. (b) HeadDelay: Similar to “Head,” but with a delay, creating a more realistic feel. (c) Path: A new perspective where the target stays in front of the moving user, unaffected by head movements. (d) World: The target is fixed at a distance on the virtual track, remaining stationary.
SwitchSpace: Understanding Context-Aware Peeking Between VR and Desktop Interfaces
Johann Wentzel, Fraser Anderson, George Fitzmaurice, Tovi Grossman, Daniel Vogel
This paper aims to improve switches between desktop and virtual reality (VR) computing that can be inconvenient or distracting during cross-reality tasks, like creating or consuming VR content. An initial formative study explores cross-reality switching habits, finding most switches are momentary “peeks” between interfaces, with specific habits determined by current context. The results inform a design space for context-aware “peeking” techniques that allow users to view or interact with desktop from VR, and vice versa, without fully switching. The paper implements a set of peeking techniques and evaluated them in two levels of a cross-reality task: one requiring only viewing, and another requiring input and viewing.
Switching between VR and desktop applications can be cumbersome, especially in workflows that make use of both. Quick, temporary "peeking" techniques can make these workflows faster, more precise, and more comfortable.
ReHEarSSE: Recognizing Hidden-in-the-Ear Silently Spelled Expressions
Xuefu Dong, Yifei Chen, Yuuki Nishiyama, Kaoru Sezaki, Yuntao Wang, Alex Mariakakis, Kenneth Christofferson
ReHEarSSE is an earbud-based system that recognizes silently spelled words and phrases by sensing subtle changes in ear canal shape using ultrasonic sensing.
(a) ReHEarSSE uses a novel earbud-based ultrasonic sensing method to infer silently spelled words, even if they are not in the training lexicon. (b) ReHEarSSE can be used while interacting with an extended-reality device for hands-free text input. (c) ReHEarSSE can also be used on-the-go while users’ hands are unavailable or inconvenient for text entry on a smartwatch or smart eyewear.
Health and Wellness
Promoting Engagement in Remote Patient Monitoring Using Asynchronous Messaging
Salaar Liaqat, Daniyal Liaqat, Tatiana Son, Tiago H. Falk, Robert Wu, Andrea Gershon, Eyal de Lara, Alex Mariakakis
Remote patient monitoring is a method of health-care delivery that uses technology to monitor patient health outside of conventional clinical settings. This work investigates the benefits that various stakeholders gain when patients and clinicians send messages to one another through such a system.
Functional Design Requirements to Facilitate Menstrual Health Data Exploration
Georgianna Lin, Pierre Lessard, Minh Le, Brenna Li, Fanny Chevalier, Khai Truong, Alex Mariakakis
Menstrual health trackers are currently limited to basic information like dates of menstrual bleeding. While many researchers argue that sensor data from smartwatches and other devices can be useful for this purpose, few have studied how this data could be incorporated into compelling visualizations for users. This paper presents a series of design recommendations for next-generation menstrual health trackers that are able to incorporate data from multiple sources.
An example design concept showcasing diverse menstrual data visualization features, including predicted menstrual phases and side-by-side cycle comparisons.
Beyond the Waiting Room: Patient’s Perspectives on the Conversational Nuances of Pre-Consultation Chatbots
Brenna Li, Mohit Jain, Khai Truong, Alex Mariakakis
To help doctors prepare for their patient consultations, many clinics ask that patients either fill out a questionnaire or speak with an intake nurse beforehand. In this work, we explored the potential for chatbots to facilitate clinical pre-consultation in a scalable and engaging manner for patients.
Leveraging Idle Games to Incentivize Intermittent and Frequent Practice of Deep Breathing — Honourable Mention Award
Book Sadprasid, Anne Mei, Alex Mariakakis, Scott Bateman, Fanny Chevalier
Deep breathing is a mindfulness technique known for its stress and anxiety-reducing benefits, but it can be tough to stick with due to its perceived monotony. To address this challenge, we turned to a video game genre called idle games, which excel at turning simple actions into captivating experiences by allowing gradual progress even when you’re not actively playing. We created BreathPurr-suade, a game that incorporates these mechanics, and discovered that it not only makes deep breathing more engaging but also encourages regular practice, offering a fun way to improve well-being.
Our BreathPurr-suade game encourages the practice of deep breathing, showing the potential of idle game mechanics to promote short and frequent sessions of therapeutic exercises.
Politics, Communities and Computing Technologies
Commoning as a Strategy for HCI Research and Design in South Asia
Aarjav Chauhan and Robert Soden
In this study, we examined human-computer interaction (HCI) research related to community-led initiatives of collectively sharing, governing, and managing resources in South Asia. We provide an overview of commons-based HCI research in the region as well as an assessment of recent work using the design principles for the sustainable governance of the commons developed by political scientist and Nobel laureate Elinor Ostrom. In our contributions, we draw upon HCI theory in infrastructuring, participatory design, and assets-based design in order to support efforts at commoning through future HCI research, especially within majority world contexts.
What’s the Rush?: Alternative Values in Navigation Technologies for Urban Placemaking
Taneea Agrawaal, Aarjav Chauhan, Carolina Nobre, Robert Soden
In this paper, we examine how navigation apps often prioritize efficient point-to-point travel, overlooking the importance of creating meaningful connections to our surroundings. Using analysis from interviews and design workshops, we argue for expanding the range of values in such technologies. This paper makes the following contributions: a) connects to ongoing discussions on the limitations of efficiency in the design of everyday technologies; b) highlight how people shape their surroundings while navigating, emphasizing the social interactions and encounters that occur; c) highlight the values beyond efficiency that individuals prioritize during navigation; and d) propose design implications for an alternatives.
Research activities included guided walks and participatory design workshops.
Politics of the Past: Understanding the Role of Memory, Postmemory, and Remembrance in Navigating the History of Migrant Families
Nabila Chowdhury, Natasha Shokri, Cibeles Herrera Valera, Azhagu Meena SP, Carolina Reyes Marquez, Mohammad Rashidujjaman Rifat, Marisol Wong-Villacres, Cosmin Munteanu, Negin Dahya, Syed Ishtiaque Ahmed
This paper presents an ethnographic work on immigrants in Canada, and critically examines how they preserve their past. This paper opens a new HCI design space that complements the traditional history-based understanding of the past with personal memories.
A photo stand kept at an immigrant participant's home depicting their connection to their memories of their past home.
In-Between Visuals and Visible: The Impacts of Text-to-Image Generative AI Tools on Digital Image-making Practices
Nusrat Jahan Mim, Dipannita Nandi, Sadaf Sumyia Khan, Arundhuti Dey, Syed Ishtiaque Ahmed
This paper reports an ethnographic study on the use of generative AI-assisted image generation tools in the Global South. This paper shows the colonial nature of these tools, and how local art practitioners are struggling with that.
Images of Bangladeshi marketplaces created by Midjourney that do not capture the actual look of those places. There are also many cultural components missing in these images.
Computing and the Stigmatized: Trust, Surveillance, and Space Politics with the Sex Workers in Bangladesh — Best Paper Award
Pratyasha Saha, Nadira Nadira Nowsher, Ayien Utshob Baidya, Nusrat Jahan Mim, Syed Ishtiaque Ahmed, S M Taiabul Haque
This paper presents an ethnographic study on the stigmatized sex workers in the Global South, and their interactions with modern digital tools. This paper demonstrates how these vulnerable populations are being marginalized over computing platforms and discusses how to combat those issues.
The bedroom of one of the sex workers that we studied. Different photos on the wall signal the sex worker’s attempts to appear as a regular Bangladeshi housewife over a video call.
Hearing Community Voices in HCI4D: Establishing Safe Places to Co-Create Counter-Collective Narratives with Women Farmers in Bangladesh
Manika Saha, Stephen Lindsay, Jessica L Watterson, Tom Bartindale, Delvin Varghese, Mallika Saha, Gillian C. Oliver, Syed Ishtiaque Ahmed, Patrick Olivier
This paper discusses how rural women farmers in the Global South could make their voices heard by making videos on their mobile phones, and what sociotechnical arrangements are needed to make that possible.
Responsible Machine Learning, Algorithmic Decision-Making and Fairness
Towards a Non-Ideal Methodological Framework for Responsible ML
Ramaravind Kommiya Mothilal, Shion Guha, Syed Ishtiaque Ahmed
This study examines the methodology used by machine learning practitioners to approach Responsible ML through the lens of ideal and non-ideal theories of justice/fairness. The results suggest that while practitioners predominantly relied on ideal theorizing to structure their approaches, some methodological decisions also resonated with non-ideal notions of fairness. Furthermore, this study puts forward a new methodological framework that encourages critical reflection to identify gaps, facilitates collaboration, and serves as an accountability tool, such as a model card.
An illustration of the proposed framework in "Towards a Non-Ideal Methodological Framework for Responsible ML."
Dipto Das, Shion Guha, Jed Brubaker, Bryan Semaan
This paper audits Bengali sentiment analysis tools across Python Package Index and GitHub to identify biases related to gender, religion, and nationality-based identities. By interpreting the computational findings through critical perspectives, this paper explains how natural language processing tools embed colonial values. Overall, this paper contributes to research on algorithmic fairness in low-resource language technologies in the Global South context.
Auditing Bengali sentiment analysis tools' identity biases and their relationship with developers' backgrounds.
A Human-Centered Review of Algorithms in Homelessness Research
Erina Seh-Young Moon, Shion Guha
In the face of rising homeless populations in developed nations, government agencies are increasingly adopting data-driven models to predict one’s risk of experiencing homelessness and assigning supportive resources to those in need. We conduct a systematic literature review of 57 papers to understand the evolution of these decision-making algorithms using a human-centered lens. We find most algorithms focus on identifying ‘who’ is at risk of homelessness without realistically considering ‘what’ services are realistically available/needed by clients, limiting the practical utility of the algorithms’ outputs.
Outcome variables used by homelessness algorithms over time.
“This is not a data problem”: Algorithms and Power in Public Higher Education in Canada
Kelly McConvey, Shion Guha
In this study, we conduct a qualitative case study of algorithms at a public college in Ontario, Canada. We explore how and why various stakeholders across the college use student data and find that the growing reliance on algorithmic decision-making has led to increased student surveillance, exacerbation of existing inequities, and the automation of the faculty-student relationship. We then highlight a cycle of increased institutional power perpetuated by algorithmic decision-making and driven by policies focused on financial sustainability.
The ASP-HEI Cycle - Algorithms, Student Data, and Power in Higher Education Institutions.
Md Romael Haque, Devansh Saxena, Katy Weathington, Joseph Chudzik, Shion Guha
We examined how different community stakeholders interact with algorithmic crime mapping and their perspectives on the use of such tools by police departments. We learned that community members could provide detailed and critical feedback that can inform the overall problem formulation, technical participants drew attention to the elastic nature of data science practice and suggested centering tools in a collaborative decision-making process, and law enforcement agents (LEAs) shared that a better understanding of nuances of their practices can help develop systems that are ethical, worker-centered, and complement their domain expertise.
Designing interactive policing allocations through crime maps in Milwaukee.
Incremental XAI: Memorable Understanding of AI with Incremental Explanations
Jessica Y. Bo, Pan Hao, Brian Y. Lim
Explainable AI (XAI) tools provide users of AI systems with transparency on how the AI makes predictions, however existing XAI methods must make a trade-off between modelling the AI accurately and providing an appropriate amount of information to not overwhelm the users. We introduce IncrementalXAI, a framework which provides more detailed explanations by leveraging the human cognitive capacity to accumulate knowledge by incrementally receiving information, achieving better understanding and memorability of the AI.
Top: A conceptual comparison between baseline XAI methods and Incremental XAI, where our method captures the AI system more accurately while using fewer modelling factors. Bottom: The interface of Incremental XAI for an outlier instance, where the special additive factors are activated.
Large Language Models for task assistance
Understanding the Role of Large Language Models in Personalizing and Scaffolding Strategies to Combat Academic Procrastination — Honourable Mention Award
Ananya Bhattacharjee, Yuchen Zeng, Sarah Yi Xu, Dana Kulzhabayeva, Minyi Ma, Rachel Kornfield, Syed Ishtiaque Ahmed, Alex Mariakakis, Mary P Czerwinski, Anastasia Kuzminykh, Michael Liut, Joseph Jay Williams
In our study, we investigated the role of large language model (LLM) based tools in helping students manage academic procrastination. By discussing with students and experts, and showcasing a technology probe, we found that such tools need to offer clear, step-by-step advice tailored to individuals' schedules and workloads. We also discovered the need for clear, deadline-specific guidance and adaptive questioning techniques, highlighting the potential of these models while advising caution in therapeutic applications.
Interactive features provided by our probe for generating context-specific messages: (a) customizable features that allow users to describe their individual situation, select the need for specific instructions, choose the tone and directedness of the message, and specify the desired length of the message; (b) a feature enabling open-ended modifications to generated messages; (c) a built-in editor that allows for direct edits to the generated message.
Mohi Reza, Nathan Laundry, Ilya Musabirov, Peter Dushniku, Zhi Yuan "Michael" Yu, Kashish Mittal, Tovi Grossman, Michael Liut, Anastasia Kuzminykh, Joseph Jay Williams
State-of-the-art large language models (LLMs) can potentially transform our approach to text composition and editing. However, the prevalent chat-based interfaces for LLMs often complicate the management of multiple text variations, leading to increased workload and interruption of the writer’s creative process. This paper introduces ABScribe, a novel interface designed to facilitate a rapid yet visually structured exploration of writing variations in human-AI writing tasks while minimizing task workload and preserving writers’ flow.
The ABScribe Interface: (1) Variation Fields: Multiple variations are stored within text fields that do not break the flow of the draft. (2) Popup Toolbar: Users can swiftly choose and compare multiple variations by hovering over a dynamically placed toolbar located above the selected Variation Field, or clone and edit variations in-place. (3) Variation Sidebar: Users can navigate through multiple variations organized as an accordion-based interface. (4) AI Modifiers: Users can create variations using AI by typing modification instructions auto-converted into reusable buttons that can be applied to other Variation Fields. (5) AI Drafter: Users can draft text from GPT-4 directly into the document by typing ‘@ai <prompt>’ and pressing enter.
The HaLLMark Effect: Supporting Provenance and Transparent Use of Large Language Models in Writing with Interactive Visualization
Naimul Hoque, Tasfia Mashiat, Bhavya Ghai, Cecilia D. Shelton, Fanny Chevalier, Kari Kraus, Niklas Elmqvist
The use of large language models (LLMs) for writing has sparked controversy both among writers, concerned about ownership, and writers. reluctant to read text generated by soulless machines. We argue that AI-assistance can also improve writing as long as writers can conform to publisher policies, and as long as readers can be assured that a text has been verified by a human. We design a system that captures the provenance of interaction with an LLM and show that it helps writers retain their agency, conform to policies, and communicate their use of AI to readers transparently.
The HaLLMark System. (A) Text editor for viewing and editing text. The system highlights text written (orange) and influenced (green) by the AI. There are three toggle buttons on top of the editor to turn on and off the three views (columns) of the interface. (B) Prompting interface for large language models (e.g., GPT-4). The user can see the prompts and AI responses for the current session. (C) Summary statistics show the number of prompts and percentage of user-written text and AI assistance. Below that, we see a timeline of a user’s writing actions (grey rectangles) and interaction with the AI (purple and blue rectangles). The user can hover over any glyphs in the timeline to see the relevant prompt and linked text in the text editor.
Sangho Suh, Meng Chen, Bryan Min, Toby Jia-Jun Li, Haijun Xia
This paper explores how we can effectively use large language models (LLMs) like those in ChatGPT. Large language models can generate many ideas, which makes them ideal for creative tasks, such as creative writing. However, the way we currently use LLMs — prompting and evaluating the returned response — does not take advantage of their creative potential. The primary mode of interaction, called prompt engineering, guides us to evaluate only a single response instead of a wide variety of responses these models can generate; this results in a missed opportunity to reflect on and explore a range of ideas AI can offer. To address this, we contribute a framework for systematically generating diverse responses and Luminate, a system that demonstrates this framework. This work has broad implications beyond creative writing, like art, design, and more, improving how we use generative AI to augment human creativity.
An illustration of a conceptual framework our work proposes. Concretely, it shows how large language models can generate diverse ideas. Once the user inputs a prompt, a large language model generates dimensions relevant to a topic or task in the prompt and a set of values for each dimension. The combination of these values is then used to generate responses, covering a variety of sub-spaces within the design space.
DirectGPT: A Direct Manipulation Interface to Interact with Large Language Models — Honourable Mention Award
Damien Masson, Sylvain Malacria, Géry Casiez, Daniel Vogel
Instead of solely relying on words, we explore how to interact with large language models through direct interactions. For example, users can select elements to clarify their intent and have better control over the edits produced by large language models. More broadly, we provide guidance on how to integrate large language models into editing software.
Demonstration of our approach to edit an image using large language models. Instead of having to describe elements textually to the model, users can (a-b) select elements and apply a modification to them; (c) refer to elements in their instructions to the model and (d) reuse previous instructions.
Majeed Kazemitabaar, Runlong Ye, Xiaoning Wang, Austin Z. Henley, Paul Denny, Michelle Craig, Tovi Grossman
Timely, personalized feedback is essential for students learning programming. LLM-powered tools like ChatGPT offer instant support, but reveal direct answers with code, which may hinder deep conceptual engagement. We developed CodeAid, an LLM-powered programming assistant delivering helpful, technically correct responses, without revealing code solutions. CodeAid answers conceptual questions, generates pseudo-code with line-by-line explanations, and annotates student's incorrect code with fix suggestions. We deployed CodeAid in a large class of 700 students over a 12-week semester to explore students’ usage patterns and perceptions, CodeAid's response quality, and educator's perceptions.
CodeAid allows students to ask five types of coding questions: General Question, Question From Code, Explain Code, Help Fix Code, and Help Write Code. In response, CodeAid uses LLMs to generate pedagogical answers that do not contain direct code solutions. When asked general questions or to generate code, it provides a natural language response with an interactive pseudo-code that allows students to hover over each line and understand what each line does. Responses also include relevant function documentations retrieved from a database to ensure factual accuracy and approved by course educator. When asked to help fix provided incorrect code, CodeAid does not display the fixed code. Instead, it highlights incorrect parts of the students’ code with suggested fixes.
Digital Identities
A Quantitative Approach to Identifying Emergent Editor Roles in OpenStreetMap
Bowen Zhang, Jennings Anderson, Dipto Sarkar, Robert Soden
This paper uses unsupervised methods to identify contributor behaviours of mappers in OpenStreetMap (OSM). By analyzing their activity, we identified eight distinct roles within the OSM community. We then investigate the composition and retention of contributors over time based on these roles. The research contributes to a detailed understanding of the work practices and social life of the world’s largest volunteered geographic data project and provides design and future research recommendations aimed at supporting contributor satisfaction and retention.
Behind the Pup-ularity Curtain: Understanding the Motivations, Challenges, and Work Performed in Creating and Managing Pet Influencer Accounts
Suhyeon Yoo, Kevin Pu, Khai N. Truong
The study explores the trend of Instagram users creating accounts specifically for their pets. It investigates the motivations behind these accounts, which range from personal enjoyment to commercial goals, and the strategies used to craft their pets’ online personas. Additionally, the research delves into the challenges of managing these digital identities and the evolving relationship between humans and pets in this context, suggesting a need for interfaces that support this unique form of identity management.
Process and factors involved in creating and managing pet specific accounts (A) Transition from personal accounts to pet accounts. (B) Three goals in creating and managing pet accounts. (C) Spectrum of representation strategy. (D) Benefits and challenges of operating pet accounts.
— All images courtesy of authors.