HandSight

2013–Present 27

GlassEar

2014–Present
Project Sidewalk

2012–Present

Project Description

HandSight augments the sense of touch in order to help people with visual impairments more easily access the physical and digital information they encounter throughout their daily lives. It is still in an early stage, but the envisioned system will consist of tiny CMOS cameras and micro-haptic actuators mounted on one or more fingers, computer vision and machine learning algorithms to support fingertip-based sensing, and a smartwatch for processing, power, and speech output. Potential use-cases include reading or exploring the layout of a newspaper article or other physical document, identifying colors and visual textures when getting dressed in the morning, or even performing taps or gestures on the palm or other surfaces to control a mobile phone.

Publications

Applying Transfer Learning to Recognize Clothing Patterns Using a Finger-Mounted Camera

Lee Stearns, Leah Findlater, Jon E. Froehlich

keywords: wearables, computer vision, visual impairments, pattern recognition, transfer learning

PDF | doi | Citation | HandSight

Design of an Augmented Reality Magnification Aid for Low Vision Users

Leah Findlater, Jon E. Froehlich, Lee Stearns

keywords: wearables, head-mounted display, low-vision, visual impairments, assistive technology, augmented reality

PDF | Citation | HandSight

TouchCam: Realtime Recognition of Location-Specific On-Body Gestures to Support Users with Visual Impairments

Lee Stearns, Uran Oh, Leah Findlater, Jon E. Froehlich

keywords: wearables, accessibility, computer vision, low-vision, blind users, on-body interaction, skin texture classification, gesture recognition

PDF | Citation | HandSight

Recognizing Clothing Colors and Visual Textures Using a Finger-Mounted Camera: An Initial Investigation

Chuan Chen, Leah Findlater, Jon E. Froehlich, Alexander Medeiros, Lee Stearns

keywords: wearables, blind, visual impairments, texture recognition

PDF | Citation | HandSight

Investigating Microinteractions for People With Visual Impairments and the Potential Role of on-Body Interaction

Uran Oh, Lee Stearns, Alisha Pradhan, Jon E. Froehlich, Leah Findlater

| Acceptance Rate: 26.2% (33 / 126)

keywords: wearables, visual impairments, mobile, on-body interaction, microinteraction

PDF | Citation | HandSight

Evaluating Wrist-Based Haptic Feedback for Non-Visual Target Finding and Path Tracing on a 2D Surface

Jonggi Hong, Alisha Pradhan, Jon E. Froehlich, Leah Findlater

keywords: wearables, haptics, accessibility, blind, blind users, haptic feedback, haptic wristband

PDF | Citation | HandSight

Augmented Reality Magnification for Low Vision Users with the Microsoft Hololens and a Finger-Worn Camera

Victor De Souza, Leah Findlater, Jon E. Froehlich, Lee Stearns, Jessica Yin

keywords: wearables, visual impairments, assistive technology, augmented reality, reading printed text, magnification

PDF | doi | Citation | HandSight

Localization of Skin Features on the Hand and Wrist From Small Image Patches

Lee Stearns, Uran Oh, Bridget Cheng, Leah Findlater, David Ross, Rama Chellappa, Jon E. Froehlich

keywords: handsight, computer vision, machine learning, on-body interaction, skin texture classification, biometrics, skin-based localization

PDF | Citation | HandSight

Evaluating Haptic and Auditory Directional Guidance to Assist Blind People in Reading Printed Text Using Finger-Mounted Cameras

Lee Stearns, Ruofei Du, Uran Oh, Catherine Jou, Leah Findlater, David Ross, Jon E. Froehlich

keywords: wearables, handsight, accessibility, visual impairments, real-time ocr, blind reading

PDF | Citation | HandSight

Evaluating Angular Accuracy of Wrist-based Haptic Directional Guidance for Hand Movement

Jonggi Hong, Lee Stearns, Tony Cheng, Jon E. Froehlich, David Ross, Leah Findlater

keywords: wearables, haptics, non-visual directional guidance, handsight

PDF | Citation | HandSight

Supporting Everyday Activities for Persons with Visual Impairments Through Computer Vision-Augmented Touch

Leah Findlater, Lee Stearns, Ruofei Du, Uran Oh, David Ross, Rama Chellappa, Jon E. Froehlich

keywords: computer vision, blind, visual impairments, finger camera, touch vision

PDF | Citation | HandSight

The Design and Preliminary Evaluation of a Finger-Mounted Camera and Feedback System to Enable Reading of Printed Text for the Blind

Lee Stearns, Ruofei Du, Uran Oh, Yumeng Wang, Rama Chellappa, Leah Findlater, Jon E. Froehlich

keywords: wearables, accessibility, real-time ocr, text reading for blind

PDF | Citation | HandSight

Talks

HandSight: A Touch-Based Wearable System to Increase Information Accessibility for People with Visual Impairments

Aug. 1, 2018 | PhD Defense, Computer Science

University of Maryland, College Park

PDF | PPTX | HandSight

Making with a Social Purpose

April 6, 2017 | Lecture Series at the Laboratory for Telecommunication Sciences

LTS Auditorium, College Park, MD

PDF | Video | HandSight • MakerWear • Project Sidewalk • GlassEar • Scalable Thermography • BodyVis

Interactive Computational Tools for Accessibility

Nov. 7, 2016 | Diversity in Computing Summit 2016

College Park, Maryland

PDF | HandSight • Project Sidewalk

Evaluating Haptic and Auditory Guidance to Assist Blind People in Reading Printed Text Using Finger-Mounted Cameras

Oct. 26, 2016 | ASSETS 2016

Reno, Nevada

PDF | PPTX | SlideShare | HandSight

We design, build, and evaluate new interactive tools and techniques to address pressing societal challenges. Makeability refers both to how our technological innovations make new abilities possible for humans as well as our educational mission to help students gain new abilities as they learn and grow through research, invention, and human-centered design.