Augmented reality powers a cognitive assistant for the blind
- California Institute of Technology, United States
- University of Southern California, United States
Figures
Figure 1 with 2 supplements
Hardware platform and object localization task.
(A) The Microsoft HoloLens wearable augmented reality device. Arrow points to one of its stereo speakers. (B) In each trial of the object localization task, the target (green box) is randomly placed …
Figure 1—figure supplement 1
Obstacle avoidance utility and active scene exploration modes.
(A) to (C) An object avoidance system is active in the background at all times. Whenever a real scanned surface or a virtual object enters a danger volume around the user (red in A), a spatialized …
Figure 1—figure supplement 2
Process of scene sonification.
The acquisition system should parse the scene (A) into objects and assign each object a name and a voice (B). In our study, this was accomplished by a combination of the HoloLens and the …
Figure 2 with 1 supplement
Spatial memory task.
(A) Five objects are arranged on a half-circle; the subject explores the scene, then reports the recalled object identities and locations. (B) Recall performance during blocks 1 (left) and 2 …
Figure 2—figure supplement 1
Mental imagery task supplementary data.
Spatial memory data (Figure 2) from blocks 1 (left) and 2 (right) by subject. Shaded areas indicate the true azimuthal extent of each object. Markers indicate recalled location. Most recalled …
Figure 3 with 2 supplements
Direct navigation task.
(A) For each trial, a target chair is randomly placed at one of four locations. The subject begins in the starting zone (red shaded circle), follows the voice of the chair, and navigates to the …
Figure 3—figure supplement 1
Direct navigation task extended data.
Trial distance (A) and trial duration (B) for the first 20 trials of all subjects. A modest effect of practice on task duration can be observed across all subjects (B). (C) Low-pass filtered, …
Figure 3—figure supplement 2
Additional experimental functions.
(A) to B) Automated sign recognition using computer vision. Using Vuforia software (https://www.vuforia.com/) the HoloLens recognizes a men’s room sign (A), image viewed through HoloLens) and …
Figure 4 with 1 supplement
Long-range guided navigation task.
(A) 3D reconstruction of the experimental space with trajectories from all subjects overlaid. (B and C) 2D floor plans with all first trial trajectories overlaid. Trajectories are divided into three …
Figure 4—figure supplement 1
Guided navigation trajectories.
(A) 3D model of the experimental space as scanned by the HoloLens. (B) Subject and guide trajectories from the long-range guided navigation task. Note small differences between guide trajectories …
Figure 5 with 1 supplement
Benchmark testing environment.
(A) A virtual living room including 16 pieces of furniture and other objects. (B) Localization of a randomly chosen object relative to the true object location (0 deg, dashed line) for four subjects …
Figure 5—figure supplement 1
Benchmark tests in a virtual environment.
Trajectories of three additional subjects. (A) Navigation to a randomly placed chair, using either CARA or vOICe, displayed as in Figure 5C. Subject #4 exhibited some directed navigation using …
Videos
Video 1
Long-range navigation.
A video recording of a subject navigating in the long-range navigation task. The top right panel shows the first person view of the subject recorded by the HoloLens.
Video 2
Automatic wayfinding explained.
A video demonstration of how automatic wayfinding works in a virtual environment.
Video 3
Automatic wayfinding in an office.
A point of view video demonstration of the automatic wayfinding function in an office space with obstacles. The path is calculated at the user’s command based on the geometry of the office.
Additional files
-
Transparent reporting form
- https://doi.org/10.7554/eLife.37841.017
