The blind and low vision (BLV) community faces many challenges in their lives and needs increased attention and support. With the rise of vision impairment in our aging digital society, more people will experience vision-related difficulties as well.
43 Million Blind (2020)[1]
295 Million Moderate to Severe Visual Impairment (2020)[1]
2.2 Billion Near or Distance Vision Impairment (2023)[2]
Through research we discovered that while there are mature solutions(e.g. VoiceOver, Brail) for reading, writing, social communication, house management, and outdoor navigation, indoor navigation, which includes activities such as visiting grocery stores, navigating buildings, and going to libraries, remains a significant challenge.
Our aim is to significantly improve the indoor navigation experience for blind and low-vision (BLV) individuals, thus enhancing their independence and quality of life.
Final Solution
We designed an innovative, user-centered product that integrated AI technology for indoor environment detection and navigation.
What is the final product?
How does it work?
Function 1
General Indoor Navigation
The camera system scans the environment for indoor maps. If a map is detected, users are prompted and guided by SeePal to scan and store the map in the system. In the absence of a visible map, SeePal guides users to building staff for assistance.
Function 2
Surrounding Recognition
SeePal offers audio cues for important surroundings like restrooms, people, etc. More details are provided only if requested to give users greater control. The conversation remains friendly and engaging, and the tone simulates Apple’s VoiceOver.
Function 3
Avoid Obstacles
When the camera system detects obstacles, it controls the physical tool to slow down, vibrate and guide users to pass around them. Audio alerts like ‘Obstacle detected’ follows by specifics such as ‘chest level’ for wires or branches, and distance cues like ‘5 meters ahead’ are relayed through earbuds.
Function 4
Locate specific items
The camera system scans the environment to pinpoint the item’s location. It stretches to the item’s height, guiding the user’s hand to touch it. A vibration signal is emitted just before the cane stretch. After users touch an item, distinct audio cues like ‘left’ or ‘right’ help identify the item’s exact location.
Discovery
Literature Review
Our review reveals that the most crucial challenges faced by BLV users are centered around daily living activities, including reading, facial recognition, and navigating unfamiliar environments [6].
The current landscape of technological support for BLV individuals related to indoor navigation includes audio-based navigation systems [4], camera-aiming guidance for obstacle navigation [3], tactile maps [5], and object recognition technology [7]. However, there remains a gap for a comprehensive solution that integrates the latest AI technology to enhance the entire indoor navigation experience, especially involving the use of computer vision to interpret the environment and support real-time interaction.
Key considerations revolve around customization, intuitive design to bolster user independenceandconfidence in daily lives [6].
Competitive Analysis
We analyzed five related products using a matrix that evaluates various criteria, including target user, prerequisites, key features, social impact, quality, technology, aesthetics, safety, and cost. Based on this analysis, we provide our critiques in the table below.
Look at the Competitive Analysis Table!
User Research
4 Participants
• 3 accessibility experts in HCI field • 1 blind user
Method
• In-depth interview ( In-person & Zoom) • Goal: Understand the current practices and challenges BLV users face in their daily life.
General Indoor Navigation BLV users find it challenging to understand a building’s layout and struggle to locate specific rooms or areas either independently or efficiently, especially since many buildings don’t have digital indoor maps.
Surrounding Recognition BLV users may struggle to perceive details like facial expressions, environment, or complex things that are unable to be conveyed through braille or screen readers.
Avoid Obstacles It’s challenging to detect various obstacles, such as holes, boxes, or wires, that may be present in their path.
Locate Specific Items Identifying particular items, such as a chair in a conference room or an item on a shelf, is often a formidable task.
Overall Findings
We used a journey graph to show key challenges in indoor navigation. For each challenge, We identify prevailing practices and their shortcomings, then analyze 1-2 popular alternative solutions with their pros and cons, and finally identify opportunities from these solutions to inspire ideation section (refer to the dotted box in the graph to identify the font color and dialog box representation).
Problem Statement
Through integrating all the findings above, our design concept is defined by the need for a comprehensive tool that aids BLV users in navigating indoor environments more efficiently and independently, enhancing their overall quality of life and autonomy.
Our design objective is to address challenges identified in four critical scenarios which are general indoor navigation, surrounding recognition, avoiding obstacles and locating specific items, meanwhile ensuring that four key principles which are enhanced independence, discreet appearance and size design, user-controlled and user-friendly interface (tactile feedback, interactive conversation) are integral to the design process so that the final product effectively meets the user’s needs.
Ideation
Brainstorm
Merge & Selection
From our brainstorming session, we have merged and selected four distinct ideas.
Idea 1
RFID Route/Tracker
Description: This system involves distributing RFID chips labeled with different room identifiers (e.g., bathroom, room 101) to individuals. Alternatively, trackers can be attached to people’s bags to map out pathways. Staff could label various items (like a specific conference chair) with RFID tags. Participants are encouraged to place these tags along their route, receiving bonuses credited to their accounts upon doing so. The system also includes a digital tracking mechanism (possibly using infrared technology) to keep the information updated, especially for refreshable items.
Features: 1. Distribution of Chips and Incentivizing Participation 2. Digital Updating System: For real-time tracking and refreshing of the RFID tag information.
The Pros and Cons
Pros: 1. Cost-Effective Mapping: Utilizes human movement to create a map, reducing the need for an extensive database. 2. Interactive Participation: Encourages active involvement from individuals, potentially enhancing the mapping accuracy.
Cons: 1. High Operational Costs: The cost and maintenance of RFID chips can be expensive. 2. Reliability Issues: The accuracy of the map depends on the correct placement of RFID tags by participants, like crowdsourcing, which may not be reliable.
Idea 2
Guide Bee
Description: The shared Guide Bee can be summoned via a phone. Upon being called, the bee-like device flies to the user and communicates the desired destination. It connects to a digital system for navigation, attaching to the user’s finger to provide physical guidance. Equipped with a camera, the Guide Bee scans the ground for obstacles and uses vibrations to alert the user. Upon reaching the destination, it can perch on the user’s shoulder for emergency situation awareness. After use, it autonomously returns to its base when instructed via phone.
Features: 1. Call and Response: Activated by phone, it autonomously locates the user. 2. Physical Guidance and Feedback: Attaches to the finger to direct the user.
The Pros and Cons
Pros: 1. Mobility Advantage: More agile and convenient than traditional guide dogs due to its ability to fly. 2. Form Factor: Small, tactile, and virtually invisible when not in use, offering discreet assistance. 3. Guiding Efficiency: Provides tactile guidance, enhancing navigation ease for users.
Cons: 1. Technical Challenges: Developing such advanced technology with reliable navigation and obstacle detection systems poses challenges. 2. Emergency Response Limitations: While it can detect emergencies, its ability to assist or guide in such situations may be limited.
Idea 3
Smart Ear Device + Cane
Description: A hybrid of a smart earpiece and a cane featuring a 180-degree micro camera. It provides real-time indoor navigation assistance with 360-degree spatial sound and interactive communication akin to ChatGPT and Siri, complete with a danger alert system.
Features: 1. Wide vision range 2. Real-time audio support with spatial sound. 3. Interactive communication (ChatGPT + Siri). 4. Danger alerts.
The Pros and Cons
Pros: 1. Interactive and immediate feedback. 2. Enhanced safety with danger alerts. 3. Intuitive 360-degree spatial orientation.
Cons: 1. Dependence on technology reliability. 2. Regular maintenance and charging needed. 3. Learning curve for combined device use. 4. Accuracy varies across environments.
Idea 4
Robot Dog
Description: This Robot Dog is equipped with cameras and offers shared usage, accessible and payable via phone. It’s designed to be foldable.
Pros: 1. Direct guidance: Capable of physically leading the user, offering clear direction. 2. Rapid response: Provides faster feedback in emergencies, enhancing user safety.
Cons: 1. Lack of experience: Many visually impaired individuals may not have prior experience with guide dogs, it’s a potential adaptation challenge. 2. Social acceptance: There could be societal challenges in accepting a robotic dog, especially in public spaces. 3. Safety concerns: Being AI-powered and autonomous, there are general safety concerns related to its operation and reliability.
Idea 1
RFID Route/Tracker
Idea 1 effectively generates indoor digital maps, but falls short in meeting user needs for recognizing surroundings and locating specific items. As it serves better as a preliminary step rather than a primary solution, we have decided to exclude Idea 1.
Idea 2
Guide BEE
Idea 2 and Idea 4 share similarities, with both employing tactile feedback from automated physical objects to guide users. However, Idea 2 presents more technological challenges than Idea 4 due to its smaller size. So we excluded Idea 2.
Our final concept combines the strengths of Idea 3 and Idea 4. This amalgamation addresses user needs effectively (as detailed in the subsequent section on design rationale) and is feasible with current technologies.
Final Idea Explanation!
The final idea merges the conversational intelligence of ChatGPT with the universally accepted form of a traditional cane and the intuitive guidance of a service dog. We want to create an interactive experience where the product not only conveys crucial information and interact with users in a friendly conversational manner, but also guides and appears like an easy-to-use cane and provides self-directed navigation and physical feedback akin to a service dog.
Final Idea Rationale
This section explains the rationale behind each design element in the final product(black italic font) and how four key principles apply (blue font). The final solution also explains how the design tackles challenges in four critical scenarios.
Implementation & Evaluation
Our prototype and evaluation process both consist of two rounds. Initially, two experts assessed our first prototype, providing crucial feedback. Following improvements, our second prototype undergoes user testing with two participants. This approach optimizes efficiency by incorporating expert insights early and streamlining user testing.
Low-Mid Prototype
We created a prototype capable of simulating various designs, such as freely rotating, stretching, and ergonomic handles. The video shows how it works.
1st Evaluation
Experts
Crescentia Jung
Info Science PhD student at Cornell Tech, expert researcher in accessibility
Mahika Phutane
PhD student at Cornell Enhancing Ability Lab, expert researcher in accessibility
Goals & Method
Goals
1. Get feedback on the 1st prototype, including its effectiveness in addressing challenges in 4 key scenarios. 2. Identify any major issues to resolve before the 2nd evaluation, which is user testing.
Method
• We conducted a 45-minute Zoom video call with each researcher to present the prototype and product video, discuss, and interview with a series of prepared questions.
Key Insights
Positive Outcomes
Neat and Intuitive Solution Experts recognized it as an innovative, intuitive, and effective solution, adeptly addressing the primary challenges faced by BLV users in indoor navigation.
Good Control Level Features such as stretchability and on/off functionality provide users with enhanced control.
Refinement Opportunities
Handle Preference Research indicates a preference among users for more intuitive handle designs.
Safety Consideration The speed of the automatic cane guidance need to match the user’s walking pace.
Audio Distraction Ensure users remain aware of their surroundings when there is an overlap between environmental sounds and SeePal audio input.
Mid Fi Prototype
Change 1
Handle
We redesigned the handle reference to service dog handle for enhanced comfort and eco-friendliness.
Change 2
Speed Control
During user testing, participants can adjust speed settings, such as requesting “slower” or “faster” responses, which we then simulate accordingly.
Change 3
Audio Interface
Through testing, we’ve trained ChatGPT to provide concise, relevant responses to user prompts to prevent users from being overwhelmed and enhance the audio interface’s user-friendliness.
2nd Evaluation
Participants
Sam
Blind, recruited from the formative research.
Mia
Low vision, particularly impaired in the lower half of her visual field.
Goals & Method
Goals
1. Assess the effectiveness of our product in addressing four key scenarios. 2. Determine if our product meets four design principles. 3. Provide future recommendations.
Method
• We conducted user testing for our prototype in the library with each participant, including a brief tutorial and the task for users to find a specific book and move to a reading area. • No time constraints and encouraged genuine interactions. • Using the ‘Wizard of Oz’ technique, one member simulated AI responses, and one member manually controlled the prototype to emulate automated reactions to user inputs. The third team member ensured participant safety and the remaining member documented observations.
Key Insights
Positive Outcomes
Enhanced Experience Participants reported significantly improved experiences across all four simulated scenarios.
Great User Controlled Participants appreciated the user-controlled design, especially stretchability, audio interaction, user-friendly autonomous guiding, and physical feedback (e.g., vibration). It demonstrated that we successfully adhered to our design principles.
Refinement Opportunities
Cues for Automatic Activities Users prefer more audio or physical cues when SeePal doing automatic activities like stretching.
Directional Guidance Users indicated a need for clearer directional guidance for locating a specific item like ‘left’, ‘right’ etc.
Obstacle Detection SeePal needs to detect and describe obstacles more precisely, including their types and height levels, and provide appropriate auditory and tactile feedback.
We incorporate these three refinement opportunities into our final solution.
Our user testing demonstrates significant improvements in the indoor navigation experience for BLV users by addressing key challenges in critical indoor scenarios and offering user-controlled and friendly navigation, thereby enhancing their independence and quality of life.
Concept Potential: Thoughts on our product and Rabbit, AI Pin
As our project concludes in November 2023, we are excited to see the launch of innovative products like AI Pin and Rabbit, with Rabbit being released in early 2024. Both products feature the integration of camera technology for computer vision and AI-driven conversational interfaces through speakers, aligning with the key concepts we’ve embraced in SeePal’s design and underscoring its feasibility. Our commitment to inclusivity and accessibility in design reflects our anticipation and empathy for the evolving AI era, highlighting the significance of these principles in the development of new technologies.
Limitation
Environmental Concern
SeePal may struggle in crowded environments due to its spatial requirements for navigation. Additionally, its dual cameras for computer vision significantly drain the battery, limiting its operational duration.
Privacy Consideration
The dual cameras on the SeePal, intended for recognizing surroundings, raise privacy concerns. It’s recommended that SeePal automatically switch from camera to radar mode in sensitive areas like bathrooms to protect privacy.
User Testing Limitation
Our evaluation involved a relatively small number of participants, and due to safety considerations, we did not test navigations at stairs —an important obstacle type. These factors could affect the robustness of our design decisions. Also due to time and resource constraints, we were unable to prototype the tutorial and test the learning curve of our product, leaving us uncertain about how long it takes for users to trust it.
Key Takeaways
1. Enhance User Testing Efficiency under Limited Conditions: We encountered difficulty reaching a sufficient number of BLV participants in person as it was a very niche group. To address it, we streamlined the process by initially testing with indoor navigation and accessibility experts before involving users. This approach not only optimizes efficiency but also provides valuable insights into the differing perspectives of experts and users.
2. Make Rapid Design Decisions as A Project Leader: Throughout the design process, I facilitated discussion sessions for teammates to present their thoughts and supporting evidence when there is a need to make quick decisions in design iterations. It went out efficiently.
3. Inclusive and Accessible Design Experience: Collaborating with BLV individuals in person has improved my ability to design for inclusivity and accessibility.
