Asian International students at Cornell often face challenges in effectively communicating their ideas and thoughts in English across various settings. Even if they are proficient in English, they still struggle with nuances like accents, slang, and idiomatic expressions, which impacts their academic participation and networking opportunities. This barrier affects their confidence and hinders their integration into U.S. society and career development.
Final Solution
What is the final product?
ChatCharm is a comprehensive ecosystem that includes a pair of earring-style earphones, a gesture-sensitive bracelet, and an accompanying mobile application, all working together seamlessly, facilitating an intuitive, interactive, and efficient experience that supports real-time, casualconversations for international students with English speakers.
How does it work?
Discovery
User Research
Our initial problem space is ‘The user group struggles to communicate their ideas and thoughts successfully in English during formal settings such as meetings, conferences, and interviews (both orally and nonverbally, online and offline), affecting their social activities, peer assessments, and career development.’ We initiated our formative user research by conducting contextual interviews with six participants, and the collected data were meticulously analyzed using an affinity diagram.
6 Participants
• International students whose native language is not English. • Need assistance communicating in English in life, such as during team projects or meetings.
Method
• Each participant is discreetly observed by a team member during their meeting for more than 0.5hr on Zoom or in-person, to note behavior. • The observer will interview the participant post-meeting for about 45 min, asking questions based on their observations and preparing queries to understand the current practice and challenges.
In our data analysis, we found 4 out of 6 participants expressed they need more help with casual conversations than formal settings. Despite their proficiency in academic English, they struggled with slang and cultural topics, hindering their ability to engage in small talk for networking and integrate socially in U.S.. Consequently, we shifted our focus from formal settings to casual conversations , emphasizing small talk, slang, and career development.
Key Findings
Understanding American culture, slang and humor Users’ educational exposed them more to formal English, and much less to language and topics used in casual conversations.
Switch between English and native language Students translated most sentences word-by-word to their native language and pieced it together for comprehension, which was ineffective.
Fear of making mistakes Afraid of making mistakes, including inappropriate grammar, vocabulary, pronunciation, and fragmented sentences.
Methods used to improve English skills Interacting with native speakers provided a more holistic learning experience than traditional tutoring and rote exams.
Persona
Grace
Age: 24 Gender: Female Occupation: International master student at Ivy School
Look at Grace’s story!
Grace is a master student at an Ivy League university in the United States. She completed high school and undergraduate studies in her native country, China, where the medium of instruction was Mandarin, her native language. Grace moved to the U.S. recently in search of opportunities to advance her career. As part of graduate school admissions requirements, she took standardized TOFEL English tests to demonstrate an adequate level of proficiency in academic settings.
While Grace actively engages in discussions about schoolwork or cultural differences, as an introvert, she finds it challenging to keep up when the topic shifts to movies, novels, songs or sports, especially with native speakers. Her unfamiliarity with American pop culture and specific slang terms makes it difficult for her to join casual conversations. Often feeling excluded, she tends to be quiet in social settings and may laugh at jokes she doesn’t understand just to fit in. Bound by their shared linguistic and cultural background, Grace naturally became involved in the Chinese community. Outside of the classroom, she socializes with Chinese friends and speaks Mandarin all the time, thereby inadvertently limiting her opportunities to engage in casual English conversations.
Aiming to integrate herself into the American society and build a career in the U.S. upon graduation, Grace wants to increase her potential employment opportunities. To successfully network with professionals in her field, she wants to participate in and initiate casual conversations in English using appropriate topics and language. She also wants to feel comfortable and confident when interacting with people outside of her Chinese friend group in the U.S., especially with English native speakers.
In one-on-one conversations, Grace struggles to keep pace, especially with fast speakers, as she mentally translates between English and Chinese. Conversing with unfamiliar people, like recruiters or professors, increases this difficulty. Although she used an English language app, its focus on academic terms and lack of oral practice were limiting. However, mentoring, volunteering as a teacher, and joining diverse clubs have been helping her improve and feel more comfortable in casual English conversations.
Problem Statement
Therefore, the problem statement arises from the insights gathered during our formative UX research above:
Asian International students who study in the U.S. and want to work in this country post-graduation need to initiate and participate in casual conversations in English using appropriate topics and language because they want to network with professionals and employers in their respective fields as well as integrate into the social activities.
Ideation
Brainstorm
Merge & Selection
Real-Time Assistance
First, we prioritized real-time assistance over personal practice due to Innovation (a noticeable gap in this market) and Utility (reduces mental stress and boosts starts).
Screen-Less Solutions (e.g. smart glass, earbuds)
Ensure active participation in casual conversations while using the product politely.
ChatCharm: Earbuds, Bracelet, App
However, we excluded smart glasses due to inclusivity and aesthetic concerns. Instead, we opted for a solution involving earbuds (input), bracelets (output), and the app (control center). We refer to this system as “ChatCharm”.
Final Idea Rationale
After we have the final idea in mind, this section explains the design rationale behind each component shown in the final solution, and how the final idea works in four scenarios to finish key tasks and achieve design objectives.
Gesture setting is the key of the system as we activate audio inputs by gestures.
Ensures two other devices are visibly confirmed as connected to the phone.
Tailor topic suggestions and flashcard reviews to match content difficulty with individual English proficiency.
The onboarding process is showcased in the second prototype instead of the first.
An important design objective is to facilitate users in initiating casual conversations with relevant topics. The context, gender, age range of the interlocutor, and the conversation location are crucial factorsfor selecting appropriate topics. Therefore we designed a screen where users can easily select this information from dropdown lists. These inputs serve as prompts for AI to generate relevant conversation topics. The quality of the AI-generated topic depends on the amount of context information users provide, although all options are optional.
Another main design objective is to facilitate users’ seamless engagement in casual conversations. There are two primary challenges. Firstly, users often struggle to comprehend slangs or culture activities, which may be the conversation’s theme. To address this, we’ve implemented a design where users can triple-tap to request immediate slang explanations. The bracelet detects this gesture, identifies the relevant words, and provides brief audio explanations, helping users maintain in the conversation flow.
Secondly, users often don’t know how to respond effectively even if they understand the conversation. To address this, users can pinch all fingers, triggering the audio to offer a response sentence immediately to facilitates their smooth participation in the conversation.
Genuinely enhance users’ authentic English skills in casual conversations is another design objective. To achieve this, we designed a review feature. First, ChatCharm can record users’ informal conversations, and retain only the user’s voice to ensure privacy.
Then, We provide AI-driven feedback on those recordings, evaluating aspects such as pace, accent, vocabulary, etc.
Finally, we set practice sessions informed by review. This approach, focusing on real-life conversations, improved personalization and effectiveness of review.
Implementation & Evaluation
The process involves a logical progression of prototype development and evaluation: low-fi prototype (paper prototype) -> 1st evaluation (3 users & basic test with the individual researcher) -> mid-hi prototype (design refine & Figma & video interpretation) -> 2nd evaluation (6 users & heuristic test & refined usability test with the team) -> hi-fi prototype (design refine & Figma refine).
Low-fi Prototype
1st Evaluation
Each team member found one qualified user (so we had 3 in total) and conducted testing individually for efficiency in the 1st evaluation. Here I showed the one I tested. The key insights in this section were derived from the collective findings of all team members from their respective user tests in this round.
Participant
Hang Zhao
A Chinese junior exchange student from a Chinese university who is studying at Cornell for his first semester, with no prior experience of studying abroad.
Goals & Method
Goals
1. Know user perceptions of our idea. 2. Gain insights and feedback on the initial prototype for iteration.
Method
• We presented our design and a brief tutorial, and required users to complete some tasks using the prototype. • We listened to user feedback, discussed and recorded testing videos to gain insights into our design.
Key Insights
Positive Outcome
Basic flow & layout Easy navigation to the different pages of the app based on the sticky bottom nav tab.
Voice interaction & hardware Listening to two ‘voices’, i.e., ‘ChatCharm’ and the person talking is not distracting, as long as the ChatCharm intervention is well-timed.
Refinement Opportunities
Gestures & Customization Challenging to remember and use all gestures during conversations, users suggested customizing gestures for specific operations based on personal preferences.
Interpretation of gestures Users expressed a desire to visually understand how each gesture works.
Organization of Recordings Users want to find a specific recording easier through different perspectives.
The timing of voice indication Unpredictable gaps in conversations made timing voice indications difficult. Users agreed some overlap with ongoing conversation is acceptable.
Mid-high Prototype
In this round, we incorporated feedback from the 1st evaluation and upgraded our app’s paper prototype to a Figma prototype. The four highlighted changes below address the refinement opportunities above in the 1st evaluation correspondingly.
In this evaluation round, we employed heuristic evaluation and conducted usability testing for the mid-high prototype. Considering the limitations in time and resources, we extract the three most important insights.
Heuristic Evaluation
Research Method
We used a structured template to evaluate the heuristic of our mid-high prototype. 1. Location: where in the prototype the usability problem has been found first. 2. Description: describe the problem in the design and how the UI violates the heuristic. 3. Severity: We use a 1-4 severity scale to rate the issues (e.g. 2 is Minor which means user can recover easily) 4. Violated Heuristic: designed a new metrics extended from Jakob Nielsen’s 10 usability heuristics(e.g. error recognition and recovery). If it violates multiple heuristics, records the one most prominently violated.
Data Analysis
• To reduce bias, each of us ran an independent inspection following the method above, documented in a spreadsheet. • Then we merged, removed repetitions, and reached an agreement for each problem. • Finally, we proposed solutions to the problems, decided which ones to fix and justified decisions. The final spreadsheet identifies 31 observations and here shows an example.
Usability Test
Research Method
We recruited 3 participants who met the user testing criteria, conducting one session with each participant. Each user testing session lasted approximately 45 minutes, there were three sections and team members assumed one of three roles.
Section 2 Facilitation of all four tasks (20 mins) • Task 1: Set up gestures, • Task 2: Initiate casual conversations • Task 3: Active participation in conversations
An example!
• Task 4: Learning and review from casual conversations
Section 3 Post-session Q&A (15 mins)
3 Roles
Facilitator The facilitator led the session according to the protocol, observed and offered prepared hints if the user deviated significantly from the expected path. Finally, the facilitator assessed task success based on predefined criteria.
Actor In certain tasks, such as “Task 3 Active participation in conversations”, we needed an actor to talk with the participant and incorporate relevant slang naturally during the conversation to prompt the participant to use gestures.
Simulator Another team member used the WoO (Wizard of Oz) method to simulate the product’s functionality and managed time and recordings.
To ensure everyone gains experience in each role, we rotated roles for each user testing session (we had 3 sessions).
Data Analysis
The process closely resembles the heuristic evaluation above. In the final spreadsheet, we have identified 32 observations, and here is an example:
Key Insights
Considering the limitations in time and resources, we extracted the three most important insights after discussion and prioritized the corresponding changes in the final prototype.
Categorization of recordings on review home The current categorization of recordings by location lacks clear and descriptive titles, preventing users from quickly finding specific conversations because it does not align with their natural recall process.
Easier adoption of gestures Users tended to ignore ‘See how ->’ button and selected gestures without fully understanding their implementation.
Navigation of the flashcards The flashcard list page, segmented by “mastered” and “under review,” lacked direct links to individual flashcards, limiting user control.
Hi-fi Prototype
The final product and how it works are detailed in the ‘Final Solution’ section above. The final app Figma prototype is here.
Discussion
Impact
Concept Potential: Thoughts on Our Product and AI Pin
As our project approached its conclusion in November, the AI Pin was introduced. We noticed some similarities between AI Pin’s features and our concept, but there are substantial differences in design philosophy. While both employ gesture-activated features, AI Pin utilizes audio input through a personic speaker, creating a noticeable sound bubble, whereas our concept employs earring-style earbuds, keeping the input discreet. One prominent feature of AI Pin is immediate language translation. Our design concept takes a step further on cultural aspect, focusing on topic suggestions, slang explanations, and immediate AI-supported replies. The presence of AI Pin and the existing earring-style earbuds in the market not only validates the feasibility of our technological approach as a future wearable AI device but also highlights the untapped potential of concepts for supporting casual conversations.
Significant Casual Conversation support
During usability testing, users found our idea to be innovative and valuable, providing significant support for their integration into American society. They informed increased confidence in initiating or participating in casual conversations with our product and recognized significant potential for its application in networking, social communities, etc. Users also suggested the extending the concept to other languages and broader user groups beyond Asia.
Limitation
Product Appearance
While we opted for unisex-style earring earbuds to appeal to a broader audience, it’s worth noting that females tend to be more inclined to wear earring-style earbuds than males. This natural preference may limit our target market.
Transparency problem
Our user-centered design aimed to make this product as unobtrusive as possible, allowing users to engage in conversations naturally and without stress. However, this design raises two potential problems.
• Firstly, there’s a privacy concern. Unlike AI pins, which are visible to others when in use due to their indicator light, our product operates without any visible indicators. While it is designed to only record the user’s voice for review during conversations, the lack of visible cues may still lead others to doubt its privacy safeguards.
• Secondly, there’s an acceptance challenge. While our product is designed to enhance communication, some may question the authenticity of our users, as the presence of audio in earbuds could distract users during face-to-face conversations. Since others can’t easily discern if someone is using it or not, there’s a dilemma: our design intended to reduce prejudice but might inadvertently increase it within our user group.
Ethical Considerations
Although we aimed for our product to be an aid rather than a crutch so that we designed review panel in the app for users to improve their true English skills, the concern that some users may become overly reliant on the immediate conversation support, hindering their true progress still exists.
Key Takeaways
1. Research Design: When choosing the research method, the normal contextual interview method couldn’t work on our theme since it would break the flow of natural casual conversation. We brainstormed together and designed unique observation and testing methods of contextual interviews in formative research session to minimize bias.
2. Systems Thinking: Our prototype encompassed not just an app but also interactions with earring earbuds and bracelets. We learned to question a lot about how a small change would influence the entire system, what was the gains and tradeoffs, and finally we were able to make many great design decisions.
3. Shift in Focus: During formative user research, we discovered that casual conversation is a more serious issue for international students aiming to integrate into U.S. society than formal setting conversations, deviating from our initial assumption. We had an intense debate with all the data from the formative research and reached an agreement to shift our focus from formal to casual conversations. It turned out to be the best decision made in this project.
4. Team Contract: This project demanded extensive teamwork. We needed intense discussions to make quick decisions based on numerous data (e.g. selecting three main changes from over 60 observations during the evaluation phase). Our team contract facilitated efficient and supportive teamwork, people attended all meetings on time and stuck to the internal deadlines with high-quality work.
