AVVA - Accessible
Future for Self Driving Cars

traffic21,
carnegie mellon

Voice Interface that increases independence for visually impaired riders by enabling their use of autonomous vehicles

PROBLEM

Associated Services for the Blind in Philadelphia, PA

Visually impaired people have a mobility problem. Public transportation and paratransit are suited to their needs but are unpredictable and limited use.

What happens when there is no human driver?

Calling a rideshare or taxi is convenient, however the future of these services (shared autonomy) will not have any human drivers to help users.

Current designs are not meeting the needs of those who stand to benefit the most from a self-driving ride hailing service, the visually impaired population.

Solution

AVVA (Autonomous Vehicle Voice Assistant) is a conversational user interface that engages visually impaired individuals inside an AV throughout the entire ride, pickup to drop-off.

Augmenting the current ride share experience to account for the removal of human drivers, allows users to take full advantage of the consistency and convenience of self-driving ride hailing. Providing a transparent and personalized experience - resulting in greater personal agency and control.

Implementing AVVA throughout an autonomous rideshare fleet will bring unprecedented levels of independence and efficiency to user’s daily lives.

Welcoming & initiating the rider

AVVA welcomes a visually impaired user and introduces the system, with triggers and utterances. This information is available throughout the ride.

Partnering with the user during the ride

AVVA helps with orienting, navigation, route options, and acts a concierge by narrating surroundings both inside and outside the vehicle.

Full service agent checkout & review

At the end of the ride, AVVA provides specific exit instructions, and a complete checkout and review process.

AVVA Conversation Design

USING SPEECH AS A GUIDE

Familiar and responsive

Speech is a familiar method of interaction. AVVA uses that to answer the unique questions of visually impaired riders, and present information in a way that allows users to customize and trust their ride - adapting to their needs over time.

Modular design system

The modular conversational voice interface guides a visually impaired user through the entire ride and is coded with JQuery based variables, events, and methods, ready to be deployed.

Welcome & Initiation

WARM & FRIENDLY TONE
Using advanced authentication AVVA verifies that they are the correct passenger, and are going to the right destination. AVVA introduces the available functionality.

Users can re-route or add details about their journey that are added to their preferences.

Route Awareness & Orientation

AGENCY THROUGH TRANSPARENCY
The user feels safe knowing that they have access to any kind of information they need - questions about the vehicle and it’s response to route and road conditions.

Navigation and wayfinding is descriptive and clear, giving just enough information without overwhelming the user.Re-routing is always an option to give the user agency over their journey.

Concierge Partner

TRIGGERS FROM USERS, INTERNAL & EXTERNAL EVENTS
Designed based on triggers from user utterances as well as internal and external events. Variables, events, and methods are coded into the responses.

AVVA helps users within the cabin, making them feel comfortable and at the center of the experience by narrating their surroundings.

Errors & Escalation

TRUST THROUGH ERROR HANDLING
AVVA works through errors with gentle clarifying questions and some levity to maintain the relationship.

For personal and physical safety reasons, a customer support agent is available on demand for either further error handling and escalation.

Trip Conclusion, Review & Payment

RESPONSIVE TO UNIQUE NEEDS
At the end of the ride, AVVA notifies the user of their surroundings, and how to leave the vehicle.

Context about the external environment is triggered both proactively and in response to questions.

To conclude the full operations of the system, users can use AVVA to leave a ride review and pay for it right then and there.

How We Got Here:
Research & Analysis

Understanding a Unique Population

Industry Analysis Takeaways

PRIMARY METHOD OF PERCEPTION IS AUDITORY
Other perception channels are limited in their use cases and are not flexible to many of the diverse environments.

TECH DEVELOPMENT ISN’T ACCOUNTING FOR ACCESSIBILITY
Most new future-looking product research is focused on getting the functionality to work, but it is not including an accessible feature set.

“SMART” TOOLS ARE MOSTLY USELESS
Technologically enhanced tools like smart canes, headphones, and goggles often transmit information that the user already knows.

Insights from Stakeholder Interviews & Academic Research

We discovered three critical pieces of information that are required in order for visually impaired people to perceive their environment.

Geographic information at street scale and within the confined space of the vehicle, understanding of their physical position in relation to their destination, and how info is presented about the context (exterior and interior).

1. NAVIGATION & WAYFINDING

Inconsistency in routes, features, and workarounds disrupts the traveller as they memorize paths.

2. ORIENTATION

Users rely on organic sensing to understand their environment (street noise, traffic, crowds, construction).

3. INFORMATION PRESENTATION

To be useful, information has to be available on demand, real time, and approachable with a familiar interaction pattern.

Diving into the transportation experience

Primary Research & Synthesis

After gaining a high level understanding of the environment, we needed to understand the transportation experience on a personal level. Through a partnership with Associated Services for the Blind in Philadelphia, we interviewed participants for phone, and in-person sessions.

RESEARCH GOALS
Understanding what information is necessary, and what is the best way to present the data points. Noting current and recent transportation modes, experiences, and behaviors.

ANALYSIS
Grounded Theory
Using our knowledge of the themes uncovered in the interviews, we were able to quickly analyze our data points, without getting bogged down in contextual inquiry.

METHODOLOGY
We used secondary research to inform our interview questions and experiment design in order to get the most actionable data.

Phone Interviews (7)
Interviewing individuals about their current experience with using public transportation, paratransit, and taxi services.

In-Person Ride-Alongs & Interviews (2)
We observed participants as they took a rideshare to a set destination, noting down all of the questions and issues that happened during the ride.

THE Fundamentals of a smooth trip

Insights from Interviews & Observation Sessions

The interviews and ride-alongs yielded a set of crucial themes that are necessary for a successful ride. Meeting all of these needs is the key to creating a product that is actually useful to visually impaired people.

1. SAFETY

Obstacles at drop-off points present a dangerous challenge if there is not enough info.

2. TRUST

Uncertainty, inconsistency, and opaqueness in info and service immediately erodes trust.

3. EFFICIENCY

Users look for a service that affords them agency, control, and independence.

4. RELATIONSHIPS

A personal connection is the key to making riders feel comfortable and relaxed.

Ideation & Prototyping

What IF there is no human driver?

Brainstorming on Information Presentation

From our research, we knew that the key is in the information presentation. Using a matrix consisting of modalities on the y-axis and the main themes on the x-axis, we explored ideas ranging from tangible interfaces like refreshing braille, to voice control.

FEATURES THAT USERS NEED
Referencing the primary and secondary research insights, we noted which features are necessary and which questions we want the interface to answer.

MODALITIES FOR INTERACTION
Although auditory perception is the primary channel for visually impaired people, touch plays a large part in their sensing.

Voice Control is Most Effective Method of Interaction

While tangible interfaces offer unique interactions and novel ideas, these are the opposite of what makes a consistent and easy-to-use experience.

USERS ALREADY USE VOICE CONTROL ON THEIR SMARTPHONES
Leveraging that interaction pattern and seamlessly transitioning it to the interior of a vehicle afford low adjustment times and learning curves.

VOICE INTERACTION CAN HAPPEN IN ANY PHYSICAL SPACE
Users don’t have to search the inside of the cabin for physical affordances.

PROTOTYPING Voice InterACTIONS

Testing with Users Using a Wizard of Oz Script

Before building out the full conversational script, we tested the script with users in low fidelity, using a barebones script and being flexible to real-time changes and feedback from the users.

IN-PERSON SESSIONS IN A CAR (2)
To get as close as possible to a real environment with real risks, we placed our participants in a car and ran the testing from inside.

PHONE SESSIONS (2)
Although it is not as immersive as in-car environment, we ran prototype testing over the phone. The audio perception channels were the same, and we were able to get quality feedback from more people.

Creating the conversational persona

We tested both a neutral and an efficient mood for the persona. Some research participants expressed the need for quick access to information, while others wanted more conversational tone. We tested both.

Insights from prototyping sessions

Design Needs to Consider the Entire Context of Use

Changes to weather, environment conditions, traffic, varying light conditions, and the users’ existing knowledge. Multiple routes need to be generated in the event a primary route isn’t available. Contingency plans must be readily available.

Give riders the agency to control their experience, at all levels of independence

Technological affordances need to be as salient as possible so access to information is simple

Provide easy access to escalation and intervention methods for personal and physical safety

Make route & journey information easy to access and change during the ride

AVVA is designed based on user’s needs and feedback

FLEXIBILITY
Catering to most levels of vision impairment, mobility, and independence.

ADAPTABILITY
Customization to user’s preferences and needs over time, by adding to a rider profile.

RESPONSIVENESS
Addressing actual user’s pain points and struggles.

TRUST
Achieved through communication & transparency.

Productization & Roadmap

AVVA as a source of data

Road testing the prototype

The first step to productization is building a model to test on the road along a predefined route, with populated meta-data that users interact with in a gated environment.

More modules & features = more user data

With every set of modules, there is greater opportunity to collect use data and improve features. By streamlining the responses and the information being presented, we can target user’s needs and feed them useful information when they need it most.

Roadmap & production phases

The project is split into phases on implementation based on modules that need to be built, integration necessary (sensors & APIs), and the data to be collected from every section.

Next Steps & Takeaways

Evangelizing our SOLUTION

Publishing our findings

The research and prototyping that we have conducted is new to the industry, so we hope to publish our findings and get it included in the next rounds of research.

Rideshare inclusion

Rideshare companies are all working to create in-car interfaces, starting with tablet devices and eventually expanding to voice interfaces. By using AVVA as part of their system, companies can create an accessible and inclusive user experience and widen their customer base.

Digital “curb cut” for all of shared autonomy

Just as most accessibility features end up being extremely useful to the broader population (for example a wheelchair accessible ramp on a sidewalk), AVVA introduces many features that can be helpful to all users.

Empathy is key to design

Finding ways to create low fidelity first

No matter the product, a low fidelity prototyping method should be employed in order to vet ideas, validate concepts, and quickly sniff out what will work. Properly prepared users will be able to suspend disbelief and give you actionable feedback at even the lowest levels.

Unique populations have unique problems

When designing for a population that has unique problems, one of the most important strategies for successful design is to gain empathy first. Whether their struggles are known, or they come out in the research phase, it’s crucial to understand them on a deep level.