Augemented reality for stroke recovery with Wabbit

Stroke is the leading causing of physical disability in the United States. Nearly 800,000 people suffer from a stroke annually causing serious, long-term impairments.

Team size

3

Roles

Design, research, prototyping

Duration

3 Weeks

Tools

Sketch, Framer, AE

Emerging technology for The AGH Comprehensive Stroke Center

This project was created for The Comprehensive Stroke Center at Allegheny Health Network to develop a solution for stroke patients in any phase ( before, during or after) with the help of emerging technology.

We conducted a rapid sprint of discovery, ideation, and validation and quickly iterated through problem spaces and solutions. Inspired by the success of FarmVille and similar applications across the demographic that also most commonly suffers from strokes, we built out the concept for Wabbit, an augmented reality mobile game designed for motor skill recovery of stroke patients.

Model from Google Poly

Getting started

Using Augmented reality and machine learning, Wabbit creates an exercise plan for the patients and converts these into game actions. Wabbit combines the soothing and playful environment of farming, uses augmented reality combined with ambient sounds to create an immersive experience and uses the game actions to help convert the benefits of physical therapy into engaging rehabilitation programs.

The game opens with a list of tasks that it creates for the patient based on physical therapy exercises recommended for them.  The game starts with exercises suitable for all patients and modifies them with machine learning based on how the patient responds and progresses.

Mobile phone use during rehabilitation

We were a little worried about designing a mobile application to help people that struggle with motor skills, and at the same time excited to explore hands-free AR. As it turns out, by the time patients are discharged from the hospital, individuals are able to walk around and perform basic tasks, but they're not confident to venture outside the comforts of their homes – their mobile phones are their only easily accessible escape.

“Patients are either well enough to be sent home, or in such a state where they have to remain in the hospital - insurance companies don't let anyone fall inbetween those categories, because at-home care is expensive. ”

Interview with Dr. Eugene Bonaroti, MD

Neurosurgeon, Allegheny General Hospital

One-handed interactions

To facilitate smooth interaction and reduce strain, the entire interface can be used single-handedly.

One handed interaction menus help patients navigate easily between tasks lined up for the day and select the one that they like.

Besides these simple interactions, users are provided with easily accessible interaction cues. An assistive animation is over layed on top of the game action to help them practice the proper exercise form.

Progress and retention

At the end of every day, the user is presented with a comprehensive progress report. Every small action is counted, making it seem like they've achieved a lot. When the patient fails to complete any of the given tasks, the game encourages them that "tomorrow is a new day!" and tells them what they can do and how it will help them. If Wabbit detects that the user's task completion rates and completion times are slowing down (indicating struggle), Wabbit adjusts its difficulty.

In our research, we found that it's incredibly important to make users aware of their progress, as progress is slow and frequently plateaus, leading them to give up on traditional self-motivated rehabilitation programs. Without family or caregivers to make them aware of it, they are no longer aware of their long-term progression (which is always more evident to bystanders than the individual). Getting this feedback from the application then becomes of paramount importance.

Merging physical and digital

Traditional physical rehabilitation relies on weights and other equipment to make the exercises more challenging as stroke victims regain motor control. Although some of this increased difficulty can be simulated through repetition, we explored the idea of specially manufactured 'smart' objects that interact with the AR game environment.

Integration beyond placing these manufactured objects in a user's environment also makes the gameplay more immersive. Imagine for example if a user could score points or complete farm tasks for taking a shower, making coffee, or watering actual plants. This gradual transition to unassisted daily living is the true goal of any rehabilitation program, and IoT devices can play a significant role in enabling that.

Discovery: Stakeholder map

To kick off our project, we started building a stakeholder map based on our own (naive) understanding of the domain and desk research. The goal of this stakeholder map is not to create an accurate picture of the entire domain, but rather highlight interesting problem spaces within the domain, and determine where potential friction could emerge.

Our literature review was especially valuable in as common problems tended to surface frequently. As a result, we narrowed down on the intersection of the stroke centre, insurance, patient (both before, during and after stroke phase) and existing tools and solutions for these stages.

Focused analysis and competitve research

A study of the chosen areas from the stakeholder maps led to the following insights. During the study, we did competitive analysis of existing products that help with stroke and also went over articles, educational videos and papers related to the subject. From this research we figured that we wanted to focus on the recovery phase. This decision was made based on the following insights:

Recovery time and disability

The first few months of stroke recovery are absolutely crucial. After this certain doors simply close, and recovery of those areas is no longer possible. Not only can this impede recovery, it may also lead to a second stroke which is often times more fatal.

Limited insurance

Insurance generally covers only the first few months of therapy. Even then, systems are set up to make patients 'too well to be covered by insurance' or bad enough to stay in the hosptial. Once sent home, many patients don't have access to expensive rehab programs.

Lack of caregivers

Patients at times may not have caregivers available if they can not afford post-hospital treatments (not covered by insurance), or family is not available. The Wabbit application can serve as their companion throughout the recovery process.

Long recovery times

The full recovery period after a stroke ranges between six months and two years. With such extended recovery times, it is hard for patients to commit to monotonous recovery routines and exercise plans.

Addressing research insights

Having scoped a problem space and explored possible solutions, we were now able to draw from targeted academic research, and identify key experts that could provide unique insights into.

To apply all the insights from the research, we decided to do design sprints involving activities such as "Reverse assumptions and 20 questions" ( down left) and crazy eights ( down right) which helped us think very creatively of the ideas. It was out of these crazy eight seeions that we first came up with the idea of "virtual gardening" which the team decided to channel into a farming game.

Brain development

We found a lot of literature studies that demonstrated that specific game actions can be targeted to develop specific parts of the brain. Using a game for recovery, we waim to provide entertainment along with brain enhancing benefits.

Independence

Wabbit can be used by any stroke patient who has been sent back home after their insurance period. All these patients have the basic motor skills to hold the phone and the game requires no further assistance. This lets stroke patients take recovery back into their own hands.

Custom-made plans

All currently available solutions offer no customization, and act the same for all types of patients. Wabbit uses machine learning to make subtle changes in exercises and daily plans based on user response and performance.

Motivation

Because the recovery period is long and progress plateaus at multiple stages, it is important to keep the patient motivated. This is especially important for patients who have no caregiver support. Wabbit has daily progress screens that shows them how far along they have come.

Validation: Expert interviews

“... stroke patients who have been sent home have sufficient motor skills to use a mobile phone ”

Dr. Eugene Bonaroti, MDt

Neurosurgeon, Allegheny General Hospital

We interviewed Dr. Eugene Bonaroti to understand the problems that stroke patients face during recovery, and validate some of the earlier assumptions we made regarding drop-off in rehab programs and the ability to use mobile devices . While at first we were toying with the idea of going for an AR-glass based solution, we now had the confidence to stick with mobile.

“Tasks involving repeated limb movement are great for the motor skill recovery of stroke patients.”

Deborah Josbeno

Associate professor of physical therapy, University of Pittsburgh

Josbeno responded positively to some of the initial drafts we showed her, and underscored the importance of repetitive movement in the recovery of upper-limb motor skills. Josbeno also mentioned that we should put extra care into correctly translating the movements required for recovery into the game actions – movements should be task oriented and provide sufficient challenge.

Microinteractions and prototyping

We wanted to build a very pleasurable and soothing game experience for our users. We experimented with sound design and microinteractions, and prototyped the entire flow from launch to completing tasks and seeing the end-of-day report in Framer.

Next steps and recognition

To kick off our project, we started building a stakeholder map based on our own (naive) understanding of the domain and desk research. The goal of this stakeholder map is not to create an accurate picture of the entire domain, but rather highlight interesting problem spaces within the domain, and determine where potential friction could emerge.

To kick off our project, we started building a stakeholder map based on our own (naive) understanding of the domain and desk research. The goal of this stakeholder map is not to create an accurate picture of the entire domain, but rather highlight interesting problem spaces within the domain, and determine where potential friction could emerge.