by Dana Chisnell, Amy Lee, and Janice (Ginny) Redish
This year, AARP has been conducting exploratory studies of its own web site, AARP.org, to find out how well the site works for its intended audience.
Thirty-four people from San Francisco, Tampa, and Baltimore participated in individual sessions: 15 people in their 50s, 9 in their 60s, and 10 in their 70s.
For more on how we did the studies, see Older Users Help Us Study AARP.org. For more on study results, see Comparing AARP's Study to Earlier Findings.
In this article, we will give tips on
- Finding participants: Understanding older adults before you recruit
- Recruiting older adults
- Screening older adults
- Scheduling sessions with older adults
- Reminding older adults of important points before they come
- Working with older adults during sessions
Finding participants: Understanding older adults before you recruit
There are 64 million Americans aged 50 to 74, according to the U.S. Census Bureau. According to a Harris Interactive survey done at the end of 2002, there are 36 million adults age 50 and over who are online (48%). Of those age 50 to 64, 61% or 25 million are online. Of those 65 and older, 32% or 11 million are online.
We only needed 36 participants for our usability studies. (We ended up with 34.) Why were they so hard to find?
The first question is where to find them.
These ideas did not work well:
- Community web sites, message boards, or chat sessions. Older adults tend not to take part in these groups, so posting ads in those places is not a fruitful way to find participants.
- Senior centers and community colleges. These are places that offer classes in using computers. If you want computer and web novices for a study, they might be good places to find appropriate participants. They weren't for us because we wanted people with enough web experience for us to see them working at the site.
- Flyers at a senior center, when they did not make clear what we were doing. Many older people are much more cautious and skeptical than younger people. They are often fearful of being cheated or "taken." For example, we had put up flyers at a senior center from which we got no response; later we learned that people thought we might be trying to sell them something.
- Cold calling from a database. This is probably again because they're afraid that they may be scammed into buying something.
Because we did not want people to practice before they came, we did not tell them that the study was for AARP. We were also looking for both AARP members and non-members. Therefore, we did not mention AARP when we were looking for, recruiting, or screening potential participants. If we had, we might have greatly increased interest and credibility; but we might also have had difficulty recruiting non-members and possibly skewed the performance in the study sessions.
These ideas did work for us:
- Calling with a personal connection. If we could say that a mutual acquaintance had suggested the contact, potential participants were much more receptive to hearing about the study and considering taking part.
- Being careful in the initial call to say where we had gotten the contact information and that we weren't selling anything.
We also recommend:
- If your company or organization's privacy policy allows, and if it fits with your study plans, contact customers or members to take part in studies.
- Consider implementing a link on your company or organization web site where people can volunteer to take part in studies.
- Hire a reputable market research or recruiting firm that has large databases or "panels" of voluntary candidates.
Recruiting older adults
Recruiting participants in their 60s and 70s is more difficult than recruiting participants in their 50s. The oldest candidates are less receptive to strangers phoning them, and they don't check email as frequently.
Recruiting by phone
Phoning is important. Plan to phone potential participants at least once (or have your recruiters do so). You need to quickly establish credibility and trustworthiness, to assure potential participants that you are not selling anything, and to establish a connection by letting them know where you got their names. When you can do that, potential participants are often glad to hear from a real person. It is also easier for them to determine legitimacy and to ask questions about the study on the phone. They'll use your answers to help them decide whether they want to take part.
Another reason for phoning potential participants is so you can judge their English language skills and whether they are hearing impaired. (You may well want to include limited English speakers and hearing impaired users in your study; if you do, you want to be aware of these specifics about the participants before they come.)
Recruiting by email
Email can be very efficient for younger participants; it's less so with adults in their later 60s and 70s. Give yourself more time for these older participants; they generally don't check their email as often. (Many don't even check it every day.) This happens for a variety of reasons: They don't feel the need to check mail frequently. They use a computer at a senior center or a library. They have limited time available through their Internet service provider.
As suspicious as older adults are of telemarketers, they are also vigilant about spam. If your email address is unknown to them, without an appropriately descriptive subject line, they may delete it. We had this happen with follow-up and confirmation emails as well as introductory emails. Always put on a very clear subject line.
Screening older adults
Screening older adults demands specificity. Many older users when asked the question "what do you do on the web?" answer "email." They also often don't think about practical activities such as banking or bill paying online as "using the web."
Many older users are also not as familiar with the language of the web as younger users are. They don't distinguish between the Internet and the web. They don't always know the difference between the web browser and the web page.
We found that self-reported data about frequency of use and numbers of hours spent online were not good indicators of proficiency, either. For example, we had one participant who spent 60 hours per week online. We didn't find out until the session started that her sole use of the web was playing games on four web sites that her friend had set up as separate shortcuts on her desktop.
So, asking a variety of specific questions to gauge potential participants' familiarity with the web can help the recruiter make judgments about how suited the person might be for the study. Even if you're looking for a mix of proficiency levels, you still have to be able to determine where in the range a potential participant fits.
We created profiles of levels of web proficiency similar to these:
| Profiency | Description |
|---|---|
| Very familiar | Spends 15+ hours on the web per week exclusive of email; uses a variety of sites routinely; has been using the Web for more than 2 years. |
| Somewhat familiar | Spends 5-15 hours on the web per week exclusive of email; uses 1 or 2 sites routinely; has been using the Web for 1-2 years. |
| Novice | Spends at least 2 hours on the web per week exclusive of email; uses 1 or 2 sites routinely; has been using the Web for 6 months to 1 year. |
To determine the level of proficiency, we included questions like these in our screener:
13. Do you use email as part of your day-to-day
routine?
_____ Yes; how often do you check it? ____
[Continue]
_____ No [Reject]
14. How many hours do you spend on the web or
Internet in an average week, NOT
including
the time you spend reading and
writing e-mail?
_____ [Record number. If less than 2 hours,
reject]
15. How long have you been using the web for
activities
other than sending and
receiving email?
_____ Less than 6 months [Reject]
_____ 6 months to 1 year [Accept up to
3]
_____ 1-2 years [Accept up to 4]
_____ More than 2 years [Accept up to
6]
16. What web sites do you visit every week?
Name at least 3 web sites,
please.
If candidate has difficulty thinking of specific websites, prompt: Do you look at news web sites? Do you play games or take part in message boards or chat rooms? Do you do banking or bill paying online? Do you track investments? Do you shop, or buy and sell things online regularly? Please tell me the web address for those web sites.
We're looking for evidence that they actually use the web. Use of sites like eBay to buy and sell items indicate a fairly high level of sophistication about using the web. Looking at statements online but not doing banking or making trades can indicate a somewhat familiar candidate.
Scheduling sessions with older adults
Scheduling sessions with older participants can present some logistical challenges that you might not think about in studies involving younger participants.
They arrive early
Because many older people are retired (or at least have ample free time), they almost always arrive for their sessions early. In a few cases, we've had participants show up an hour early. Be sure to have someone to greet them and set up a comfortable place for them to wait.
They bring their spouses
Older participants often bring their spouses with them. They may have traveled some distance to get to the session; they may have planned activities for after their session; or they simply may not like driving alone. Have magazines, a phone, and a comfortable chair available for the spouse.
They do best in the morning
Even though people in their later 60s and 70s are vital and energetic, they usually have more— and better— attention to give earlier in the day. Try to schedule people who are in their late 60s,70s and 80s in the morning and save any afternoon sessions for participants in their 50s or early 60s. We don't recommend running evening sessions.
They don't like driving in rush hour
If you are holding sessions in a central place (rather than meeting participants in their homes or workplaces), schedule them outside of peak traffic times, if possible.
Reminding older adults of important points before they come
When they got to the session, a few of our participants became nervous and uncomfortable as they realized that they would be the only participant in the session. Usability studies are still fairly new to the general population. The recruiting firms we use to get participants are more often recruiting for focus groups, and participants who come through these firms usually assume that they are coming to a focus group.
Reminders about videotaping and observations
Although we asked for permission to record and to have people observing the sessions when we recruited, people tend to forget that. Make sure that the person who calls the participant to confirm the session also tells the participant that:
- "You will be videotaped and observed by people you won't be able to see during the session."
- "This is a one-on-one session. You will be the only participant in the study room with a moderator."
Special reminders for older adults
Computer glasses
Many of our participants had special glasses for using the computer. So another important reminder is
"Don't forget your computer glasses!"
Eat first
Also, for long sessions— ours were two hours long— ask participants to make sure they eat before they arrive. Because many participants expect to take part in focus groups rather than individual sessions, they also expect to be fed. If you have snacks available, try to have fruit and nuts or other relatively healthy food. Many older participants are diabetic.
Working with older adults during sessions
Many older participants won't know what to expect coming into a usability study session. Be clear in setting their expectations and be firm but polite about keeping the session focused on what you're trying to find out.
Make participants comfortable
Be respectful without being patronizing. You can be a neutral moderator but still be polite. "Please" and "thank you" are important. Many older adults expect more statements of politeness like these than younger participants do.
Older participants also deserve extra consideration, politeness, and detailed information about the session. They will feel more comfortable if they know what to expect up front. We recommend that you:
- Clearly explain the session plan, timing, and what they can expect.
- Warn participants that you'll interrupt them and that you may stop them before they have completed tasks.
- Schedule breaks for long sessions (and tell them they can take breaks whenever they need to).
- Have them practice thinking aloud.
- Consider including a practice task to help participants understand how the session will work.
- Take account of beliefs that participants may have learned or created about how to work with computers.
- Remember that older participants often are not versed in computer and web terminology, so avoid using this jargon when working with them.
- Be extra patient with older participants; wait longer than you normally might to prompt; consider giving participants permission to ask for hints.
- If participants stop talking, consider letting them continue that way; try reflecting on the task later.
- Teach participants something at the end of the session.
Keep them on track tactfully
Most of the participants we've had in sessions are interesting, charming, and very talkative. Many older participants have a lot of stories to tell. Their stories say a lot about who they are, and where they have been— and often provide a context for interpreting data.
But it may be easy for participants to get off track during the session, and while it may feel awkward or mean sometimes, it is the moderator's job to keep the participant focused on the task, talking about it, and getting data for the test. This is the main reason for warning participants in the introduction to the session that you may interrupt them and that you may stop tasks before they've completed them.
Listen for their beliefs about computers and the web
Many people in their late 60s and older never used computers in their work, so they have no previous experience from which to make inferences about how a computer or an application might work. Many learn how to use computers and the Internet through friends, family, and neighbors. They inherit the superstitions and myths that those people have developed to help themselves work around problems. Then the older adults bring these myths into sessions with them, and you'll hear about them as task-solving strategies and workarounds. It's important to capture these; they are part of the users' reality and we have to deal with these beliefs when we design web sites.
Be careful of the words you use; avoid computer jargon
Older computer users rarely know much about computer-related terminology, so you should avoid using these terms during your sessions. Older participants often don't know the names of widgets such as drop-down boxes or cascading menus. Most of our participants also had little knowledge of web-related terminology. For example, they weren't sure about terms such as "link," "URL," and "login." Many were unclear about the meanings of "online community" and "message boards." "Browsing" wasn't always meaningful in the context of a feature called "browse by topic." The word "emoticon" and the concept behind it were completely foreign to most of our participants. This means that you must pay close attention to what participants do and point at on the screen.
Give them time
Older participants almost always take longer to do tasks than younger participants. And, although they seem to struggle, the oldest participants also expect using the computer to be difficult. Plan for tasks to take much longer for older participants than they would for younger participants— up to 25 percent longer in our experience.
Help participants understand the time constraints of the session by explaining the session format in your introduction. Also, wait longer to prompt than you normally might. You might also consider giving participants permission to ask for hints when you introduce the session.
If necessary, hold the think aloud and ask participants to reflect later
When tasks become complex or difficult, participants may stop talking. Use your best judgment about nudging them to tell you what they're thinking. For some participants with short-term memory loss or other cognitive impairment (such as that caused by pain medication), your asking for their thoughts may interrupt their task enough that it causes them to make errors. In those cases, you may get more usable data without the think aloud protocol by asking participants to reflect later.
Don't lead even when you want to
If, as a session moderator, you have a soft spot in your heart at all for participants, working with older participants will exercise that spot a lot. You may be tempted to give hints; worse, you may lead them in ways you don't intend. Be patient and firm but polite while keeping to your agenda.
When appropriate, teach something at the end
If the session has been difficult for the participant, or, if there is some small thing that would make using the computer or the web easier, take a little time at the end of the session to teach the participant something. For example, one participant really liked how the mouse worked, and complained that his was "sticky." At the end of the session, we showed him how to take the ball out and clean the rollers. He was delighted. We've also shown participants how to change the text size in their browsers and shortcuts for copying and pasting and printing.
In summary: We started out with assumptions about recruiting older participants that did not all turn out to be accurate. For example, we thought that if they used computers and the web, they would know web vocabulary. For another example, we thought that they would check their email often. Our experiences with recruiting participants for our three studies showed us some important differences in the patterns of use of the web for older adults, especially those in their 60s and 70s. These differences have important implications for where to find participants and how to recruit, schedule, and work with participants. We hope that our experiences can help you arrange successful usability tests with older adults.
