Category Archives: Usability Testing

Usability Testing Content

For the past three years, I’ve been running usability tests on software applications. Most of the tests have been to measure and iterate on designs and interactions.  However, lately many of my projects my team has taken on have been related to *testing content*.  I realized that I needed a refresher on how to test content, and wanted to share with you what I’ve been reviewing. Highlighted below are some of the methods identified in the UX community to help evaluate whether the content on your Web sites is useful for your customers and prospects.  Not surprisingly, testing content is actually very similar to testing design/interactions.

Content Testing Methods:

Iterative Testing. Kevin O’Connor and Colleen Jones’s 2010 IA Summit talk: “Testing Content: Early, Often, & Well” is a great resource on testing content (here is the audio). Not surprisingly, she talked about how important it is to test content iteratively, just like we test designs and interactions.  She will often start with a baseline test of the content that is on the live site, then follow that up with concept testing, and then do a validation test.  What was interesting was how she focused the *protocol* on three key content questions:  Can users find and read the content they need? Do they understand the content? Can and will they *act* on the content?

5 Second Tests. Christine Perfetti wrote an excellent article: “5-Second Tests: Measuring Your Site’s Content Pages.”  Using this method, you’d basically show participants a web page for 5 seconds, take it away, and then ask them for their initial impressions.  Christine says you can ask them to describe back to you what they saw.  She says users make important judgements in the first moments they visit a page.  This method helps to uncover those judgements.  Also, it can measure whether the calls to action on the page are apparent enough.  She describes a case study with the American Red Cross, where 5 Second Tests helped them iterate on the design of their donation page, to make sure users knew what all the donation options were (i.e., donating airline miles, stock, clothes, etc.).

Inherent Value Tests. Christine Perfetti and Jared Spool talk about this method in a couple of articles, and in a podcast.  They say you’d want to run an Inherent Value Test when your team needs to know how well a Web site communicates the inherent value the designers have put into it, and whether new customers understand the true value of the service.  How it works:  recruit two user groups, in the first phase, you recruit existing users, and interview them about what they like about the product (what they find valuable).  This will help to identify features loyal customers miss.

Eye Tracking, Gaze Plotting, and Web Analytics. I talked with a our Technical Writer, Maureen Lau, who had recently attended the Nielsen Norman Group‘s course on “Writing for the Web.”  I learned from her that the Nielsen Norman Group tests a lot of content with Eye Tracking, Gaze Plotting, and Google Web Optimizer.  Eye Tracking tests how long they look at a particular area (the more red, the more they looked at it).  Gaze Plotting, tests where their eyes jump to.  She said you go through a similar process with content as you do with interactions (paper prototyping, iterating, trying to understand users’ needs, etc.).  She said Eye Tracking and Gaze Plotting are great to see where people look, and what catches their eye, however you need to interview people at the same time, to see if they are actually comprehending what they have looked at. They will use the Google Web Optimizer to roll out an AB test, and see which content “performs” better.  She told me about how readers typically read in an F-shaped pattern.  Finally, she said we want to create tasks around the content. Since users often search for keywords, they suggested getting a better sense of what SEO keywords we use, and testing to see if they look for those key words, where they’d expect to look for it, etc.

Testing Content Concepts. Colleen Jones wrote an excellent article called “Testing Content Concepts” that walks through the specific protocol, probes, and questions to ask participants about when testing content.  Colleen says a “Content Concept” is a mockup or draft of your content.  She says if your run a usability test on your content, and your content is not working well, you should fix the problems and test again until it does. Colleen recommends testing in three levels of fidelity:  content only, content in a wireframe, content in polished design.  What I found most interesting in her article was her recommendation to focus observations on how usability participants work with the content. I also liked her recommendation of measuring how successful people are at reading, understanding, and remembering key messages in the content. Colleen discusses the biggest challenge in testing content:  measuring how the content has influenced the participant to take action. She says “don’t press too hard, and thus, end up with misleading rationalizations.” One idea Colleen has that could work well for us, is to conduct a closing questionnaire  that asks whether participants would now make a decision that differs from their typical decision, and to rate how well the content informed their decision.

I’ve never been so convinced of how critical it is to test not only how to navigate to content and how to find content, but to test the content itself.  It is the content itself that drives influence and calls readers to action.

If you have any comments, questions, or alternative methods to test content, please leave a comment below!

Seven Tips for Presenting Usability Issues to Stakeholders

In a previous life, I was a Library and Information Science masters student, and part-time reference desk librarian. I worked in the Biomedical library, connecting medical students and professionals to journals, articles, research, studies, books, knowledge…I attended medical library conferences and studied how to help medical professionals with “evidence-based practice.” As a
result, I became adept at helping people find evidence to base their practice on in medical research literature. That skill has certainly served me well as a user experience researcher as I work closely with designers and stakeholders, and now unsurprisingly, finding and delivering evidence to inform design decisions is my absolute favorite part of this wonderful job.

I’m in the thick of user research on my current project, and along the way I’ve been reminding myself of all sorts of best practices I learned from the brilliant Joan Kaplowitz, an Information Literacy Librarian.  Using her advice, I do everything I can to make sure the people consuming the information I present, are getting what they need and informing the design questions they have.

My hope is what works well for me, using Joan’s principles, will help you too when you have to communicate your results to a team of stakeholders.

1. Engage your stakeholders

“Active learning is better than passive reception for retention and transfer of learning.” Joan Kaplowitz

In my experience, the best way to engage stakeholders, and have them actively learn usability issues is to involve them in the research.  Having them come to observe usability test sessions and contribute their observations to an affinity diagram is a very effective way for them to retain the feedback they hear, and see trends across users.  One strategy I learned from attending the Nilesen Norman Group Usability Bootcamp in 2007 was from Kara Pernice: have your observers take notes on post-its (quotes, observations, one issue per post-it), and then change color post-its for different participants. Then you can start to group issues together with stakeholders, and see usability trends across users (colors). Your diagram of groups then can turn into your report, transcribing your wall of insights into a presentation slide deck or any other format.

2. Keep it lightweight

“Less is more. It is better to teach a few things well than to overwhelm learners with so much information that they become frustrated, anxious, and unable to retain anything from the instruction.” Joan Kaplowitz

The best way to describe this principle as it applies to reporting usability findings was best articulated by Todd Wilkens from Adaptive Path: “The effectiveness of your research report is inversely proportional to the thickness of it’s binding.”  You will be so much more effective in making change happen, and getting people to take action on your findings, if you don’t overwhelm them with new information. Your goal is to inspire your stakeholders to address the usability issues, and have designers solve the usability problems.  Then, validate those changes were successful with another round of testing!

3. Get them talking

“The instructor’s voice should be the one heard least during teaching.” Joan Kaplowitz

The best presentations I’ve ever given have been because I met with stakeholders ahead of time to get a sense of what they learned from watching usability sessions. Get designers to tell you what they learned, because it doesn’t matter so much what is in your head, as it does what is in theirs. Before giving your final presentation to the larger group of stakeholders, talk with your key stakeholders, and have them tell you what they learned, observed, and took away from the sessions.  Review the takeaways you intend to present, and invite them to add to it. Your final presentation will be far more effective at inspiring change to happen.

4. Grab them in the first 5 minutes

“You win or lose your audience in the first five minutes. Talk to your learners as they enter the room. Provide them with something to do and/or think about as people are arriving. Engage them from the start.” Joan Kaplowitz

Sometimes the beginning of the findings presentation is the most challenging, because all your great insights are coming later in your slide deck.  However, the beginning of a presentation ALWAYS sets the stage for the message you are giving. Get to the presentation room early, and build rapport with each person who comes through the door.  Great them with a hello, and say their name (people love to hear their own name).  The goal before the presentation starts is to get “audience members” on your side, because in my experience, these are the folks that that are most likely to help fend of hecklers in your defense! Also, you want a strong introduction.  You definitely don’t want to stumble through your first 2 minutes of speaking.  Here is a basic introduction that I use often – but the more creative you can be the better!

“Great, let’s go ahead and get started. Today we are going to talk about_____________. We have about 45 minutes for the discussion, and I’ll leave some time at the end for questions, but if you have any questions along the way, please feel free to ask. First I want to ask… who had a chance to observe the sessions?  Great, well if there is anything you would like to add that you remember observing, please feel free…”

5. Demonstrate each key issue

This advice I learned on the ground floor at the UCLA Biomedical Library reference desk.  Students, nurses, patients, etc. would come to the desk asking how to use the library databases.  The most effective way for them to learn how to use the databases wasn’t just telling them, but actually demonstrating it to them. In fact, it was best if you could hand the keyboard to them, and have them drive themselves.  In an usability findings presenation, you really want to re-create the issues that participants had during the session.  Stakeholders tend to be quite visual, and greatly benefit if you show them what you mean.  Take full screen shots, and simulate the experience of using the technology using basic animations such as fading in.  This seems to be the best way to re-tell the story, and have your findings really sink in with stakeholders.

6. Keep track of your effectiveness

“Always include a way to assess your outcomes. Otherwise you will have no idea if your learners have attained them.” Joan Kaplowitz

In the world of UX Research, the best way to assess your outcomes is to keep track of which issues are on the plan to getting fixed, and which ones have not been fixed. You can keep track of these in an Excel spreadsheet on your own, but the best is to have your issues make it to the product roadmap.  You want to see your usability issues turn into “user stories” if you are working with Agile teams, or possibly the roadmap, bug fixes, or “change requests” if you are working in a waterfall environment. However your organization keeps track of changes that need to be made to the system, do what you can to make sure usability issues are represented there.

7. Take a multi-format approach

“Use handouts, web pages, and pre and post activities to extend your contact time with your learners. Offer ways for learners to keep the conversation going after instruction is completed — through email, blogs etc.” Joan Kaplowitz

A PowerPoint presentation is a great way to walk through usability findings.  However, you don’t want to only deliver a slide deck. In addition to your presentation, you will want to give a “hand out” to your key stakeholders, the people who need a “check list” of things to address.  Don’t rely on them taking their own notes during the presentation, give them a list that they can easily make notes to.  You don’t need to pass this out to everyone, just to your 1 or 2 key stakeholders. Also, I’ve seen a follow up email work really well, with a link to your presentation, to the recording of the discussion, and a couple bullets that summarize the key takeaways.  Inviting users to continue the conversation using wiki pages is a great way to keep them engaged, and continue with next steps.

So there you have it. This is my very best advice on how to present usability findings so that it inspires your stakeholders to take action. Many of the information literacy principles listed here apply to basing design decisions on evidence.   How do you communicate your usability issues?