Posts Tagged ‘bias’

It’s the questions, stupid: less biased and more accurate tests

October 7, 2011 4 comments

True or false: User testing is about getting into the head of the user?

gorgeous bald-headed Natalie
Pictured: the head of a user.

Before I began this post I would told you that absolutely, user testing is all about getting into the head of the user. Why else would we ask them questions and go to such great expense to make those questions accurate, valid and reliable test instruments if not to get in their head?

Liar, liar pants on fire…

But we know that people often don’t mean what they say. Studies have shown that in a typical 10-minute conversation, the average person will tell 2.92 falsehoods. In essence, we filter what we say based on our context and situation. Much of what we say is coded for transmission based on the receiver. That’s a nice way of saying: I’m going to tell you what you want hear. It’s just a little white lie. Don’t worry about it, it’s not gonna hurt anybody.

It’s easier to do with speech than with actions. Our physical bodies have their own way of communicating. We call it body language.

Body Language

It’s been said that “hips don’t lie”.

She’s got a point.

But your body is constantly broadcasting what you’re feeling. Micro gestures in the face can tip off a liar or whose guilty. It’s how magicians guess names. Witness in the below video how the “magician” guesses the name of the ex-boyfriend by asking the woman a rapid series of questions by which he can observe her facial micro gestures.

Did you know that your feet are the most honest part of your body? Almost nobody pays attention to what they are doing with their feet. You can observe openness or defensiveness by whether the feet are in front or are behind and crossed or behind and wrapped around the chair legs.  (You checked your feet, didn’t you?)

Check out the nervous guy at the end of the bar… What is he doing with his hands? Posture?

I’m pretty sure he’s just happy to see me.

What does this mean for the evaluator or user test creator? Does it mean we have to check to see if they’re packing heat or if they’re Shakira? Are these examples helping?

What it really means is that it’s very difficult to write (verbalize) a test question and not tip off your bias.

It seems to me that most of the standard test questions are leading the users. For example, “How do you feel about this test?” or “Tell me about your Mother.“.

What if the user doesn’t feel anything about that? Once you force someone to generate a feeling, I figure they fall back onto their default response. For some it’s a negative response. Some, it’s a positive response.

Tester default response (call it tester bias? Tester prejudice?) must be accounted for. Personally, I believe in the “innocent until proven guilty” adage. The website I’m viewing is presumed innocent of design mistakes, until it breaks a design rule. Websites are presumed innocent. Finding out this tester default response is key to understanding user testers.

But it that getting into their head?

All of this is academic and probably doesn’t really effect in a measurable way the outcome of tests-especially at this basic level. However I feel it is good to visit and share these thoughts as we encounter them. The pitfall is thinking, “I will not past because I can’t be scientific in my measurement.”

Here are some ways I plan to alter the script based on my understandings stated above:

Tasks first, impressions later

For the practical user testing scenario, I suggest tasks first and impressions later. Allow the user to work with the site in form and impressions in a natural way. I understand that the research shows that you only have 5 seconds for someone to determine whether or not they want to stay on that page versus clicking the back button. But, in the testing environment it doesn’t really translate. Perhaps a good compromise is that ask them to make a personal note, not to be shared, about their first impression. And then, after the tasks are complete, ask them to revisit that first impression and see how it changed. This might be more beneficial and give a truer sense of the users impression.

What this protects against is my initial reaction to prejudice. By asking me my impressions after only 5 seconds you are really just determining my prejudice. All sites are innocent until proven guilty of wasting my attention or stealing my cognitive load.

Go ahead it’s okay to say it stinks

There has got to be a way to set the environment-and I’m talking about actively setting up the testing environment- to be open and honest. That doesn’t mean setting nice music playing and making sure you talk in a soothing voice-that crap would set me on edge. In such a clinical environment, I would feel less inclined to give my actual opinion. To be frank I don’t know exactly how to do this. Anonymous testers might feel more inclined for negative feedback. Perhaps the sandwich method of positive-negative-positive could be used. Are remote tests better than face to face?  Strangers vs Friends?

I can’t do it with you watching me

Big EyesSeriously. Stop that.

Being observed makes me nervous. It’s unfortunate because user testing is all about making observations. The point I want to make is in order to make the environment open and honest for feedback-both positive and negative-it could be important to make the recordings and note taking inconspicuous. People understand they are being recorded, but will quickly return to a more natural expression when the recording isn’t conspicuous.


The whole point of user testing is to observe users interacting with your design with the intent to improve that interaction. There will always be some degree of bias in both the evaluator and the tester. It’s not required to eliminate it in order to find the majority of  problems. However, if you think about and make a few changes to the standard scripts for user testing you can get a less biased and more accurate test of actual user experience.


Do you have to get into user’s heads for a good user test?

How do you set the tone for a user test in order to elicit honest and open feedback?

Do you account for the natural, tester baseline / default response?

Do you, in a systematic way, observe and measure body language from the testers?

Here’s How User Experience Testing Can Be Better

September 30, 2011 Leave a comment

Last week we drafted a usability test and tested one user in order to get a real experience with the theories and abstractions we were researching and discussing. Our results were surprising.

What We Wanted

We wanted to figure out a repeatable process for conducting a user test that would improve upon the simple watercooler test – The type of test that my friend at conducted. Additionally we wanted to “user test the user test” by way of quickly getting feedback on an early draft of our test scheme in the hopes of creating a more effective testing tool. And, we wanted to dive in.

What We Did

I downloaded and edited a script from the site which came from the SUS framework and a usability test from The way I customized it was to look at each page type in the JavaJack’s site and create a task that would engage the user with that page.  I didn’t make the tasks ‘hard’ but I didn’t want them to be too specific.  I didn’t want to test the users ability to read, for instance.  Our friend, Anna, came by just in time to be the tester.

What We Got

The results of the very unscientific test were intriguing and beneficial to the overall design of the site.  As Anna talked thru using the site with Ben and I there watching, we noticed some things that could be improved and changed. Time well spent.

After the test, our thoughts turned to the testing process itself.  Ben’s post explains his thoughts on the critical path and the purpose, goals and objectives of the sites and how they relate to user testing. Here are other thoughts about the user-test and the testing process in general.

Design Dunce? Maybe. Personal Experience Expert? yep.

Design Dunce? Maybe. Personal Experience Expert? yep.

User’s aren’t designers, don’t ask them to critic a site design

Most scripts I’ve seen ask the user this question in the beginning of the test:

Please give me your initial impressions about the layout of this page and what you think of the colors, graphics, photos, etc.

When you asked this question, it seems that the users become unsettled and defensive.  They have to form an opinion and defend it. They stop using the site.  They critique the site instead. They start to look at the site as a designer.  The user is tainted after that simple little question.  Ask the user about their own experience… and why not ask them AFTER they have the experience.

To fix this we came up with a few ideas. Let them complete the tasks and then ask them about their experience.    Record the session (the screen, webcam and audio) without you in the room. Just don’t ask the question, there are better ways to get initial reactions from users – the user-testing sites in the sidebar.  Or, conduct separate tests for reaction and impression.

Fly on the wall

Fly on the leaf?

Hiring UX facilitators: Flys apply within. Humans need not apply

You want to be a ‘fly on the wall’  as much as possible.  Humans suck at this.  Evaluator bias is rampant and I’m doubtful you can eliminate it. Simply put it’s when the testing-user feels that they should answer a certain way or feel the agenda of the test questions.  Who doesn’t feel that in every survey!  Human societal norms get in the way here (Perhaps not in New York City, granted).  People tend to be polite to strangers and people in authority. I figure negative answers questions are rare.  Users will try to figure out what you want and try to give you that.   The act of testing will influence their use. I feel the designer is the least desirable person to be the facilitator of the test. If the user feels that the facilitator has an agenda or prefers one outcome over the other, then the test is compromised.

You can’t hire actual flys to do your testing (they don’t make lab coats and clipboards small enough).  Here are a few ideas we had to correct the problem on Evaluator Bias and human factors.  You could use a remote testing service – like the ‘Mouse Tracking Tools’ in the sidebar. Run face to face tests in a familiar place to the user – office, coffee shop, mall, home – so the user is more comfortable giving honest opinions. The facilitator should not be perceived as someone affiliated with the site – regardless if they are or not – nor an authority of any kind.  Perhaps, you can test several other sites to hide the site you are actually testing (This might be time/resource intensive)

Something is better than nothing

It works better than nothing.

Something is better than nothing.

However, doing any type of test is better than nothing. The simple act of watching somebody go through the site is very desirable. Testing gives you insight into the flow of the site, if the site is mechanically (or functionally) sound and working and if the user finds and stays on the critical path or primary site goal.

Alien canals on the planet Euphorbia Pulcherrima

Only natural

You sellin’ what I’m buying? Great, let’s do this.

Each user has their own goal when coming to the site. Each site owner has a goal when building a site.  If the two goals match, Great! Now get out of the way and let the site churn out money.  The user clicks thru the site and their goals is met.  This click trail thru the site is called the Critical Path.  And, it’s what you should test in the ‘Tasks’ portion of the standard usability test.

There can be many paths, but only one critical path on a site.  For example, I go to to watch movie trailers. Would Apple user-test my experience in getting to and watching movies? Perhaps, but I bet they measure how easy it is for me to ‘jump over’ and buy some music or a new computer. The purpose – the critical path –  of the site is to sell (I’ll give you branding and customer service, as additional paths) and all other features and functions of the site support that goal.

How well the site moves visitors along the path is the effectiveness of the site.  And, we test for it by asking testers to assume they have the same goal as the site.  Likewise, we test for satisfaction. Did they complete the task, but were pissed because of something else? Did they expect one thing and get an unpleasant surprise?  Also, we test for efficiency. Did they complete the task, but it took 15 clicks and 20 minutes to complete?

You can test this by doing a very simple user tests. That’s the low hanging fruit of user tests.  Site effectiveness, Satisfaction, and Efficiency.

In conclusion,  don’t ask about impressions before your evaluation of the critical path. Do test even if the conditions are not scientific. Let users use. Visitors visit.  Don’t force them to have an opinion and then needle them about it. This is a carry over from the designers perspective.  User’s aren’t lab rats. Your color palette isn’t of supreme importance.  Listen close and you can hear a user. They are probably saying, “Just give me the dang banana already”


Is the user qualified to speak to design?
How do you get around the evaluator bias?
Can one site have multiple critical paths?