Posts Tagged ‘questions’

User-Testing in 3 Questions: Why Drive-By User Testing Works

December 7, 2011 1 comment

[Note: In Monday’s podcast I said we were going to be doing a real user test today. We’ve decided to push that back to next week. We had two more items we wanted to get to first.]

When we first started learning about user testing back in August, I was under the presumption that user testing was essentially a controlled, systematic endeavor. But as Steve Krug and Jakob Neilsen will tell you, there’s more than one way to test a cat. (That’s a direct quote, I believe.) A lot can be gained just by getting feedback from somebody. They argue that the name of the game is improving your website (not testing a hypothesis) and that most major issues on a website are visible to most users.

Krug advocates companies set aside one morning a month to do three user tests. Then, debrief over lunch.

This rapid-testing idea was inspiring. It’s like everybody knows what’s not working on your site, and to learn the secret to what’s broken, all you have to do is ask them to tell you about it. Whoa!

You said it.

But is that true?

Are we all walking around with an innate sense of what sucks on websites? And if so, why can’t we design a perfect one out of the gate? If we all have the magical power of user testing, why can’t we user-test our own site and save the time, cost, and hassle of asking other people?


The reason you can’t user test your own website is because you’re too close to it. You’re like a director who can’t enjoy his own movie in the same way as a regular viewer because you remember what happened off screen. When you look at your own website you remember the versions of the site that almost were and you know intimately the reasons and compromises that went into creating your current website. You can’t see the forest for the trees.

Your average user is focused on a completely different experience. They are focused on what game theorists call “utility-value maximization”. In plain English, it means that people are goal oriented on websites. It’s built into the way the Internet works. We click on links because we want to access new content. Each click is a statement of purpose. When a user visits your website, they are there for a reason and they have a goal in mind. Han Solo had it right about web design when he said, “Give the wookie what he wants.”

Mental Models

Have you ever helped a relative get a computer or get on the Internet or otherwise interface with a completely new piece of technology for the first time? Do you remember the first time you opened Photoshop or Illustrator or Dreamweaver or the first time you tried to center something with CSS and you had that feeling that you were in completely new territory?

That happens when you have no way to contextualize what you’re seeing with your previous experience.

People create mental models of all their behaviors. Have you ever caught yourself having a dumb conversation with somebody about the weather when (a) you don’t really know why you’re talking to them in the first place and (b) there’s nothing special about the weather? Why do we do that?

Or how about in movies and TV shows when people hang up the phone without saying goodbye first. That’s always weird to me. It’s because my mental model includes goodbye language at the end of phone conversations and in TV shows, it doesn’t advance the plot so it isn’t necessary.

Once you get your non-techie relative online, they too will begin to create a mental model of how the Internet works. Some of it will be based on fact. A lot of it will be based on feelings and intuition that is not correct. (I’m reminded of an old boss who, every time he saw the Blue Scree of Death on his PC would scream “BILL GATES!”) But people don’t get retrained when they learn the Internet wrong. Nope, we just deal and hope they catch up.

Jakob Nielsen points out many of the points of technical confusion in his blog post on the same topic.

  • Operating-system windows vs. browser windows
  • A window vs. an application,
  • Icons vs. applications,
  • Browser commands vs. native commands in a Web-based app
  • Local vs. remote info
  • Different passwords and log-in options (users often log in to other websites as if they were logging in to their email)

In short, when it comes to computers, a lot of us are still getting our act together. What happens as people gain experience using the Internet is they gain an intuition for how websites are supposed to work. And that’s why there’s a great list of usability conventions to use when developing your website.

What does this mean for web sites and user testing?

Signal vs. Noise

All communications come down to the issue of signal vs. noise. Those of us old enough to remember when Sprint was a long distance company who had sound quality so clear you could hear a pin drop have experience with the problem. In any communication medium, whether it’s spoken and heard, or transmitted by radio, TV, print, telegram, fax machine, or Internet, the message must be transmitted and it must be received. The message is encoded in a language and in a medium, and in a context. In order for it to be decoded correctly, the person receiving the message must understand the language, have access to the medium, and know the context in which the message is meant.

See? Easy.

Think of a physical long-distance phone line. The reason Sprint was so happy about their clear signal is because it’s hard to eliminate all the noise. Static, buzzing, clicks, pops – all could (and did) effect the ability to be heard.

When you think about a telephone line, the communication can be broken down into two parts:

  1. The mechanical workings of transmitting a person’s voice from one place to another and,
  2. The content of the communication

If you were to user-test the phone line, you could test for the same two attributes above:

  1. The quality of message transmission and,
  2. The content of the message

Unless you own a tin-foil hat, phone companies today don’t care about the content of your phone calls, texts, and messages. They are focused on delivering on the quality of message transmission. Also, because we all have such a complete mental model on how to make a phone call, it turns out that isn’t the part of the phone business that’s growing now. The user experience for making and receiving phone calls is essentially complete. The new wrinkle is welding the digital side of things to the phone. Now it’s all about experiencing the mobile Internet. And that, invariably, leads to more half-baked mental models for using the Internet.

The Medium vs. the Message

Like the double rainbow dude, we’re left to wonder, “What does it all mean!?”

Let’s cut to the chase.

When you add up the fact that there are so many ways to leave a page – from going to another site, clicking the back button, closing the window, getting up and walking away, clicking into a different window, getting distracted, plus the various mental models that people have for how the Internet should work, it’s safe to say that there’s a lot of noise in the channel.

In the physical long-distance phone line we discussed earlier, the medium was the phone line. In your website your critical path is the medium.

Now let’s think about that for a second. Remember how earlier we said that clicking a link is a statement of intent. It could be that the user is expecting to end up on a YouTube video or is expecting to see an Amazon page for digital cameras or is looking for an article on Wikipedia or a house on an MLS. The content is the message. The Internet is the medium. And getting to your content is part of the medium.

We’ve all dropped a call on our mobile phone. There’s a distinct difference between a phone call where you’ve said everything you wanted to and hung up and when the call drops. When the call drops – you can’t communicate any longer. In the same way, if your website can’t get the user to the content they’re looking for, then it can’t deliver the message. It’s a failure of the medium to deliver the message.  The noise in your website has to be low enough so that people can find the signal they are searching for.

This signal, the message, is what the message says. It’s the literal content of your website. This is why developing a strong critical path is so crucial. The only way to be heard above the din of distraction is to have a focused message and to provide an obvious way to access that content.

What we’re talking about here is signal quality. And all that’s necessary to test a signal is somebody on the other end to describe what they’re getting.

If we were testing long-distance phone service we’d whip on some glasses, stomp through the woods and ask “Can you hear me now?“. But since we’re testing websites, all that’s required is to ask somebody else what they see on your website.

User Testing in 3 Questions

When you’re looking for quick insights about your website, you’re looking to know something about signal strength. It’s evident in the type of questions that are asked.

Take the basic questions from’s test:

  1. What frustrated you most about this site?
  2. If you had a magic wand, how would you improve this site?
  3. What did you like about the site?

None of them concern the content on the website. All of them are testing either signal or noise.

Break down the expected answers:

  1. What frustrated you the most? 99% of the time this is going to be an answer that relates to not being able to find something. It could also have something to do with your messaging.
  2. How would you improve the site? Because you directed them to the site and not what your site sells or provides, this answer is going to relate somehow either to accessing information or clarifying options.
  3. What did you like about the site? This will help with positive feedback and will help you confirm areas of the website that perform well or stand out to the user.

Nothing in there explicitly dealt with the quality of the material. What’s important is strengthening the signal.

I should state that these are not the only questions you could use in a 3-question test. You can test the critical path, a secondary path, or test for comprehension and cognitive load with your site structure without knowing anything about the user.

And we’re going to prove it. Next week, with a little luck, we’ll bring you a video of our own 3 question drive-by user test.

How to Hire a Better Web Developer

November 2, 2011 Leave a comment

It’s hard to be a small business owner. You’re literally responsible for everything.  When we hire outside help, we’re looking for someone to solve a problem, not become a new one.  As is often the case, when shopping for a professional, it’s often what you don’t know that can hurt you.

Often times when we get into these relationships for the first time, we don’t know what to look for. So we end up with sub-par results and think it’s par for the course.

But what if you knew going into the initial meeting the most important factors to consider when hiring a web developer? That would practically guarantee that you’d have a better relationship with your developer, would be better served by that developer, and they would turn out a better product.

So what are they?

We’ve compiled 5 of the most important questions to consider when looking to hire a new web developer.

Is This A One Night Stand Or Are We Making This A Relationship?

The first thing you need to decide is what you’re looking for. Are you looking for a new website? To refresh an old website? For somebody to manage your Internet presence?

Are you looking for a partner or are you looking for a specific product?

To put it another way, if you were looking to hire a mechanic, would he be changing your oil or would be become your go-to guy for your car’s maintenance?

This is an easy one: you either need something specific or you’re looking for someone to look out for your larger Internet interests.  Decide what you want and make sure they want that back.  A long-term relationship is asking them to become a stakeholder in your business, at least as far as the website is concerned, and as such, is no trivial thing.  This is different that using the same guy for a number of years when you need website updates.  A real long-term relationship is based on a Internet marketing plan.

Do You Masturbate?

I don’t mean to be blunt, but seriously… do you?

I once had a boss who fired that question off to a new marketer who had just joined our team.  The guy stammered back “uh.. yea…yeah”.  And my boss said, “Good! Now I know you’re honest. Now we can do business together!”  Admittedly, tossing around questions like that is a good way to attract a harassment lawsuit but I never forgot the lesson: you need to trust the members on your team.

If you’re in for a long term relationship with a web developer, you need to know that you’re getting good, honest communication from them.

What do you do, exactly?

You’d be surprised how many web developers (or web designers) forget what their job is.  They think it’s about designing websites. It’s not. It’s about making you more money. If they don’t know that, then you can bet that the website they build for you will not be very good at doing that.

Does the web developer have any strategies in place to identify and fix underwhelming, broken, or under-performing parts of your website?

How do you develop and maintain my Internet presence?

On today’s Internet, a company website is the beginning of how you interact with your customers online, not the sum total.  The whole thing – website, Facebook, YouTube, etc. – is your ‘Internet Presence’ and somebody has to manage it.  Sure, the web developer can set up a Facebook page for your business, but can they effectively weave all of the social tools along with your website into more than the sum of its parts?

If not, then I would seriously consider if I wanted to be in a long-term relationship with this developer.  We all need short-term work done from time to time and not everybody has to be able to see the 35,000 foot view.  But those that do will provide you a better product.

When it goes wrong, who’s wrong?

Invariably, at some point something’s going to go wrong. Whether it’s not enough traffic to the website, a website that doesn’t convert at a high enough rate, or the whole endeavor seems like its turned into a money pit, you want to know who’s wrong.  The best way to make people accountable for their mistakes (whether it’s you, the client, or them, the developer) is to set goals, set a time for accomplishing those goals, and then to measure the activity undertaken to reach those goals.  Regular goal setting and assessment, along with a period of testing and improvement should be part of every website strategy. If your web developer doesn’t talk about those things, they’re not going to take responsibility for how your site works.

These five questions will go a long way towards making sure that you get the right web developer for the job.

It’s the questions, stupid: less biased and more accurate tests

October 7, 2011 4 comments

True or false: User testing is about getting into the head of the user?

gorgeous bald-headed Natalie
Pictured: the head of a user.

Before I began this post I would told you that absolutely, user testing is all about getting into the head of the user. Why else would we ask them questions and go to such great expense to make those questions accurate, valid and reliable test instruments if not to get in their head?

Liar, liar pants on fire…

But we know that people often don’t mean what they say. Studies have shown that in a typical 10-minute conversation, the average person will tell 2.92 falsehoods. In essence, we filter what we say based on our context and situation. Much of what we say is coded for transmission based on the receiver. That’s a nice way of saying: I’m going to tell you what you want hear. It’s just a little white lie. Don’t worry about it, it’s not gonna hurt anybody.

It’s easier to do with speech than with actions. Our physical bodies have their own way of communicating. We call it body language.

Body Language

It’s been said that “hips don’t lie”.

She’s got a point.

But your body is constantly broadcasting what you’re feeling. Micro gestures in the face can tip off a liar or whose guilty. It’s how magicians guess names. Witness in the below video how the “magician” guesses the name of the ex-boyfriend by asking the woman a rapid series of questions by which he can observe her facial micro gestures.

Did you know that your feet are the most honest part of your body? Almost nobody pays attention to what they are doing with their feet. You can observe openness or defensiveness by whether the feet are in front or are behind and crossed or behind and wrapped around the chair legs.  (You checked your feet, didn’t you?)

Check out the nervous guy at the end of the bar… What is he doing with his hands? Posture?

I’m pretty sure he’s just happy to see me.

What does this mean for the evaluator or user test creator? Does it mean we have to check to see if they’re packing heat or if they’re Shakira? Are these examples helping?

What it really means is that it’s very difficult to write (verbalize) a test question and not tip off your bias.

It seems to me that most of the standard test questions are leading the users. For example, “How do you feel about this test?” or “Tell me about your Mother.“.

What if the user doesn’t feel anything about that? Once you force someone to generate a feeling, I figure they fall back onto their default response. For some it’s a negative response. Some, it’s a positive response.

Tester default response (call it tester bias? Tester prejudice?) must be accounted for. Personally, I believe in the “innocent until proven guilty” adage. The website I’m viewing is presumed innocent of design mistakes, until it breaks a design rule. Websites are presumed innocent. Finding out this tester default response is key to understanding user testers.

But it that getting into their head?

All of this is academic and probably doesn’t really effect in a measurable way the outcome of tests-especially at this basic level. However I feel it is good to visit and share these thoughts as we encounter them. The pitfall is thinking, “I will not past because I can’t be scientific in my measurement.”

Here are some ways I plan to alter the script based on my understandings stated above:

Tasks first, impressions later

For the practical user testing scenario, I suggest tasks first and impressions later. Allow the user to work with the site in form and impressions in a natural way. I understand that the research shows that you only have 5 seconds for someone to determine whether or not they want to stay on that page versus clicking the back button. But, in the testing environment it doesn’t really translate. Perhaps a good compromise is that ask them to make a personal note, not to be shared, about their first impression. And then, after the tasks are complete, ask them to revisit that first impression and see how it changed. This might be more beneficial and give a truer sense of the users impression.

What this protects against is my initial reaction to prejudice. By asking me my impressions after only 5 seconds you are really just determining my prejudice. All sites are innocent until proven guilty of wasting my attention or stealing my cognitive load.

Go ahead it’s okay to say it stinks

There has got to be a way to set the environment-and I’m talking about actively setting up the testing environment- to be open and honest. That doesn’t mean setting nice music playing and making sure you talk in a soothing voice-that crap would set me on edge. In such a clinical environment, I would feel less inclined to give my actual opinion. To be frank I don’t know exactly how to do this. Anonymous testers might feel more inclined for negative feedback. Perhaps the sandwich method of positive-negative-positive could be used. Are remote tests better than face to face?  Strangers vs Friends?

I can’t do it with you watching me

Big EyesSeriously. Stop that.

Being observed makes me nervous. It’s unfortunate because user testing is all about making observations. The point I want to make is in order to make the environment open and honest for feedback-both positive and negative-it could be important to make the recordings and note taking inconspicuous. People understand they are being recorded, but will quickly return to a more natural expression when the recording isn’t conspicuous.


The whole point of user testing is to observe users interacting with your design with the intent to improve that interaction. There will always be some degree of bias in both the evaluator and the tester. It’s not required to eliminate it in order to find the majority of  problems. However, if you think about and make a few changes to the standard scripts for user testing you can get a less biased and more accurate test of actual user experience.


Do you have to get into user’s heads for a good user test?

How do you set the tone for a user test in order to elicit honest and open feedback?

Do you account for the natural, tester baseline / default response?

Do you, in a systematic way, observe and measure body language from the testers?

The Ben and Newman Show Podcast #004

September 12, 2011 Leave a comment

This week’s podcast is a good one. Honestly, I thought we were going to end up trashing based on Newman’s experiences with the site last Friday but instead we saw the light and figured out how to put it to good use. We also devised a Deathmatch event between and Feng-GUI and Newman will be sharing the results in his blog post on Friday. It’s a lot of fun.

Check back on Wednesday for a great blog I’m cooking up on the most useful Google Analytics reports for small business owners.

And, if you haven’t already, be sure to subscribe to our podcast on iTunes – new episodes are available every Monday!

1:15 – Chatting with your buddy about User testing

2:30 – Newman Yawns

3:00 – Talking about and

5:13 – Rushing through a  Designer or User fault?

7:00 – Train users to use

8:00 – 5 seconds to judge a website?

8:28 – lyric site vs ecommerce –

10:30 –  mp3 buying UX – size of the market –

12:00 –  Valid test?  out of context search

13:32 – confuse the users – Hidden tester, crouching usability expert

14:50 –  “sifting out sites” – the search process –

15:40 – “Not about yes, about getting rid of no.”

16:30 – Beta testing?

17:50 – Who is qualified to create and analyze these tests?

18:30 – Generic questions okay?

20:50 – 5 second thought exercise

21:40 – “What would you want someone to notice in the first 5 seconds?”

24:15 – “Why would you design a page for 5 seconds? New users”

25:47 –  user testing  and demographics

28:10 – Talking smack… but not throwin’ down … on

29:45 – A / B testing

30:32 –

31:29 – Basic training on how to create / analyis a user test

32:45 – Three waves of web design focus

34:04 – science journalism comic

36:10 – Changing goals interior to websites -searching vs. finding

38:00 – Newman loses his brain

39:30 –, ripping it?

42:30 – Smackdown Feng-GUI vs

43:50 – 6 revision post on Introduction text and Headlines

44:52 – a/b test on the Bella Florals with headline change

48:00 – Gaming the system.

48:50 – We are Attention Wranglers

49:40 – Attention / Conversion funnel

50:10 – Elevator pitch to the girl

50:40 – one goal per page?

52:50 – Wrapping it up – Steven Pressfield, Alexander, Joe Rogen

Categories: podcasts Tags: ,

The Ben and Newman Show Podcast #003

September 5, 2011 Leave a comment

Newman has the bright idea this week to give you guys show notes for our podcast. You’ll see links to The Booth at the End and a video on the double-slit experiment below. Be sure to check those out. Quantum physics is a favorite topic of mine and The Booth at the End is a really great show. It’s not directly related to web usability but as you’ll hear in the podcast, you can see a vague similarity.

And that ??:?? below… that goes to the validity and reliability part of the podcast. We can’t remember when we talked about it but we didn’t want to deprive you of the link. So, bam!, there it is.

Podcast Show Notes:

2:41 – The Tools
5:20 – The Booth at the End
12:17 – Motorcycle
13:30 – Terminology
15:51 – Quantum Physics
17:54 – Justify Design Decisions
21:00 – Monkey wants a banana
25:00 – Design of e-commerce
27:00 – Summary

??:?? – Validity and Reliability

The Double-Slit Experiment by Dr. Quantum

Categories: podcasts Tags: , ,