User-Testing in 3 Questions: Why Drive-By User Testing Works

[Note: In Monday’s podcast I said we were going to be doing a real user test today. We’ve decided to push that back to next week. We had two more items we wanted to get to first.]

When we first started learning about user testing back in August, I was under the presumption that user testing was essentially a controlled, systematic endeavor. But as Steve Krug and Jakob Neilsen will tell you, there’s more than one way to test a cat. (That’s a direct quote, I believe.) A lot can be gained just by getting feedback from somebody. They argue that the name of the game is improving your website (not testing a hypothesis) and that most major issues on a website are visible to most users.

Krug advocates companies set aside one morning a month to do three user tests. Then, debrief over lunch.

This rapid-testing idea was inspiring. It’s like everybody knows what’s not working on your site, and to learn the secret to what’s broken, all you have to do is ask them to tell you about it. Whoa!

You said it.

But is that true?

Are we all walking around with an innate sense of what sucks on websites? And if so, why can’t we design a perfect one out of the gate? If we all have the magical power of user testing, why can’t we user-test our own site and save the time, cost, and hassle of asking other people?

Proximity

The reason you can’t user test your own website is because you’re too close to it. You’re like a director who can’t enjoy his own movie in the same way as a regular viewer because you remember what happened off screen. When you look at your own website you remember the versions of the site that almost were and you know intimately the reasons and compromises that went into creating your current website. You can’t see the forest for the trees.

Your average user is focused on a completely different experience. They are focused on what game theorists call “utility-value maximization”. In plain English, it means that people are goal oriented on websites. It’s built into the way the Internet works. We click on links because we want to access new content. Each click is a statement of purpose. When a user visits your website, they are there for a reason and they have a goal in mind. Han Solo had it right about web design when he said, “Give the wookie what he wants.”

Mental Models

Have you ever helped a relative get a computer or get on the Internet or otherwise interface with a completely new piece of technology for the first time? Do you remember the first time you opened Photoshop or Illustrator or Dreamweaver or the first time you tried to center something with CSS and you had that feeling that you were in completely new territory?

That happens when you have no way to contextualize what you’re seeing with your previous experience.

People create mental models of all their behaviors. Have you ever caught yourself having a dumb conversation with somebody about the weather when (a) you don’t really know why you’re talking to them in the first place and (b) there’s nothing special about the weather? Why do we do that?

Or how about in movies and TV shows when people hang up the phone without saying goodbye first. That’s always weird to me. It’s because my mental model includes goodbye language at the end of phone conversations and in TV shows, it doesn’t advance the plot so it isn’t necessary.

Once you get your non-techie relative online, they too will begin to create a mental model of how the Internet works. Some of it will be based on fact. A lot of it will be based on feelings and intuition that is not correct. (I’m reminded of an old boss who, every time he saw the Blue Scree of Death on his PC would scream “BILL GATES!”) But people don’t get retrained when they learn the Internet wrong. Nope, we just deal and hope they catch up.

Jakob Nielsen points out many of the points of technical confusion in his blog post on the same topic.

• Operating-system windows vs. browser windows
• A window vs. an application,
• Icons vs. applications,
• Browser commands vs. native commands in a Web-based app
• Local vs. remote info
• Different passwords and log-in options (users often log in to other websites as if they were logging in to their email)

In short, when it comes to computers, a lot of us are still getting our act together. What happens as people gain experience using the Internet is they gain an intuition for how websites are supposed to work. And that’s why there’s a great list of usability conventions to use when developing your website.

What does this mean for web sites and user testing?

Signal vs. Noise

All communications come down to the issue of signal vs. noise. Those of us old enough to remember when Sprint was a long distance company who had sound quality so clear you could hear a pin drop have experience with the problem. In any communication medium, whether it’s spoken and heard, or transmitted by radio, TV, print, telegram, fax machine, or Internet, the message must be transmitted and it must be received. The message is encoded in a language and in a medium, and in a context. In order for it to be decoded correctly, the person receiving the message must understand the language, have access to the medium, and know the context in which the message is meant.

See? Easy.

Think of a physical long-distance phone line. The reason Sprint was so happy about their clear signal is because it’s hard to eliminate all the noise. Static, buzzing, clicks, pops – all could (and did) effect the ability to be heard.

When you think about a telephone line, the communication can be broken down into two parts:

1. The mechanical workings of transmitting a person’s voice from one place to another and,
2. The content of the communication

If you were to user-test the phone line, you could test for the same two attributes above:

1. The quality of message transmission and,
2. The content of the message

Unless you own a tin-foil hat, phone companies today don’t care about the content of your phone calls, texts, and messages. They are focused on delivering on the quality of message transmission. Also, because we all have such a complete mental model on how to make a phone call, it turns out that isn’t the part of the phone business that’s growing now. The user experience for making and receiving phone calls is essentially complete. The new wrinkle is welding the digital side of things to the phone. Now it’s all about experiencing the mobile Internet. And that, invariably, leads to more half-baked mental models for using the Internet.

The Medium vs. the Message

Like the double rainbow dude, we’re left to wonder, “What does it all mean!?”

Let’s cut to the chase.

When you add up the fact that there are so many ways to leave a page – from going to another site, clicking the back button, closing the window, getting up and walking away, clicking into a different window, getting distracted, plus the various mental models that people have for how the Internet should work, it’s safe to say that there’s a lot of noise in the channel.

In the physical long-distance phone line we discussed earlier, the medium was the phone line. In your website your critical path is the medium.

Now let’s think about that for a second. Remember how earlier we said that clicking a link is a statement of intent. It could be that the user is expecting to end up on a YouTube video or is expecting to see an Amazon page for digital cameras or is looking for an article on Wikipedia or a house on an MLS. The content is the message. The Internet is the medium. And getting to your content is part of the medium.

We’ve all dropped a call on our mobile phone. There’s a distinct difference between a phone call where you’ve said everything you wanted to and hung up and when the call drops. When the call drops – you can’t communicate any longer. In the same way, if your website can’t get the user to the content they’re looking for, then it can’t deliver the message. It’s a failure of the medium to deliver the message.  The noise in your website has to be low enough so that people can find the signal they are searching for.

This signal, the message, is what the message says. It’s the literal content of your website. This is why developing a strong critical path is so crucial. The only way to be heard above the din of distraction is to have a focused message and to provide an obvious way to access that content.

What we’re talking about here is signal quality. And all that’s necessary to test a signal is somebody on the other end to describe what they’re getting.

If we were testing long-distance phone service we’d whip on some glasses, stomp through the woods and ask “Can you hear me now?“. But since we’re testing websites, all that’s required is to ask somebody else what they see on your website.

User Testing in 3 Questions

When you’re looking for quick insights about your website, you’re looking to know something about signal strength. It’s evident in the type of questions that are asked.

Take the basic questions from usertesting.com’s test:

1. What frustrated you most about this site?
2. If you had a magic wand, how would you improve this site?
3. What did you like about the site?

None of them concern the content on the website. All of them are testing either signal or noise.

Break down the expected answers:

1. What frustrated you the most? 99% of the time this is going to be an answer that relates to not being able to find something. It could also have something to do with your messaging.
2. How would you improve the site? Because you directed them to the site and not what your site sells or provides, this answer is going to relate somehow either to accessing information or clarifying options.
3. What did you like about the site? This will help with positive feedback and will help you confirm areas of the website that perform well or stand out to the user.

Nothing in there explicitly dealt with the quality of the material. What’s important is strengthening the signal.

I should state that these are not the only questions you could use in a 3-question test. You can test the critical path, a secondary path, or test for comprehension and cognitive load with your site structure without knowing anything about the user.

And we’re going to prove it. Next week, with a little luck, we’ll bring you a video of our own 3 question drive-by user test.