Posts Tagged ‘user-test’

5 Things We Learned From Our First User Test

December 23, 2011 Leave a comment

Do you remember what a mystery sex used to be? (Don’t worry Mom, it still is, I swear. ) You’d talk about it endlessly with your friends just telling and halfway believing each others lies. And because you didn’t know what you were talking about, you boned up on your reading. [Boned up?  good grief – ed] All of the sudden, Cosmo – either your sister’s, your Mom’s with their “100 ways to spice up your sex life” was interesting reading.

You mean they just make lists of this stuff and print it for you to read?

At some point, the reading up becomes boring. The boundaries are well known and what used to be exciting is now just another list of stuff that other people are doing right.

That’s when you know it’s time to get down to business.


That’s basically where Ben and I have been with this user testing stuff for the past few weeks. You can hear it in the podcasts. Enough with the reading! Let’s make with the fu…n user tests!

And so we did.

On Wednesday, we showed you the video of how it all went down. If you missed that post, we’ll save you the trouble. Here ya go.

Could you tell how vulnerable I was in that clip? You just wait Kate, let’s do it again in a few years. I’ll be way better.

Coming out of that experience, we learned a lot. Here’s a list of the Top 5 takeaways.

5. Be prepared

I suppose that we were preparing the whole 4 months leading up to this first user test.  We had practiced on ourselves and friends, but this was the first ‘real’ test with a stranger in a strange place.   The rehearsals were key in getting ready the mechanics and hardware.  But there is no way to prepare completely. There’s no way around sucking the first time and being un-smooth.

Smoothness didn’t seem to matter, luckily.  Perhaps authenticity trumps smoothness.  If you’re genuine and honest, people may overlook a herky-jerky delivery.

4. Getting permission is the hardest part

Getting the permission to record and use the coffee shop was tough.  We had to explain who we are and what we are doing.  If you’re not used to doing that, it can seem really awkward. But you have to do it.

It’s true that you really only need to get permission one per place and that it’s not really necessary to do a user test. I mean, you can do one at home or in your office just the same. But doing it this way, it was much more of a challenge than actually conducting the test.

This is an example of activation energy. The hard part is starting and now it’s much easier to keep it going.

3. It Gets a Reaction

While we were at the coffee shop we ran into four folks who work at a hotel on the island. It was a manager and her sales team. The minute they found out that we were doing website user testing, their body language became more guarded and the manager slowly made her way towards the door. We had nothing to sell and weren’t even interested in getting them to sit down for a test but just the words were enough to cause them discomfort!

Equally interesting, after their initial reaction, they warmed up to it.  They talked about how they user tested the items that go into each hotel room. The manager told a story about having to get new alarm clocks for the rooms and that she wanted to test it before they used them. In her words, “if I can’t work it in two minutes then it’s not going in the hotel rooms”.  She found a way to apply user testing to her own field.

We’ve speculated that the strong reaction is because small business owners have their personal identity wrapped up in their business. It’s hard to willingly submit to judgment.  But, once they realize that it’s a way to improve their bottom line and that people are ALREADY judging them, they tend to get on board.

2. Good Things Come Available in Bite Size

We wanted to see if it was true that you could get value out of asking a stranger what they think of a website. Is it valuable or is it just a stunt? It turns out that we got some good data that we can give to the website owner. The moral of the story is to get out there and ask people their opinion about your website. You will learn something. Simple as that.

1. People Are Cool

Meeting people is always a highlight.  Humans are social animals.  We need to communicate and interact.  That’s a big part of what this blog is about.  The folks we met at the Daily Grind helped us dive deeper into what User Testing is all about.  We are thankful for that.  It didn’t have to go that way – we could have been chased out of there. Instead, it went really well.

It’s important to say that too. If you read most media, it’s mostly one group of people complaining about another group of people. You’d think that we are all angry people but that’s just not the case. People are cool.

Bonus: What we learned about the website in the user test

  1. Better and more headlines – People skim websites. It’s an issue of cognitive load. Successful user testing, as a rule, reduces the cognitive load a website imposes on its viewers. Better headlines would help the user quickly scan the page and drill down to the relevant content and would reduce cognitive load.   At 3:10 our user-tester, Kate, responded to a question about services with, “I probably gotta read about it first, right?”.  She was under a higher cognitive load because of the camera and the test, but that helped to reveal that one couldn’t skim the headlines for meaning. They weren’t there.
  2. Include pricing info (5:50) – As a professional we all get used to not seeing prices on a website that sell a service. “If they want to know, they’ll call” is a common thought. Be that as it may be, it’s also true that most people would prefer to at least know the general pricing structure. If you choose to not have prices on the site, you should at least address it.  “Prices range from”, “Free Consultation” or, as our user suggested “10% off” are examples.  You could also explain how your prices are structured – by the hour, by site, do you have packages of services, etc.
  3. Improve some of the photos (1:20 and 4:10) – The compass image could be replaced.  Kate mentioned the color balance being ‘off’.  We feel it draws attention without leading to the critical path of the site.  It’s not imediately clear why the compass is there.  Suggest: Either create a new photography or create a headline that explains the compass.

Implementing these suggestions would significant improve the user experience of your site.

Enjoy the holidays and be on the lookout for day-after-Christmas podcast!

Categories: articles Tags: , ,

We Did a Drive-By User Test and Lived to Tell About It

December 21, 2011 Leave a comment

How do you follow up a week full of looking back? By pushing forwards!

This week we finally put down the books, stopped all the chitter-chatter and went out and did a user test.

I know. We were excited too.

If you’re anything like us, you read a lot about user testing online. Five reasons you should do this or 10 ways to improve that. It’s not that those are bad articles, it’s just that they’re not touching what happens in a real test.

Last Friday, Newman and I went down to the local coffee shop in Wrightsville Beach, NC. It’s called The Daily Grind. We walked in to an empty house, except for Kate, the lovely barista on staff. We chatted with her and her boss, Tony, for a few minutes to explain what we were up to and before you know it Kate had volunteered to take a user test.

Our goal was to do what we’re calling a “drive-by” user test. Basically, it means you can just roll up on somebody – ask them a question about a website – and see what you get back. It’s not meant to be scientific. Just more of a survey. It’s our belief that even this quick method will yield valuable insights.

We could tell you all about what we did but we shot a video of it instead.

Way better.

In the video you’ll see the entire user test (roughly 6 minutes) and Newman’s thoughts afterward.  All-in-all, I have to say, it was a pretty good first attempt.

Thanks again to Kate – your pictures rawk! – for being our user tester and for Tony and the gang allowing us to bother their shop. As you’ll hear in my voice on Monday’s podcast where we did a whole before and after thing, their coffee will put a motor in you. No doubt.

Join us on Friday when we talk about what we learned from the user test – both about user testing and about the website we tested – and then on Monday  – the day after Christmas – for your aural pleasure, we have a podcast lined up and ready to go about the whole event.

Categories: articles Tags: ,

User-Testing in 3 Questions: Why Drive-By User Testing Works

December 7, 2011 1 comment

[Note: In Monday’s podcast I said we were going to be doing a real user test today. We’ve decided to push that back to next week. We had two more items we wanted to get to first.]

When we first started learning about user testing back in August, I was under the presumption that user testing was essentially a controlled, systematic endeavor. But as Steve Krug and Jakob Neilsen will tell you, there’s more than one way to test a cat. (That’s a direct quote, I believe.) A lot can be gained just by getting feedback from somebody. They argue that the name of the game is improving your website (not testing a hypothesis) and that most major issues on a website are visible to most users.

Krug advocates companies set aside one morning a month to do three user tests. Then, debrief over lunch.

This rapid-testing idea was inspiring. It’s like everybody knows what’s not working on your site, and to learn the secret to what’s broken, all you have to do is ask them to tell you about it. Whoa!

You said it.

But is that true?

Are we all walking around with an innate sense of what sucks on websites? And if so, why can’t we design a perfect one out of the gate? If we all have the magical power of user testing, why can’t we user-test our own site and save the time, cost, and hassle of asking other people?


The reason you can’t user test your own website is because you’re too close to it. You’re like a director who can’t enjoy his own movie in the same way as a regular viewer because you remember what happened off screen. When you look at your own website you remember the versions of the site that almost were and you know intimately the reasons and compromises that went into creating your current website. You can’t see the forest for the trees.

Your average user is focused on a completely different experience. They are focused on what game theorists call “utility-value maximization”. In plain English, it means that people are goal oriented on websites. It’s built into the way the Internet works. We click on links because we want to access new content. Each click is a statement of purpose. When a user visits your website, they are there for a reason and they have a goal in mind. Han Solo had it right about web design when he said, “Give the wookie what he wants.”

Mental Models

Have you ever helped a relative get a computer or get on the Internet or otherwise interface with a completely new piece of technology for the first time? Do you remember the first time you opened Photoshop or Illustrator or Dreamweaver or the first time you tried to center something with CSS and you had that feeling that you were in completely new territory?

That happens when you have no way to contextualize what you’re seeing with your previous experience.

People create mental models of all their behaviors. Have you ever caught yourself having a dumb conversation with somebody about the weather when (a) you don’t really know why you’re talking to them in the first place and (b) there’s nothing special about the weather? Why do we do that?

Or how about in movies and TV shows when people hang up the phone without saying goodbye first. That’s always weird to me. It’s because my mental model includes goodbye language at the end of phone conversations and in TV shows, it doesn’t advance the plot so it isn’t necessary.

Once you get your non-techie relative online, they too will begin to create a mental model of how the Internet works. Some of it will be based on fact. A lot of it will be based on feelings and intuition that is not correct. (I’m reminded of an old boss who, every time he saw the Blue Scree of Death on his PC would scream “BILL GATES!”) But people don’t get retrained when they learn the Internet wrong. Nope, we just deal and hope they catch up.

Jakob Nielsen points out many of the points of technical confusion in his blog post on the same topic.

  • Operating-system windows vs. browser windows
  • A window vs. an application,
  • Icons vs. applications,
  • Browser commands vs. native commands in a Web-based app
  • Local vs. remote info
  • Different passwords and log-in options (users often log in to other websites as if they were logging in to their email)

In short, when it comes to computers, a lot of us are still getting our act together. What happens as people gain experience using the Internet is they gain an intuition for how websites are supposed to work. And that’s why there’s a great list of usability conventions to use when developing your website.

What does this mean for web sites and user testing?

Signal vs. Noise

All communications come down to the issue of signal vs. noise. Those of us old enough to remember when Sprint was a long distance company who had sound quality so clear you could hear a pin drop have experience with the problem. In any communication medium, whether it’s spoken and heard, or transmitted by radio, TV, print, telegram, fax machine, or Internet, the message must be transmitted and it must be received. The message is encoded in a language and in a medium, and in a context. In order for it to be decoded correctly, the person receiving the message must understand the language, have access to the medium, and know the context in which the message is meant.

See? Easy.

Think of a physical long-distance phone line. The reason Sprint was so happy about their clear signal is because it’s hard to eliminate all the noise. Static, buzzing, clicks, pops – all could (and did) effect the ability to be heard.

When you think about a telephone line, the communication can be broken down into two parts:

  1. The mechanical workings of transmitting a person’s voice from one place to another and,
  2. The content of the communication

If you were to user-test the phone line, you could test for the same two attributes above:

  1. The quality of message transmission and,
  2. The content of the message

Unless you own a tin-foil hat, phone companies today don’t care about the content of your phone calls, texts, and messages. They are focused on delivering on the quality of message transmission. Also, because we all have such a complete mental model on how to make a phone call, it turns out that isn’t the part of the phone business that’s growing now. The user experience for making and receiving phone calls is essentially complete. The new wrinkle is welding the digital side of things to the phone. Now it’s all about experiencing the mobile Internet. And that, invariably, leads to more half-baked mental models for using the Internet.

The Medium vs. the Message

Like the double rainbow dude, we’re left to wonder, “What does it all mean!?”

Let’s cut to the chase.

When you add up the fact that there are so many ways to leave a page – from going to another site, clicking the back button, closing the window, getting up and walking away, clicking into a different window, getting distracted, plus the various mental models that people have for how the Internet should work, it’s safe to say that there’s a lot of noise in the channel.

In the physical long-distance phone line we discussed earlier, the medium was the phone line. In your website your critical path is the medium.

Now let’s think about that for a second. Remember how earlier we said that clicking a link is a statement of intent. It could be that the user is expecting to end up on a YouTube video or is expecting to see an Amazon page for digital cameras or is looking for an article on Wikipedia or a house on an MLS. The content is the message. The Internet is the medium. And getting to your content is part of the medium.

We’ve all dropped a call on our mobile phone. There’s a distinct difference between a phone call where you’ve said everything you wanted to and hung up and when the call drops. When the call drops – you can’t communicate any longer. In the same way, if your website can’t get the user to the content they’re looking for, then it can’t deliver the message. It’s a failure of the medium to deliver the message.  The noise in your website has to be low enough so that people can find the signal they are searching for.

This signal, the message, is what the message says. It’s the literal content of your website. This is why developing a strong critical path is so crucial. The only way to be heard above the din of distraction is to have a focused message and to provide an obvious way to access that content.

What we’re talking about here is signal quality. And all that’s necessary to test a signal is somebody on the other end to describe what they’re getting.

If we were testing long-distance phone service we’d whip on some glasses, stomp through the woods and ask “Can you hear me now?“. But since we’re testing websites, all that’s required is to ask somebody else what they see on your website.

User Testing in 3 Questions

When you’re looking for quick insights about your website, you’re looking to know something about signal strength. It’s evident in the type of questions that are asked.

Take the basic questions from’s test:

  1. What frustrated you most about this site?
  2. If you had a magic wand, how would you improve this site?
  3. What did you like about the site?

None of them concern the content on the website. All of them are testing either signal or noise.

Break down the expected answers:

  1. What frustrated you the most? 99% of the time this is going to be an answer that relates to not being able to find something. It could also have something to do with your messaging.
  2. How would you improve the site? Because you directed them to the site and not what your site sells or provides, this answer is going to relate somehow either to accessing information or clarifying options.
  3. What did you like about the site? This will help with positive feedback and will help you confirm areas of the website that perform well or stand out to the user.

Nothing in there explicitly dealt with the quality of the material. What’s important is strengthening the signal.

I should state that these are not the only questions you could use in a 3-question test. You can test the critical path, a secondary path, or test for comprehension and cognitive load with your site structure without knowing anything about the user.

And we’re going to prove it. Next week, with a little luck, we’ll bring you a video of our own 3 question drive-by user test.

4 MORE Points of Wisdom from Steve Krug’s ‘Rocket Surgery Made Easy’

November 30, 2011 2 comments

About a month ago, Newman wrote a post titled 4 Points of Wisdom from Steve Krug’s ‘Rocket Surgery Made Easy’. He was reading the book and wanted to share his insights. At the time, I was immersed in the James Gleick book ‘The Information’. And if you’re a regular reader, you know that led us on a week long journey exploring how entropy is related to web design and user testing. Oh yes, it got serious.

Now it’s my turn to go through ‘Rocket Surgery Made Easy’ and I have 4 more points of wisdom that I learned from reading through the book.

#1. Prove vs. Improve

This was a bit of a revelation to me. When I think of user testing, I think of trying to make a website better. It never occurs to me that I’m trying to prove something. It just seems obvious that I would be trying to improve things. But when it comes to user testing, it’s possible to do both.

To put it in scientific terms, it’s quantitative testing vs. qualitative testing.

Quantitative testing involves creating a testing methodology adhering to a strict testing protocol to ensure non-biased results. If it sounds like a science test, with a hypothesis and all of that, you’d be right. And because there’s a hypothesis and you’re looking for valid data feedback, it is setting out to prove something.

Qualitative testing is much less formal. It’s not focused on proving anything. Instead, its focus is on making things better.

Certainly, there’s room for both types of testing but when it comes to actually doing the testing, for most businesses, it will be more time and cost effective to concentrate on improving things. And doing that is as easy as asking for an opinion.

#2. Why Down-and-Dirty Qualitative Testing Gets Results

Deep down, we all know that nothing is free. So what gives here? Why does it seem like the testing type that requires less rigor, time, effort, and money seem to be the one that actually works? Simply put: it’s because ‘good enough’ is good enough. And by the time you’ve exhausted insights from qualitative testing, you’ll be in a better position to do quantitative testing.

Krug lays out three reasons in particular why qualitative testing works:

1. All sites have problems

Settle down.

I’ve never come across a website that couldn’t use a little work. Apparently, neither has Steve Krug. One of the main reasons that water cooler user testing works is because there’s always room for improvement.

2. Most of the serious problems tend to be easy to find

In the dust of creating a website, it’s easy to get too close to the whole operation. A common issue comes from troubleshooting problems. These problems can be technical or organizational or even baked into the business plan. Eventually a solution is found and implemented. Sure, it managed to go all Matrix on all of the problems and managed to dodge all of the bullets but that doesn’t mean that the solution was the right one for the user.

The first hint that you're too close.

When you find a non-interested party and get feedback, the major issues will crop up again and again. You can’t see the forest for the trees. They can.

The forest from 'Return to Zork' still haunts my dreams.

3. Getting stakeholders involved in user testing gives them a reality check on who their users are

Another common mistake that’s made during the planning phase of web design is that it’s created for an ‘average user’.  The problem is, the average user tends to bear little resemblance to their actual average user. The obvious remedy is to go find some ‘average users’. Technologically, we can do just that.

Use a tool like Inspectlet (which Newman has been using and blogging about) or one of the other ‘Heatmaps / Mouse Tracking Tools’ we have in the sidebar to record your user’s sessions. Share those  videos with all the stakeholders and then stand back. Everybody will get a new insight about the ‘average user’ and will immediately want to talk about it. It’s pretty remarkable to watch, actually.

Who you think your users are.

Who your users actually are.

This has the wonderful effect of getting everybody to focus on the right thing: improving the user experience.

#3. Test other people’s websites

This is just brilliant. User testing can (and should!) happen even before the first napkin sketch is drawn. How? Test the websites of your competitors or of somebody else in the same field. Test sites that have features you’re thinking of implementing. A cup of coffee and a conversation could save you weeks of work.

Remember, this is not rocket surgery. It’s basically asking people their thoughts about a website. Nothing says that you only can ask people about your website. As Krug says, “someone has gone to the trouble of building a full-scale working prototype of a design approach to the same problems you’re trying to solve, and then they’ve left it lying around for you to use.”

#4. Test for ease of understanding

That was easy!

This bit seemed to echo the Smashing Magazine article ‘Easier is Better Than Better‘ we discussed on Monday’s podcast. The gist of the article is that:

“People choose not on the basis of what’s most important, but on what’s easiest to evaluate.”

Or more simply stated, “Don’t Make Me Think!“.

Websites do two things: provide information for a user to consume and provide a way to filter out all of the other information.

You may know this as our ‘filter’ and ‘confirming’ pages that we talk about again and again. Web pages, until you get to an information page, whether that’s a YouTube video, or a contact page, or a product description page, or MLS search results, are known as ‘filters’. This is because their main role is to get you to the information you’re there to view. The most obvious of these filter pages is the front page.

On most front pages of website, they exist to shuffle visitors off to other pages. They are not a destination page in-and-of themselves. The essence of filtering is clarity. And clarity can be measured by watching how people interact with a page. The easier it is to navigate the page, the lower cognitive load and the higher the success rate.

People want ‘easy’ more than they want ‘better’. If your site isn’t easy, it won’t have a chance to show that it’s ‘better’ because finding another website is just as easy.

This testing can be done even at the napkin-sketch phase. Just ask somebody who isn’t involved with the project to tell you what they see in the sketch. Listen to what they have to say. Chances are they’ll say “oh, this looks like a site for ____ and what’s this ‘experience’ button?” or something to that effect and you’ll immediately hone in on what doesn’t make sense to your users.

Final Thoughts

I can see why this book is the go-to resource for easy and effective user tests. It maintains a laser-like focus on how to improve your website with user testing. It’s filled with good nuggets. Certainly enough to do at least a third installment of this series. I really can’t recommend it highly enough. It’s the perfect compliment for the various user testing tools that we sample here on the site.

4 Points of Wisdom from Steve Krug’s ‘Rocket Surgery Made Easy’

October 26, 2011 6 comments

The book I’m reading: Steve Krug’s “Rocket Surgery Made Easy“. It’s a short 168 page instruction manual for performing user-tests on websites.  Following the success of his first book on web usability, “Don’t make me think”, Steve taught workshops for businesses and organizations to instruct them in how to implement the ideas in the book. “Rocket Surgery” is that workshop in book form.

First name basis with Steve

I like Steve enough to call him by his first name. His style is very personable and direct. He is obviously insightful and thoughtful, and yet communicates in a concrete style that makes the topic easy to understand. He does use clichés too much and speaks with quotes and axioms often – I cut him some slack because that’s what I do.

I think it’s easy to understand – where other, more pedantic manuals are not – is the personalification principle (I could be making this up b/c  I can’t find a reference) . He speaks directly to the reader, as if engaged in conversation. He is writing for a clearly define audience – those wanting to use the ideas in the first book or wanting to user-test websites. He speaks with the audience, not at the audience. You become engaged with the material and you learn more.

I like the book and can recommend it whole heartedly. Here are just a few takeaways that I’d like to share and promote among other web usability practitioners.

Test everything, all the time

Feng-GUI of the Rocket Surgery Made Easy book cover

Test the sketch on the napkin, test the wire-frame, test the page mockups, test the landing pages, EVENT test competitor’s websites before you even have a sketch.(great way to do market analysis) He illustrates the common pitfall of testing too late and the reluctance to test in the early stages of design. Any product of the design process can be tested and those tests can lead to small corrections that avoid big mistakes later in the process

He says it’s never too early to start testing a website – I agree. A small thing in the beginning of a project can become a big thing later. Small and early is easy to fix while big and late is hard to fix. Don’t delay, test today. Don’t worry about not being perfect or not being ready, because…

Perfect is the enemy of Good

Voltaire says "Perfect is the Enemy of Good" and I believe it.

There has to be some truth to this or there wouldn’t be so many dang quotes about it. Personally, I have three or four I use and switch between – “Put the ball in the fairway” and “Put the ball in play” and “Base hits, not homeruns”. … it keeps going, “you can’t catch fish, unless your line is in the water” and “A good plan in action is better than the best plan on the shelf.”

It means that we should resist the temptation towards perfectionism and that we should think about gradual improvement instead of “all or nothing”. There is a Japanese term for it that is popular in management circles – “Kaizen“. This idea fits with Steve’s practical, simple style – it works and that’s good enough.

KISS – Keep it simple, smarty-pants

Keeping it simple, I put this here for your enjoyment.

Simple and practical is serviceable and useful. Simple holds up under pressure and close inspection. Everything about the book and its procedures is to reduce the complexity in order to make user-testing more likely to occur. Simple tests, simple reports, and simple improvements are essential to building a user-testing culture in an organization. Why? Because, complexity doesn’t yield better results. This is especially true when compared to not doing any testing at all.

One of his axioms states “one morning a month, that’s all we ask”. He is making it clear that the investment is small and the return can be big. His ‘small, non-honkin’ report’ should be written in 30 minutes and read in 2.

Lo and Behold “Maxims”

Steve uses 6 maxims to summarize his advice to user-testers. I’ve listed them here with a few notes:

  1. A morning a month, that’s all we ask
  2. Start earlier than you think makes sense
  3. Recruit loosely and grade on a curve
  4. Make it a spectator sport
  5. Focus ruthlessly on a small number of the most important problems
  6. When fixing problems, always do the least you can do

And, here is my interpretation. (matched by number)

  1. We (user-testing group) need resources and attention (from the organization), but it’s not going to be much.
  2. Always be testing and test as much as you can afford.
  3. Finding the ‘right’, target market users isn’t as important as you may think – problems and improvements will still be found.
  4. The more stakeholders you can get to view the tests and be involved the better – seeing is believing.
  5. Create a priority list of observed problems and follow through on fixing them.
  6. Simple fixes are better than a total redesign. Subtraction is often better than addition.