Archive

Posts Tagged ‘user testing’

Better User Experience Podcast #20: A Test-Case of Drive-By User Testing

December 26, 2011 2 comments

It’s the day after Christmas and boy am I fat and happy. Great time and good food with the family.

Last week, before all the holiday cheer broke out, Newman and I did our first drive-by user test.

You might have seen it.


On today’s podcast we have a little before and after action going on. This podcast was actually shot before and after the video above. We captured our thoughts on what we thought would happen and then take a look back at what happened and what we learned from the experience. It’s a good one!

Listen Now

 

Better User Experience Podcast #19 – The Top 10 Key UX Concepts We’ve Learned in the Past 4 Months

December 19, 2011 Leave a comment

We’ve been doing this learning-about-UX thing for a little over 4 months now. On today’s podcast we take a look back at the Top 10 things we learned in that time. If you’ve been following the site over the past week you know that we’ve already covered this material in two posts. But like any good discussion, we dig a bit further into each topic on the podcast.

Top 10 Key UX Concepts We’ve Learned (So Far)

10. Proactive vs. Reactive
9. UX Testing Resistance
8. The Power of Process
7. The Difference Between User Experience and Usability
6. The Critical Path
5. CARP
4. Prove vs. Improve
3. Signal vs. Noise / Information Theory / Entropy
2. Quick Tests Can Be Valuable
1. There’s a Great UX Community

Remember you can subscribe to our podcast on iTunes – new episodes each week!

Follow us on Twitter: @BUXofficial

Friend us on Facebook: facebook.com/BetterUserExperience

View us on Youtube: youtube.com/user/BetterUserExperience

Hierarchy of Needs – the Deprecated Pyramid

December 9, 2011 Leave a comment

Essential Question: Can User Experience and Usability needs fit into a  Maslow’s hierarchy of needs type pyramid?

All these pyramids are deprecated.  I’ve spent the last few hours slowly fading into madness and … here is how it began…

Ben and I got to discussion a post on UXMovement.com titled “Are You Meeting the User Experience Hierarchy of Needs?” by Anthony.  It was interesting and had a bunch of comments.  I decided to research it more and share it with you. here:

First Maslow’s

Pretty much everything from the 40s sucked... name one, really

Everyone ‘sort of’ understands Maslow’s pyramid. They may not understand it fully.  It may be like Darwin’s theory of evolution – everyone can give you the basic tenants, but it is probably misunderstood. Here’s my understanding of Maslow’s hierarchy of needs. Food and water and shelter are basic needs that take precedence over things like morality and respect of others.  You must satisfy the basic, base levels of the pyramid BEFORE you can / want to satisfy the higher levels.  Maslow’s hierarchy of needs is often cited when explaining why a typically law-abiding person will break the law (higher order) in order to get food or sex (lower order).  It’s a theory I have problems with.

Here are my questions: In order to be a whole person do you need to satisfy all needs?  Is it harder to meet higher,  mid, or lower needs?  Does the pyramid establish a sequential order for achieving needs?  And, to transition to interface design, must a “quality” website satisfy the all needs?  Is it harder to meet the higher needs than the lower needs? Must the design process start with satisfying lower basic needs and move on to higher needs?

UXMovement.com article by Anthony Tseng

Tseng's Hierarchy

This article and chart puts functionality as the base of the pyramid.  Together with information, this forms the first half of basic needs.  The top half are higher needs with aesthetics below and usability on top.

Usability breaks out of the pyramid top suggesting that it is not inside the pyramid – or this could have been done to promote readability as the font would be very small if made to fit in the tip of the pyramid.

He outlines a model that says you must begin interface design with functionality and move on to content and aesthetics in order to achieve usability. Anthony defines usability as “the ease-of-use of an interface that increases user productivity.” There was some debate on this article and several commenters wanted to put aesthetics at the top of the pyramid – which makes me scratch my head.

Flickr hypercatalecta UX hierarchy

Pretty Bullshit or Insightful visualization?

Even though Anthony did not like this pyramid Chart, I like it quite a bit.  Anthony called it pretty bullshit (which I thought was slightly rude). Just like a cow patty, this chart can teach a lesson. Working from the outside in, the chart establishes two contexts. At the top we have subjective/qualitative and ‘focused on experiences( people, activities, context)’ with an arrow pointing downward.  The second context is at the bottom it states: objective/quantifiable ( products, features) and ‘focused on tasks’ with an arrow pointing upward.

Within the context of tasks (going up), the graphic seems to relate that the priority sequence is: functional( useful), reliable, … and peaking at meaningful.

Within the context of experience (going down), the graphic seems to relate that the priority of sequence is reversed – that meaningful is the first priority and functional useful is the last priority.

This chart tries to make the distinction between subjective experiences and objective tasks.  That’s what I like about it. It seems state that if a subjective experience is meaningful and pleasurable that it does not have to be reliable or functional. Or that a subjective experience that is meaningful and pleasurable need not be reliable or functional. And this DOES seem to be descriptive of my personal experience.

If you have a meaningful and pleasurable relationship with a beautiful girlfriend, you don’t really care if it’s reliable or functional [Radio Edit]. But I don’t think he’s talking about relationships. If you just love your iPhone , you can subjectively overlook usability [Siri, why can’t I just drag and drop my files from my computer to my iPhone.  Siri?] – Think supercars in the 80s – Toyota’s were WAY more usable. If you were focused on tasks, then you just care if it’s useful and reliable (Toyota)-you don’t really care that it’s not meaningful or pleasurable (Ferrari).

The fact that convenient lies in the center of hypercatalecta‘s diagram  is important. That says that convenience is of equal value for objective users and subjective users. Both those focused on experiences and those focused on tasks care about convenience.

But here’s my problem: why is it a pyramid?. I have a feeling they just did that to make it look pretty. Perhaps this is why Anthony called it pretty bullshit (He called another guy pretentious )  But, because it defines the subjective / objective vectors, it gives me valuable insights.  When working with some clients – subjectively oriented clients – they really only care about aesthetics… but here is where I think folks are missing the point – the reason they feel that way, is because that’s their experience.  That’s how they experience the site.  It’s their user experience.

This is why I concur with Paul Veugen –  and others –   that user experience is a super-system of usability.  And, user experience is a complex interaction of many facets.  Why?  That little word USER.  People are complex – they are different shapes and sizes, speak different languages, and use websites in different ways and for different reasons.  Simply put – WE are the definition of complexity on this planet …currently

Alternatively, usability is technical.  Usability can be measured… and can be measured by any asshole.  Can you hear me now?

Voted most popular (non-contexual) user tester!

Moving forward –

Here’s a slideshare from Lyndon Cerejo from back in 2001.  This is actually a very good article and I need to spend more time with it… but I’m getting pretty sidetracked here.

lyndon Cerejo graphic

Lyndon Cerejo's insightful pyramid from a decade ago

 

Ugh… let’s look to Smashing Magazine, surely they are shed some light here.

Smashing Mag pyramid

Smashing Mag, per usual, has the issues covered

This article,’Designing For A Hierarchy Of Needs‘ starts off with:

Based on Maslow’s hierarchy of needs, the idea of a design hierarchy of needs rests on the assumption that in order to be successful, a design must meet basic needs before it can satisfy higher-level needs. Before a design can “Wow” us, it must work as intended. It must meet some minimal need or nothing else will really matter.

Is this true? Or could a design that’s hard to use still succeed because it makes users more proficient or meets certain creative needs? Do you have to get all of the low-level needs exactly right before considering higher-level needs? To answer these questions, let’s start by looking at Maslow’s hierarchy.

Notice the author () is talking about general design. Ugh!  Sure enough, we have some semantic issues here.  Damn you language and our plastic language brains.

I feel much of the confusion and debate is sparked by imperfect language and vocabulary.  Design, user experience, and usability: These terms are sometimes used inter-changeably.  Yet, they can be very different.

Can you use these pyramids to build sites and interfaces?  Sure.  But, here’s my insight, you CAN test this way – Regardless of if testing Design, User Experience, or usability.  First test for the base needs and then test for the higher needs.  Why do this?  Well, why not.  Think of it like a map and a way to segment the complex task into manageable chunks.  And, it can help prioritize your testing and help you make decisions and communicate problems to the stakeholders.

For your viewing Pleasure, I’ve collected all the Pyramids on my own private Giza.

image of several UX pyramids

Let's get it all on the table

Final Thoughts

Let’s face it – People love charts and pretty bullshit.  Pyramids are old and busted – honeycomb is the new hotness. Users are complex.

So!  There’s my insight. There’s my learning to share.  I hope you enjoyed it. Now, I’m off to share a beer!

User-Testing in 3 Questions: Why Drive-By User Testing Works

December 7, 2011 1 comment

[Note: In Monday’s podcast I said we were going to be doing a real user test today. We’ve decided to push that back to next week. We had two more items we wanted to get to first.]

When we first started learning about user testing back in August, I was under the presumption that user testing was essentially a controlled, systematic endeavor. But as Steve Krug and Jakob Neilsen will tell you, there’s more than one way to test a cat. (That’s a direct quote, I believe.) A lot can be gained just by getting feedback from somebody. They argue that the name of the game is improving your website (not testing a hypothesis) and that most major issues on a website are visible to most users.

Krug advocates companies set aside one morning a month to do three user tests. Then, debrief over lunch.

This rapid-testing idea was inspiring. It’s like everybody knows what’s not working on your site, and to learn the secret to what’s broken, all you have to do is ask them to tell you about it. Whoa!

You said it.

But is that true?

Are we all walking around with an innate sense of what sucks on websites? And if so, why can’t we design a perfect one out of the gate? If we all have the magical power of user testing, why can’t we user-test our own site and save the time, cost, and hassle of asking other people?

Proximity

The reason you can’t user test your own website is because you’re too close to it. You’re like a director who can’t enjoy his own movie in the same way as a regular viewer because you remember what happened off screen. When you look at your own website you remember the versions of the site that almost were and you know intimately the reasons and compromises that went into creating your current website. You can’t see the forest for the trees.

Your average user is focused on a completely different experience. They are focused on what game theorists call “utility-value maximization”. In plain English, it means that people are goal oriented on websites. It’s built into the way the Internet works. We click on links because we want to access new content. Each click is a statement of purpose. When a user visits your website, they are there for a reason and they have a goal in mind. Han Solo had it right about web design when he said, “Give the wookie what he wants.”

Mental Models

Have you ever helped a relative get a computer or get on the Internet or otherwise interface with a completely new piece of technology for the first time? Do you remember the first time you opened Photoshop or Illustrator or Dreamweaver or the first time you tried to center something with CSS and you had that feeling that you were in completely new territory?

That happens when you have no way to contextualize what you’re seeing with your previous experience.

People create mental models of all their behaviors. Have you ever caught yourself having a dumb conversation with somebody about the weather when (a) you don’t really know why you’re talking to them in the first place and (b) there’s nothing special about the weather? Why do we do that?

Or how about in movies and TV shows when people hang up the phone without saying goodbye first. That’s always weird to me. It’s because my mental model includes goodbye language at the end of phone conversations and in TV shows, it doesn’t advance the plot so it isn’t necessary.

Once you get your non-techie relative online, they too will begin to create a mental model of how the Internet works. Some of it will be based on fact. A lot of it will be based on feelings and intuition that is not correct. (I’m reminded of an old boss who, every time he saw the Blue Scree of Death on his PC would scream “BILL GATES!”) But people don’t get retrained when they learn the Internet wrong. Nope, we just deal and hope they catch up.

Jakob Nielsen points out many of the points of technical confusion in his blog post on the same topic.

  • Operating-system windows vs. browser windows
  • A window vs. an application,
  • Icons vs. applications,
  • Browser commands vs. native commands in a Web-based app
  • Local vs. remote info
  • Different passwords and log-in options (users often log in to other websites as if they were logging in to their email)

In short, when it comes to computers, a lot of us are still getting our act together. What happens as people gain experience using the Internet is they gain an intuition for how websites are supposed to work. And that’s why there’s a great list of usability conventions to use when developing your website.

What does this mean for web sites and user testing?

Signal vs. Noise

All communications come down to the issue of signal vs. noise. Those of us old enough to remember when Sprint was a long distance company who had sound quality so clear you could hear a pin drop have experience with the problem. In any communication medium, whether it’s spoken and heard, or transmitted by radio, TV, print, telegram, fax machine, or Internet, the message must be transmitted and it must be received. The message is encoded in a language and in a medium, and in a context. In order for it to be decoded correctly, the person receiving the message must understand the language, have access to the medium, and know the context in which the message is meant.

See? Easy.

Think of a physical long-distance phone line. The reason Sprint was so happy about their clear signal is because it’s hard to eliminate all the noise. Static, buzzing, clicks, pops – all could (and did) effect the ability to be heard.

When you think about a telephone line, the communication can be broken down into two parts:

  1. The mechanical workings of transmitting a person’s voice from one place to another and,
  2. The content of the communication

If you were to user-test the phone line, you could test for the same two attributes above:

  1. The quality of message transmission and,
  2. The content of the message

Unless you own a tin-foil hat, phone companies today don’t care about the content of your phone calls, texts, and messages. They are focused on delivering on the quality of message transmission. Also, because we all have such a complete mental model on how to make a phone call, it turns out that isn’t the part of the phone business that’s growing now. The user experience for making and receiving phone calls is essentially complete. The new wrinkle is welding the digital side of things to the phone. Now it’s all about experiencing the mobile Internet. And that, invariably, leads to more half-baked mental models for using the Internet.

The Medium vs. the Message

Like the double rainbow dude, we’re left to wonder, “What does it all mean!?”

Let’s cut to the chase.

When you add up the fact that there are so many ways to leave a page – from going to another site, clicking the back button, closing the window, getting up and walking away, clicking into a different window, getting distracted, plus the various mental models that people have for how the Internet should work, it’s safe to say that there’s a lot of noise in the channel.

In the physical long-distance phone line we discussed earlier, the medium was the phone line. In your website your critical path is the medium.

Now let’s think about that for a second. Remember how earlier we said that clicking a link is a statement of intent. It could be that the user is expecting to end up on a YouTube video or is expecting to see an Amazon page for digital cameras or is looking for an article on Wikipedia or a house on an MLS. The content is the message. The Internet is the medium. And getting to your content is part of the medium.

We’ve all dropped a call on our mobile phone. There’s a distinct difference between a phone call where you’ve said everything you wanted to and hung up and when the call drops. When the call drops – you can’t communicate any longer. In the same way, if your website can’t get the user to the content they’re looking for, then it can’t deliver the message. It’s a failure of the medium to deliver the message.  The noise in your website has to be low enough so that people can find the signal they are searching for.

This signal, the message, is what the message says. It’s the literal content of your website. This is why developing a strong critical path is so crucial. The only way to be heard above the din of distraction is to have a focused message and to provide an obvious way to access that content.

What we’re talking about here is signal quality. And all that’s necessary to test a signal is somebody on the other end to describe what they’re getting.

If we were testing long-distance phone service we’d whip on some glasses, stomp through the woods and ask “Can you hear me now?“. But since we’re testing websites, all that’s required is to ask somebody else what they see on your website.

User Testing in 3 Questions

When you’re looking for quick insights about your website, you’re looking to know something about signal strength. It’s evident in the type of questions that are asked.

Take the basic questions from usertesting.com’s test:

  1. What frustrated you most about this site?
  2. If you had a magic wand, how would you improve this site?
  3. What did you like about the site?

None of them concern the content on the website. All of them are testing either signal or noise.

Break down the expected answers:

  1. What frustrated you the most? 99% of the time this is going to be an answer that relates to not being able to find something. It could also have something to do with your messaging.
  2. How would you improve the site? Because you directed them to the site and not what your site sells or provides, this answer is going to relate somehow either to accessing information or clarifying options.
  3. What did you like about the site? This will help with positive feedback and will help you confirm areas of the website that perform well or stand out to the user.

Nothing in there explicitly dealt with the quality of the material. What’s important is strengthening the signal.

I should state that these are not the only questions you could use in a 3-question test. You can test the critical path, a secondary path, or test for comprehension and cognitive load with your site structure without knowing anything about the user.

And we’re going to prove it. Next week, with a little luck, we’ll bring you a video of our own 3 question drive-by user test.

Quick and Dirty User Testing – Now on Youtube

December 2, 2011 Leave a comment

Steve Krug’s “Rocket Surgery made Easy” makes you want to run out and user test some websites.  And, that’s what we did.

Here are two videos of Ben and Me testing some random websites.  We did it to test our screen recording software. And, we did it to get the feel for facilitating a test. And, we did it to test out our New YOUTUBE Channel!

Of course, we had some insights to share – but it’s too dang late for typing this out.  We’ll talk about it on Monday’s Better User Experience Podcast!

For your viewing pleasure!

and the Friday night Double Feature!

Categories: articles Tags: , ,

It’s the questions, stupid: less biased and more accurate tests

October 7, 2011 4 comments

True or false: User testing is about getting into the head of the user?

gorgeous bald-headed Natalie
Pictured: the head of a user.

Before I began this post I would told you that absolutely, user testing is all about getting into the head of the user. Why else would we ask them questions and go to such great expense to make those questions accurate, valid and reliable test instruments if not to get in their head?

Liar, liar pants on fire…

But we know that people often don’t mean what they say. Studies have shown that in a typical 10-minute conversation, the average person will tell 2.92 falsehoods. In essence, we filter what we say based on our context and situation. Much of what we say is coded for transmission based on the receiver. That’s a nice way of saying: I’m going to tell you what you want hear. It’s just a little white lie. Don’t worry about it, it’s not gonna hurt anybody.

It’s easier to do with speech than with actions. Our physical bodies have their own way of communicating. We call it body language.

Body Language

It’s been said that “hips don’t lie”.

She’s got a point.

But your body is constantly broadcasting what you’re feeling. Micro gestures in the face can tip off a liar or whose guilty. It’s how magicians guess names. Witness in the below video how the “magician” guesses the name of the ex-boyfriend by asking the woman a rapid series of questions by which he can observe her facial micro gestures.

Did you know that your feet are the most honest part of your body? Almost nobody pays attention to what they are doing with their feet. You can observe openness or defensiveness by whether the feet are in front or are behind and crossed or behind and wrapped around the chair legs.  (You checked your feet, didn’t you?)

Check out the nervous guy at the end of the bar… What is he doing with his hands? Posture?

I’m pretty sure he’s just happy to see me.

What does this mean for the evaluator or user test creator? Does it mean we have to check to see if they’re packing heat or if they’re Shakira? Are these examples helping?

What it really means is that it’s very difficult to write (verbalize) a test question and not tip off your bias.

It seems to me that most of the standard test questions are leading the users. For example, “How do you feel about this test?” or “Tell me about your Mother.“.

What if the user doesn’t feel anything about that? Once you force someone to generate a feeling, I figure they fall back onto their default response. For some it’s a negative response. Some, it’s a positive response.

Tester default response (call it tester bias? Tester prejudice?) must be accounted for. Personally, I believe in the “innocent until proven guilty” adage. The website I’m viewing is presumed innocent of design mistakes, until it breaks a design rule. Websites are presumed innocent. Finding out this tester default response is key to understanding user testers.

But it that getting into their head?

All of this is academic and probably doesn’t really effect in a measurable way the outcome of tests-especially at this basic level. However I feel it is good to visit and share these thoughts as we encounter them. The pitfall is thinking, “I will not past because I can’t be scientific in my measurement.”

Here are some ways I plan to alter the script based on my understandings stated above:

Tasks first, impressions later

For the practical user testing scenario, I suggest tasks first and impressions later. Allow the user to work with the site in form and impressions in a natural way. I understand that the research shows that you only have 5 seconds for someone to determine whether or not they want to stay on that page versus clicking the back button. But, in the testing environment it doesn’t really translate. Perhaps a good compromise is that ask them to make a personal note, not to be shared, about their first impression. And then, after the tasks are complete, ask them to revisit that first impression and see how it changed. This might be more beneficial and give a truer sense of the users impression.

What this protects against is my initial reaction to prejudice. By asking me my impressions after only 5 seconds you are really just determining my prejudice. All sites are innocent until proven guilty of wasting my attention or stealing my cognitive load.

Go ahead it’s okay to say it stinks

There has got to be a way to set the environment-and I’m talking about actively setting up the testing environment- to be open and honest. That doesn’t mean setting nice music playing and making sure you talk in a soothing voice-that crap would set me on edge. In such a clinical environment, I would feel less inclined to give my actual opinion. To be frank I don’t know exactly how to do this. Anonymous testers might feel more inclined for negative feedback. Perhaps the sandwich method of positive-negative-positive could be used. Are remote tests better than face to face?  Strangers vs Friends?

I can’t do it with you watching me

Big EyesSeriously. Stop that.

Being observed makes me nervous. It’s unfortunate because user testing is all about making observations. The point I want to make is in order to make the environment open and honest for feedback-both positive and negative-it could be important to make the recordings and note taking inconspicuous. People understand they are being recorded, but will quickly return to a more natural expression when the recording isn’t conspicuous.

Conclusion

The whole point of user testing is to observe users interacting with your design with the intent to improve that interaction. There will always be some degree of bias in both the evaluator and the tester. It’s not required to eliminate it in order to find the majority of  problems. However, if you think about and make a few changes to the standard scripts for user testing you can get a less biased and more accurate test of actual user experience.

Questions:

Do you have to get into user’s heads for a good user test?

How do you set the tone for a user test in order to elicit honest and open feedback?

Do you account for the natural, tester baseline / default response?

Do you, in a systematic way, observe and measure body language from the testers?

Here’s How User Experience Testing Can Be Better

September 30, 2011 Leave a comment

Last week we drafted a usability test and tested one user in order to get a real experience with the theories and abstractions we were researching and discussing. Our results were surprising.

What We Wanted

We wanted to figure out a repeatable process for conducting a user test that would improve upon the simple watercooler test – The type of test that my friend at textWoo.com conducted. Additionally we wanted to “user test the user test” by way of quickly getting feedback on an early draft of our test scheme in the hopes of creating a more effective testing tool. And, we wanted to dive in.

What We Did

I downloaded and edited a script from the usability.gov site which came from the SUS framework and a usability test from 3w.org. The way I customized it was to look at each page type in the JavaJack’s site and create a task that would engage the user with that page.  I didn’t make the tasks ‘hard’ but I didn’t want them to be too specific.  I didn’t want to test the users ability to read, for instance.  Our friend, Anna, came by just in time to be the tester.

What We Got

The results of the very unscientific test were intriguing and beneficial to the overall design of the site.  As Anna talked thru using the site with Ben and I there watching, we noticed some things that could be improved and changed. Time well spent.

After the test, our thoughts turned to the testing process itself.  Ben’s post explains his thoughts on the critical path and the purpose, goals and objectives of the sites and how they relate to user testing. Here are other thoughts about the user-test and the testing process in general.

Design Dunce? Maybe. Personal Experience Expert? yep.

Design Dunce? Maybe. Personal Experience Expert? yep.

User’s aren’t designers, don’t ask them to critic a site design

Most scripts I’ve seen ask the user this question in the beginning of the test:

Please give me your initial impressions about the layout of this page and what you think of the colors, graphics, photos, etc.

When you asked this question, it seems that the users become unsettled and defensive.  They have to form an opinion and defend it. They stop using the site.  They critique the site instead. They start to look at the site as a designer.  The user is tainted after that simple little question.  Ask the user about their own experience… and why not ask them AFTER they have the experience.

To fix this we came up with a few ideas. Let them complete the tasks and then ask them about their experience.    Record the session (the screen, webcam and audio) without you in the room. Just don’t ask the question, there are better ways to get initial reactions from users – the user-testing sites in the sidebar.  Or, conduct separate tests for reaction and impression.

Fly on the wall

Fly on the leaf?

Hiring UX facilitators: Flys apply within. Humans need not apply

You want to be a ‘fly on the wall’  as much as possible.  Humans suck at this.  Evaluator bias is rampant and I’m doubtful you can eliminate it. Simply put it’s when the testing-user feels that they should answer a certain way or feel the agenda of the test questions.  Who doesn’t feel that in every survey!  Human societal norms get in the way here (Perhaps not in New York City, granted).  People tend to be polite to strangers and people in authority. I figure negative answers questions are rare.  Users will try to figure out what you want and try to give you that.   The act of testing will influence their use. I feel the designer is the least desirable person to be the facilitator of the test. If the user feels that the facilitator has an agenda or prefers one outcome over the other, then the test is compromised.

You can’t hire actual flys to do your testing (they don’t make lab coats and clipboards small enough).  Here are a few ideas we had to correct the problem on Evaluator Bias and human factors.  You could use a remote testing service – like the ‘Mouse Tracking Tools’ in the sidebar. Run face to face tests in a familiar place to the user – office, coffee shop, mall, home – so the user is more comfortable giving honest opinions. The facilitator should not be perceived as someone affiliated with the site – regardless if they are or not – nor an authority of any kind.  Perhaps, you can test several other sites to hide the site you are actually testing (This might be time/resource intensive)

Something is better than nothing

It works better than nothing.

Something is better than nothing.

However, doing any type of test is better than nothing. The simple act of watching somebody go through the site is very desirable. Testing gives you insight into the flow of the site, if the site is mechanically (or functionally) sound and working and if the user finds and stays on the critical path or primary site goal.

Alien canals on the planet Euphorbia Pulcherrima

Only natural

You sellin’ what I’m buying? Great, let’s do this.

Each user has their own goal when coming to the site. Each site owner has a goal when building a site.  If the two goals match, Great! Now get out of the way and let the site churn out money.  The user clicks thru the site and their goals is met.  This click trail thru the site is called the Critical Path.  And, it’s what you should test in the ‘Tasks’ portion of the standard usability test.

There can be many paths, but only one critical path on a site.  For example, I go to Apple.com to watch movie trailers. Would Apple user-test my experience in getting to and watching movies? Perhaps, but I bet they measure how easy it is for me to ‘jump over’ and buy some music or a new computer. The purpose – the critical path –  of the site is to sell (I’ll give you branding and customer service, as additional paths) and all other features and functions of the site support that goal.

How well the site moves visitors along the path is the effectiveness of the site.  And, we test for it by asking testers to assume they have the same goal as the site.  Likewise, we test for satisfaction. Did they complete the task, but were pissed because of something else? Did they expect one thing and get an unpleasant surprise?  Also, we test for efficiency. Did they complete the task, but it took 15 clicks and 20 minutes to complete?

You can test this by doing a very simple user tests. That’s the low hanging fruit of user tests.  Site effectiveness, Satisfaction, and Efficiency.

In conclusion,  don’t ask about impressions before your evaluation of the critical path. Do test even if the conditions are not scientific. Let users use. Visitors visit.  Don’t force them to have an opinion and then needle them about it. This is a carry over from the designers perspective.  User’s aren’t lab rats. Your color palette isn’t of supreme importance.  Listen close and you can hear a user. They are probably saying, “Just give me the dang banana already”

Questions:

Is the user qualified to speak to design?
How do you get around the evaluator bias?
Can one site have multiple critical paths?