Last week we drafted a usability test and tested one user in order to get a real experience with the theories and abstractions we were researching and discussing. Our results were surprising.
What We Wanted
We wanted to figure out a repeatable process for conducting a user test that would improve upon the simple watercooler test – The type of test that my friend at textWoo.com conducted. Additionally we wanted to “user test the user test” by way of quickly getting feedback on an early draft of our test scheme in the hopes of creating a more effective testing tool. And, we wanted to dive in.
What We Did
I downloaded and edited a script from the usability.gov site which came from the SUS framework and a usability test from 3w.org. The way I customized it was to look at each page type in the JavaJack’s site and create a task that would engage the user with that page. I didn’t make the tasks ‘hard’ but I didn’t want them to be too specific. I didn’t want to test the users ability to read, for instance. Our friend, Anna, came by just in time to be the tester.
What We Got
The results of the very unscientific test were intriguing and beneficial to the overall design of the site. As Anna talked thru using the site with Ben and I there watching, we noticed some things that could be improved and changed. Time well spent.
After the test, our thoughts turned to the testing process itself. Ben’s post explains his thoughts on the critical path and the purpose, goals and objectives of the sites and how they relate to user testing. Here are other thoughts about the user-test and the testing process in general.
User’s aren’t designers, don’t ask them to critic a site design
Most scripts I’ve seen ask the user this question in the beginning of the test:
Please give me your initial impressions about the layout of this page and what you think of the colors, graphics, photos, etc.
When you asked this question, it seems that the users become unsettled and defensive. They have to form an opinion and defend it. They stop using the site. They critique the site instead. They start to look at the site as a designer. The user is tainted after that simple little question. Ask the user about their own experience… and why not ask them AFTER they have the experience.
To fix this we came up with a few ideas. Let them complete the tasks and then ask them about their experience. Record the session (the screen, webcam and audio) without you in the room. Just don’t ask the question, there are better ways to get initial reactions from users – the user-testing sites in the sidebar. Or, conduct separate tests for reaction and impression.
Hiring UX facilitators: Flys apply within. Humans need not apply
You want to be a ‘fly on the wall’ as much as possible. Humans suck at this. Evaluator bias is rampant and I’m doubtful you can eliminate it. Simply put it’s when the testing-user feels that they should answer a certain way or feel the agenda of the test questions. Who doesn’t feel that in every survey! Human societal norms get in the way here (Perhaps not in New York City, granted). People tend to be polite to strangers and people in authority. I figure negative answers questions are rare. Users will try to figure out what you want and try to give you that. The act of testing will influence their use. I feel the designer is the least desirable person to be the facilitator of the test. If the user feels that the facilitator has an agenda or prefers one outcome over the other, then the test is compromised.
You can’t hire actual flys to do your testing (they don’t make lab coats and clipboards small enough). Here are a few ideas we had to correct the problem on Evaluator Bias and human factors. You could use a remote testing service – like the ‘Mouse Tracking Tools’ in the sidebar. Run face to face tests in a familiar place to the user – office, coffee shop, mall, home – so the user is more comfortable giving honest opinions. The facilitator should not be perceived as someone affiliated with the site – regardless if they are or not – nor an authority of any kind. Perhaps, you can test several other sites to hide the site you are actually testing (This might be time/resource intensive)
Something is better than nothing.
However, doing any type of test is better than nothing. The simple act of watching somebody go through the site is very desirable. Testing gives you insight into the flow of the site, if the site is mechanically (or functionally) sound and working and if the user finds and stays on the critical path or primary site goal.
You sellin’ what I’m buying? Great, let’s do this.
Each user has their own goal when coming to the site. Each site owner has a goal when building a site. If the two goals match, Great! Now get out of the way and let the site churn out money. The user clicks thru the site and their goals is met. This click trail thru the site is called the Critical Path. And, it’s what you should test in the ‘Tasks’ portion of the standard usability test.
There can be many paths, but only one critical path on a site. For example, I go to Apple.com to watch movie trailers. Would Apple user-test my experience in getting to and watching movies? Perhaps, but I bet they measure how easy it is for me to ‘jump over’ and buy some music or a new computer. The purpose – the critical path – of the site is to sell (I’ll give you branding and customer service, as additional paths) and all other features and functions of the site support that goal.
How well the site moves visitors along the path is the effectiveness of the site. And, we test for it by asking testers to assume they have the same goal as the site. Likewise, we test for satisfaction. Did they complete the task, but were pissed because of something else? Did they expect one thing and get an unpleasant surprise? Also, we test for efficiency. Did they complete the task, but it took 15 clicks and 20 minutes to complete?
You can test this by doing a very simple user tests. That’s the low hanging fruit of user tests. Site effectiveness, Satisfaction, and Efficiency.
In conclusion, don’t ask about impressions before your evaluation of the critical path. Do test even if the conditions are not scientific. Let users use. Visitors visit. Don’t force them to have an opinion and then needle them about it. This is a carry over from the designers perspective. User’s aren’t lab rats. Your color palette isn’t of supreme importance. Listen close and you can hear a user. They are probably saying, “Just give me the dang banana already”
Is the user qualified to speak to design?
How do you get around the evaluator bias?
Can one site have multiple critical paths?
If you’ve been following the website or podcasts over the past few weeks you’ve seen Newman and I dig into the topic of web usability. We approached it, at first, from a philosophical perspective. We talked about the web design process and where web usability fits into it. We found various tools online to assist in doing user tests and we’ve found various models for how to conduct user tests.
And we started to realize something: these models tend to be obnoxiously vague. Even when they make intuitive sense, they still leave the reader with more questions than answers.
I feel like they keep creating iterations of the old Sony business model:
Step 1: Bright idea
Step 2: ????
Step 3: Profit!
It’s Mr. Sony and he’s a naughty kitty!
WTF is Step 2? Nobody I’ve read (and I’ll readily admit that I have much more to read) has articulated a quality strategy for conducting a user test. And all the user test examples I’ve seen have bared this out.
What I’ve noticed is that somebody will develop a model. For example:
I actually like this model. I think it does a good job of describing the steps and grouping the various considerations into each step. But what drives me crazy about it is that it doesn’t compel me to do anything specific. This is a 25,000 foot view of the web ux universe. And as you’ll see, it’s practically the only view these models will present:
Same thing with this model, except in this model, they’re using words and phrases I’m familiar with but use them in ways that are not intuitive and meaningful.
You can look at those two models or even the one we were initially so high on in Monday’s podcast at usability.gov until blood shoots out of your nose and you’ll never get to a better understanding of how to conduct a user test.
What we need to do is cut the chute.
We’re sailing along at 25,000 feet looking at our surroundings. If we were really sky diving we’d see a number of geological formations beneath us: flatland, rivers, mountains, and so on. That’s what’s amazing about being that high up – it’s possible to see how the mountains are connected to the rivers. You see how the rivers feed and nourish the surrounding area because the surrounding area is lush with vegetation and various critters. Finally you see that river find its delta where it meets the ocean. It’s a point-of-view that’s hard to visualize from ground-level.
What it doesn’t do is tell you why there are mountains and rivers; why life crops up around those rivers, or most importantly for this analogy: how to terraform.
This analogy, like life, revolves around water. In this case, a river.
There’s an idea that we’ve previously covered called the critical path.
This idea, for the uninitiated, is extremely simple.
Pictured: A critical path.
A critical path is the path a user must take through your website in order to complete the website’s primary goal.
Visualize it in its most common form: an e-commerce cart. The critical path looks something like this:
FRONT PAGE –> CATEGORY PAGE –> PRODUCT DESCRIPTION PAGE –> CART –> CHECKOUT –> RECEIPT
Now, obviously, it’s not necessary for somebody to start at the front page of a web site to buy something. Furthermore, this might not be the only way that a person could navigate the site that would result in a purchase. But what we just described is a from-the-rooter-to-the-tooter critical path. It’s got a little bit of everything.
If we look at this critical path more closely we can really divide it into two smaller sections based on what the pages do.
When a person buys something, they always go through this process:
Step 1: Find a product to buy
Step 2: Buy the product
Similarly, a website’s critical path can be divided into two sections: pages that help the user find a product and pages that help the user to buy.
I like to think of the pages that fit in Step 1 as FILTERS and the pages that fit in Step 2 as CONFIRMERS.
Let’s look at that critical path again:
Filters: FRONT PAGE –> CATEGORY PAGE –> (SUB-CATEGORY PAGE) –>
Confirmers: PRODUCT DESCRIPTION PAGE –> CART –> CHECKOUT –> RECEIPT
The Front Page, Category Page, (and sub-category pages if necessary) are all filters. Their purpose is to direct traffic to the Product Description page.
The Product Description page is really the beginning of the checkout process. Done right it will either sell the product using pictures, text, and multimedia; or it will confirm the shopper’s good sense to buy that particular product.
From the moment the shopper adds the product to the cart, the only job your website has is to get them into and through the checkout. This part of your website is so important that web analytics packages like Google Analytics include a tool that specifically measures this part of the process. It’s commonly known as a Conversion Funnel or Goal Conversion Funnel.
It looks like this:
The action in a conversion funnel is centered around the sales process – as it should be – because we want shoppers to buy the stuff they put in their cart.
But what about those FILTER pages? Where’s their visualization tool?
Web Usability folks have the tools to do the job but they’re rarely considered as tools to measure the effectiveness of how a user filters the information on your site. The tools are the click-maps that visualize where users click on a page. If one has the resources to do an eye-tracking test, then the heat-maps are just as valid a tool but they track the user’s eyes instead of their mouse. It’s the same underlying principal that’s been taken to another level of granularity.
What does this mean for designing a web usability test?
To me, it means that we’re out of the clouds where we talk about the three tiers of goals
Tier 1: Business and User Goals
Tier 2: Site Goals
Tier 3: Page Goals
and into a place where we connect Tier 1 to Tier 3. Once you see it, it’s so obvious it’s hard to imagine that you hadn’t seen it before.
The way you do it is by drawing a path from Step 1/Tier 1 (Goals) to Step 3/Tier 3 (Profit). Because it’s such an important path, one might even call it a critical path.
So you see, now we have the ability to design a user test because now we know the goal of user testing.
The goal of usability testing is to determine the effectiveness of your website’s critical path.
This makes all the sense in the world. The website is a machine built specifically to achieve an objective. User testing needs to be focused around measuring how well the machine achieves the objective.
Now, knowing that, how does that translate into actually designing a web usability test?
Newman will have to speak to that. Fortunately that’s what he’s preparing for Friday.
On Friday, Newman is going to break down for you the basics of creating a user test and how designing a user test around the critical path forces radical deviations from the standard questions that are often found on web usability tests. It’s sure to cause some controversy because for some jacked up reason it seems to fly in the face of conventional wisdom.
3 Questions For You
1. What web ux tests have you done and what was its primary goal?
2. Have you ever done a web ux test where evaluating the efficiency of the site’s critical path?
3. Can there be valuable non-critical paths in a website?
This week Ben and I discuss the usability design process as detailed in the Usability.gov website. We decided to focus on that because “God loves a working man” and we figure it’s the most general, benign UX method.
We also make a bunch of religious references – All in fun, nothing serious.
In order to dig into this new process and all things usability, we introduce Little Wing Marketing’s project with Java Jack’s Coffee shop. It’s a website revision project that Ben is doing and we are going to run it thru a testing process – Here, for you, live (recorded) on the Better User Experience Podcast.
Some system notes: We have changed the name of the podcast from “The Ben and Newman Show” to “The Better User Experience Podcast” and recorded the show intro using a new format. And, we’ve included sponsors – this week is Little Wing Marketing 🙂
Listen and Enjoy. Links and Images for your reference below. – Newman
Usability.gov User Centered Design Process
Java Jacks Home page design
Thoughts on my user testing design process
Based on my background with instructional design and guerrilla web building, I’ve tried take a new look at my design process from the perspective of usability and user centered design.
I’m sure it fits in with the larger context and conversation going on in the very robust user-centered design community. My purpose isn’t to make it fit, but rather just notice how I would go about conducting a user centered design. I know it will change- it’s already changed twice this week. It seems I change this design process each time I read an article on smashing Magazine.com or usability counts.
With most good things, my user-centered design process has a beginning, middle and end… Thinking of changing that to include the word terminus which is a beginning and ending, but I digress.
Beginning – “Stand in the place where you are” – R.E.M
I like to start with what I (or the client) know. Business goals should not be confusing, abstract nor complicated. “Sell more widgets”, “Make money”, “Provide health care to my workers”, etc. They should be clearly stated in a business plan or firmly in the mind of the business owner. From these business goals we can derive site goals and page goals.
Site goals can be anything:
- to inform
- to sell
- to build trust
- to reduce paperwork
- to recruit new employees
- to build a community
- to entertain
Each page should have a primary goal-or the thing you want the user to do. Don’t give your pages an identity crisis. Don’t make your users search for the purpose, or ‘call to action’, or banana. Pages, like PowerPoint slides, are free. Use as many as you need. The idea is to have a clear goal for each page. You can test this clarity directly in your user tests.
The clearer the goal, the more straightforward the user test.
Aligning the business goals, site goals and page goals is key. If the business goal is “to sell” and the site goals are “to entertain”, then there’s going to be a problem. The biggest problem I see in websites is unclear or confusing site goals.
The last part of the defining what you know is to collect information for baseline performance. You may need to survey company employees or site users to collect this data. Much of this information is”laying around” or extant within the organization. Examples of this are business plans (for business goals), previous website’s design documents or web traffic data (for redesigns) or advertisements or marketing materials.
Now that you’ve collected and define what you know, it’s time to make a few estimates and educated guesses. This could be called a hypothesis. You’re going to make guesses about the users, and how and where they use your site. Why is this important? One, this is what you will confirm or refute with your test results. Two, a Saturday morning website is very different than a Saturday night website. Other questions are, are people using the site at work? At their desk? On the couch? On a mobile? Users determine how your site will be used and you have to make design decisions based on these assumptions.
You want to dive deeper into who these users are. What are their goals? Why are they going to your site? How did they get there? What are their demographic attributes?
Closing the beginning of this design process is choosing the testing environment and tools. Basically-whether you use technology or not – it comes down to interviews, surveys, and small-group testing [watching / observing the user interact with your site]. I suppose for websites you can also add automated reports and services like the W3C site checkers.
The middle part – a.k.a. the fun part
Once you have all of your “knowns” about the business and you’re educated guesses about the user and their context, you are ready to draft and refine your testing instruments. What sort of things are you still unsure of? Which assumptions are not agreed upon-there will be some in any group. That’s what and why you test.
You generally test a website or any system for these three things:
- Effectiveness – Can users do what they ( And you) expect? mechanically, does it work?
- Efficiency – Is it easy to use? Does it work naturally or does it take concentration?
- Satisfaction – Sometimes called the ‘smile test’. Do people smile when they use it? Are they happy to use it? is it delightful?
I won’t go into too much detail here, but the general structure of a user test is:
- Pre-test Questions
- Participant Tasks
- Post-test Interview
- Post-test Survey
Before you conduct the test there are a few steps to follow. You have to:
- Choose a facilitator-preferably one without much experience with the design or site itself. Recruit testers-preferably of the target market demographic
- Decide on the links and location of the test – preferably close to the environment that the user will interact with the site, at home, at work or in a mobile situation.
- Prepare a script for the facilitator-this avoids bias delivery and provides a standard control especially if you’ve got multiple facilitators
Fin – the end
The last part is to collect, analyze and report the results. If you prepared and conducted a well made test (lucky?) clear patterns will emerge from the test results. You may see that people are unclear about your menus and can’t find certain information. Or you may find that there is a loophole in the structure such that people get in but can’t get out.
It’s important to remember to test the user experience – not your design. The users are experts in their own experience. They couldn’t care less about color scheme or fonts. They won’t tell you that experience and less you ask them. If you ask about design, they will tell you about design.
The goal of all of this is to make new assumptions about your users. These assumptions are now based on your test results. You have a more complete picture of your users and your site. Hopefully there aren’t some clear improvements you can make to your site. You can make those changes and test again to see if “The needle moved”, the changes were effective as anticipated. Then, you start the process over and test again. Each time you will get a more complete picture of your users and insights into how to make their experience with your website better.Thanks to Flickr: Sebastian Bergmann
This week Newman and I sit down to discuss brass tacks about user design for the coffee website that we’ve been working on. On the way there we take a large detour through the land of hacky web design. That’s right, I talk about my thought process of web design and see how my serious lack of forethought into a systematic way of approaching design allowed me to create a functional website without having to ask any of the big user testing questions.
How does this happen? Why is it bad? What can be done to fix the situation now and to avoid it in the future?
It’s all in this week’s podcast.
As always, if you haven’t yet, be sure to subscribe to the podcast on iTunes and check back on Wednesday for that long-promised blog post on the Top 5 Useful Reports in Google Analytics.
When I first sat down to write this blog post, my primary goal was to answer these questions.
- What is relationship between the results of Feng-GUI and of fivesecondtest.com?
- Which website is better if you had to pick ONLY ONE to use?
They are certainly not the same test – One is human driven and the other is completely automated, a machine.
My secondary goals are:
- to learn the interface (and back end) for these two tools and,
- sharing them with you, my dear reader(s).
There’s a third goal – which normally would be the primary goal – which is to get some feedback on a client’s website – mushroommaestros.com – but this time, I consider it to be a bonus. In the interest of full disclosure I should mention that mushroommaestros.com is my brother’s site.
What I did
I’m going to take a screenshot and submit it to FENG-GUI. I’ll use their full service – which is pay as you go, $2.5 per image. I’ll buy 10 images for $25, and I’ll have to use these within one month.
Next, I’ll submit the same screenshot to fivesecondtest.com. And, write questions that would measure the user’s impression of the site.
Roadblocks or interesting questions:
Do I submit a screenshot of the whole page or just the visual portion? The limit of both of these testing options is they don’t mimic the actual performance environment of viewing a webpage. Simply put, viewing a screenshot is not viewing a webpage. When viewing a screenshot you can scroll up and down, click away or use any interactive elements. I think I’ll use the ‘visible portion’ screenshot for both tests. I may go back and change to the full page screenshot.
Intermediately as I look at the screenshot, I’m interested to change a few elements on the page. I would make the menu text bigger and I would put a stronger call to action visible ‘above the fold’.
I signed up for the account by using PayPal and they send me an e-mail with the account information. I uploaded a photo and following the instructions click on the analyze button. Very simple and very quick – I started (and finished) the test in less than 30 minutes.
Reading the Feng-GUI help section, I learned that they recommend using a screen capture from “above the fold”. Problem solved.
The Feng-GUI test generates several different reports
- Attention heat map – displays the most attractive elements
- Gaze plot report – shows the scan paths and order between elements and includes the shortstops (fixations)and fast movements (saccades) of the eye.
- Opacity map report-this is another way to visualize the heat map report
- Area of interest (AOI) report-this report gives a percentage of attention based on predefined areas. I didn’t do this report because I forgot to predefined my areas.
You can download a zip file of the individual report graphics – your original image with an overlay of the four reports.
Impressions of Feng-GUI:
I’m happy with the reports. I was worried that the image of a girl would distract attention from other elements. I’m happy that the first ‘fixation’ was the text, “We sell fresh mushrooms”. Now the questions are, weathered the text is readable and attractive. Or, rather, does it meet the goals of the page. The goal of this page is to bring people farther into the site and make it clear of the primary business goal of selling mushrooms.
How might this affect my tests with 5 second test.com? What are the questions that I want to ask?
I had signed up for the sevice before – So I login and click create test. There are 3 type of tests you can do – Click test, navFlow and FiveSecond test. A start a new fivesecond’er and type in this information:
Prompt or Task:
- You searched for “fresh mushrooms in Oakland California” and clicked on “Mushroom Maestros: fresh mushrooms, urban myco-farm supplies…” (added – Please maximize your browser window and look for 5 seconds.)
- What does this company do?
- What is the name of this company?
- Does this website look professional? or Does this company seem professional?
- What stood out most to you?
Once you input the questions – you can have a maximum of five – you may preview the entire test process before publishing the test.
On previewing the test I decided to remind the user’s to expand their browsers. The new prompt looks like this now:
You searched for “fresh mushrooms in Oakland California” and clicked on “Mushroom Maestros: fresh mushrooms, urban myco-farm supplies…” – Please maximize your browser window and look for 5 seconds.
I set the tests to give 15 responses.
Impressions of Fivesecondtest.com
After a few days and only using the fivsecondtest users, I’ve received three responses to the survey. While it’s too early to tell anything definitive, I do have a couple of thoughts. Because the ‘game’ of the site (Do other’s test to get credits to give your tests), the responses will be from the demographic of the site users-which may not be your target audience. And, they may not really care about your test and the real purpose is to generate credits to do more of their tests… It could be a good thing but your responses may lean towards the technical and ‘hasty’. For instance, I got a response that said “needs a stronger call to action”, which is something I don’t think the general public thinks of.
I think I will hold off on distributing the test link to my extended network, so that I can continue to get this web designer, http://usabilityhub.com – user perspective. Perhaps I’ll wait until I get 10 responders and then link to it on my facebook and twitter accounts – Or ask a friend to do it.
Summary and conclusion
What is the relationship between 5 second test.com and Feng-GUI? It’s too early to tell. There is too little data from the human test and I don’t think that my questions really were valid to answer that primary question.
If I had to only pick one to use, at this point, I would use Feng-GUI because it very quickly gives me the specifics and actionable information. I can tell my client, “I changed this because of that.”
I’m very happy to learn how to use both of the tools. And, I got a few ideas on how to tweak the mushroom maestros design. Ultimately, that’s the best result – We test to answer questions and generate more questions, generating more tests and so forth. The iterations are what I’m after.
PS: I spent a solid hour trying to get that gallery to work. The captions wouldn’t work and I couldn’t figure out how to exclude the Watson image using the visual interface – had to use the Gallery shortcode and then I lost the settings for the rest of it. Regardless, it’s Friday and count the stars in the sky.
It’s late Wednesday – later than I’d like for my weekly blog post but I have an excuse! Newman and I ended up having a long conversation instead. We taped it, called it a podcast, and now, here it is for your aural enjoyment.
Regular readers know that new podcasts are a Monday thing and this is clearly not Monday.
And you are correct! Way to know your days of the week.
Today’s topic though is messy. It’s better explained through a series of ramblings than through a discrete article. In fact, the basis of the podcast is about that very thought. Newman and I talk about the book The Information by James Gleick, or at least about the prologue.
You know a book is going to be good if you can do 45 minutes on something you read from the chapter before the first chapter. If you want to “Take A Look Inside the Book” on Amazon, you can do that here.
We also refer to a few TED Talks, which you can find below.
TED Talk: David Christian: Big History
TED Talk: Aaron O’Connell: Making sense of a visible quantum object
Enjoy the podcast (subscribe!) and be sure to join us on Friday for Newman’s epic blog post where he compares fivesecondtest.com to Feng-GUI. It may be a tag-team match, but you’ll have to wait until Friday to know exactly what that means. In the meantime, hit us up in the comments.