Happy Friday to you all!
Thanks to the good folks at Smashing Magazine, UX Department and their article titled: Comprehensive Review Of Usability And User Experience Testing Tools we have updated our list of UX Tools. They will reside permanently on the menu and are displayed below for your browsing pleasure. Enjoy the weekend.
Conduct a User Test
Heatmaps / Mouse Tracking Tools
Have you heard the good news?
Web usability testing works.
What’s more awesome is that just the act of focusing on your website and thinking about it systematically can result in website improvements. I’ve spoken with two business owners in the past week – one a DJ and the other an accountant about their websites. I talked to the DJ a few weeks back about defining a critical path on his website and helped him think through what services he offers and how to make them part of the critical path. He called me earlier this week ecstatic. He had made the changes I recommended and in the few intervening short weeks, incoming calls from new perspective clients increased 300%. Many of those callers specifically mentioned that the website was what provoked them to pick up the phone.
He couldn’t be happier.
The other guy is an accountant and I just spoke to him today about defining his website’s critical path and redesigning the website around it. After years of having the same website, just talking about a critical path made his eyes light up. He immediately saw what I was getting at and it caused him to think about his website and how it can be improved. If he can maintain the dissatisfaction with his current site, based on the idea that it doesn’t create a good flow through a critical path, then he will eventually decide to make the changes that will help his business, like the DJ above.
I also have a good friend who runs a web design business. We’ve been talking about web usability and employing the techniques we discuss here to improve the websites of his clients.
To get him in the mood to talk web usability, Newman and I decided to take the front page of his website and run it through Feng-GUI to see what it thought of the website.
Here’s what we tested: (I’ve removed the name of the company and the location in which he works out of respect for my buddy.)
And here’s what we got back:
What you’re looking at is a gaze-plot chart. The circles represent the places a web user would look on the site. The number in the circle tells you the order and place where the user looks.
Now, the $64,000 question. What are the take-aways from the gaze-plot chart?
Remember that the purpose – in my mind – of user testing is to shore up the website’s critical path.
What we see in the above image is that the user is utterly confused about what to do. There’s a small giveaway that the company knows their website is difficult to navigate too. They have a small “Questions?” pop-up balloon at the bottom of their page where users can chat with somebody from the company.
To me, rather than looking like a nice way to make yourself available to prospective clients, what the Questions bubble says to users is that their website is too dense and that you can short-cut your way around the whole thing just by talking to somebody directly.
But if the user wanted to talk to somebody directly, why didn’t they just pick up the phone?
So right away we have some cause to be concerned about determining the critical path of the website. The “Questions” balloon isn’t inspiring confidence and the gaze-plot seems to confirm the idea that the site is confusing to visitors.
The next thing I wanted to do was look at what their critical path should be. To do that I looked at their menu and listed out all of their options. They are:
- Client Login
Additionally they have three tabs that also seem to be part of the menu so I will add them here as well. They are:
- Web Services
- Network Services
Remember, a critical path is the path that a user takes in order to get what they (and you) want. In this instance, because they are a service-oriented firm, the website is supposed to provide leads. We know that to be true because the company went so far as to put a “Request A Quote” on the front page. So we can list a contact as the ultimate goal, and the front page as the logical place where the user will start their journey.
Let’s write all that out.
Step 1: Front page
Step 2: ???
Step 3: Contact (Profit!)
What you’ll see here is that, as seems to be the norm, step 2 needs to be better defined. Step 2 is the path between the front page and getting that contact.
In order to determine what step 2 should be, we need to look at what it is this company does. Using the menu items above, I will now divide that list into Things They Do and Other Stuff.
Things They Do
- Web Services
- Network Services
- Client Login
When we group their menu items by what they do, it immediately becomes apparent that they are not giving themselves a fair shake. Part of this has to do with terminology and part of this has to do with the fact that not everything they do is listed on the website. This is a problem because I know they didn’t mean to be intentionally vague.
That leads to the obvious question, what do they do?
Since I have experience with this firm, I can tell you that they do five main things:
- Web Design
- Web Marketing
- Hardware Sales
- Networking Services
All of which, except for hardware sales, is accounted for (if just barely) by their list. The difference between the two lists is that my list is consistent and their list is alternately too specific and too vague. They list ‘design’ as something they do but it’s not self-evident that it’s web design. They do SEO, PPC, and social network marketing but all that’s listed in the menu is ‘SEO’. The phrase ‘web services’ is confusing because it sits next to ‘design’ and ‘SEO’ which makes you wonder what else ‘web services’ could be referring to. And finally there’s ‘Network Services’ which is something they do, but for some reason when I was first doing this, I ignored.
In place of their list, I would substitute mine.
Next, I’d take a look at all of the remaining menu items and I would try to combine and arrange them into a coherent secondary menu. The reason we have these items on the secondary menu is because we are distinguishing between pages that get a user onto the critical path and pages that get a user onto a secondary path.
In this instance, the above Other Stuff list can be condensed to:
- Client Login
Furthermore I would substitute the phrase “About Us” for “Company” and would replace the “Contact” page with a “Customer Service” page.
This leaves two menu items left, the portfolio page and the testimonials page. I’ve left them out because while they aren’t technically part of the critical path, it’s virtually guaranteed that this page will be viewed by potential clients before they make the decision to reach out and contact the company. This makes it necessary to promote it to the main menu rather than to keep it on the secondary menu. It’s such a substantial part of the selling process that I believe it’s better to weave the portfolio and testimonials throughout the website.
This leaves us with the following primary and secondary menus:
- Web Design
- Web Marketing
- Hardware Sales
- Network Services
- Client Login
Now all that’s left is to draw some boxes that bring all of this together. Here’s what I did:
From there Newman mocked it up and ran it through Feng-GUI.
What we were hoping to see was a visual narrative that made sense. In our original test, the gaze-plot was all over the place. Now that we’ve put the most important things out front and center, we should see the gaze-plot be a lot more orderly, an indication of understanding.
Here’s what Feng-GUI thought of the redesign:
It’s a mock up that’s for demonstration purposes only, not meant to be other than looking at how the content blocks are seen. And I think we can safely say that this is a BIG improvement over what currently exists there now.
What we’ve proved is that if you keep your eye on the critical path of your website and design your information so that it can be digested at a glance, it will yield valuable results.
For Further Consideration
One thing that kind of bugs me about the above gaze plot is that nobody seems to be reading the information that’s in the main image. Arguably, that should be some important info there. It leads me to think that this opens us up to testing different headline sizes and different messages in that space to determine what works most effectively.
- This whole post presumes that Feng-GUI returns valuable data. Is it dangerous to make all of the assumptions that I did in this post based on an algorithm that simulates eye-tracking rather than doing it with a real test subject?
- Does this mean that we’re forced into a box model for most service-oriented websites?
On this week’s podcast Ben and I discuss the critical path for websites, explore secondary website paths and talk more about how to design a proper web usability test.
Listen and Enjoy. Links for your reference below. – Newman
Definition of manipulation on Thesaurus.com
What Every Body is Saying by Joe Navarro
Oh, and the music around the 15 minute mark is by Skrillex.
Last week we drafted a usability test and tested one user in order to get a real experience with the theories and abstractions we were researching and discussing. Our results were surprising.
What We Wanted
We wanted to figure out a repeatable process for conducting a user test that would improve upon the simple watercooler test – The type of test that my friend at textWoo.com conducted. Additionally we wanted to “user test the user test” by way of quickly getting feedback on an early draft of our test scheme in the hopes of creating a more effective testing tool. And, we wanted to dive in.
What We Did
I downloaded and edited a script from the usability.gov site which came from the SUS framework and a usability test from 3w.org. The way I customized it was to look at each page type in the JavaJack’s site and create a task that would engage the user with that page. I didn’t make the tasks ‘hard’ but I didn’t want them to be too specific. I didn’t want to test the users ability to read, for instance. Our friend, Anna, came by just in time to be the tester.
What We Got
The results of the very unscientific test were intriguing and beneficial to the overall design of the site. As Anna talked thru using the site with Ben and I there watching, we noticed some things that could be improved and changed. Time well spent.
After the test, our thoughts turned to the testing process itself. Ben’s post explains his thoughts on the critical path and the purpose, goals and objectives of the sites and how they relate to user testing. Here are other thoughts about the user-test and the testing process in general.
User’s aren’t designers, don’t ask them to critic a site design
Most scripts I’ve seen ask the user this question in the beginning of the test:
Please give me your initial impressions about the layout of this page and what you think of the colors, graphics, photos, etc.
When you asked this question, it seems that the users become unsettled and defensive. They have to form an opinion and defend it. They stop using the site. They critique the site instead. They start to look at the site as a designer. The user is tainted after that simple little question. Ask the user about their own experience… and why not ask them AFTER they have the experience.
To fix this we came up with a few ideas. Let them complete the tasks and then ask them about their experience. Record the session (the screen, webcam and audio) without you in the room. Just don’t ask the question, there are better ways to get initial reactions from users – the user-testing sites in the sidebar. Or, conduct separate tests for reaction and impression.
Hiring UX facilitators: Flys apply within. Humans need not apply
You want to be a ‘fly on the wall’ as much as possible. Humans suck at this. Evaluator bias is rampant and I’m doubtful you can eliminate it. Simply put it’s when the testing-user feels that they should answer a certain way or feel the agenda of the test questions. Who doesn’t feel that in every survey! Human societal norms get in the way here (Perhaps not in New York City, granted). People tend to be polite to strangers and people in authority. I figure negative answers questions are rare. Users will try to figure out what you want and try to give you that. The act of testing will influence their use. I feel the designer is the least desirable person to be the facilitator of the test. If the user feels that the facilitator has an agenda or prefers one outcome over the other, then the test is compromised.
You can’t hire actual flys to do your testing (they don’t make lab coats and clipboards small enough). Here are a few ideas we had to correct the problem on Evaluator Bias and human factors. You could use a remote testing service – like the ‘Mouse Tracking Tools’ in the sidebar. Run face to face tests in a familiar place to the user – office, coffee shop, mall, home – so the user is more comfortable giving honest opinions. The facilitator should not be perceived as someone affiliated with the site – regardless if they are or not – nor an authority of any kind. Perhaps, you can test several other sites to hide the site you are actually testing (This might be time/resource intensive)
Something is better than nothing.
However, doing any type of test is better than nothing. The simple act of watching somebody go through the site is very desirable. Testing gives you insight into the flow of the site, if the site is mechanically (or functionally) sound and working and if the user finds and stays on the critical path or primary site goal.
You sellin’ what I’m buying? Great, let’s do this.
Each user has their own goal when coming to the site. Each site owner has a goal when building a site. If the two goals match, Great! Now get out of the way and let the site churn out money. The user clicks thru the site and their goals is met. This click trail thru the site is called the Critical Path. And, it’s what you should test in the ‘Tasks’ portion of the standard usability test.
There can be many paths, but only one critical path on a site. For example, I go to Apple.com to watch movie trailers. Would Apple user-test my experience in getting to and watching movies? Perhaps, but I bet they measure how easy it is for me to ‘jump over’ and buy some music or a new computer. The purpose – the critical path – of the site is to sell (I’ll give you branding and customer service, as additional paths) and all other features and functions of the site support that goal.
How well the site moves visitors along the path is the effectiveness of the site. And, we test for it by asking testers to assume they have the same goal as the site. Likewise, we test for satisfaction. Did they complete the task, but were pissed because of something else? Did they expect one thing and get an unpleasant surprise? Also, we test for efficiency. Did they complete the task, but it took 15 clicks and 20 minutes to complete?
You can test this by doing a very simple user tests. That’s the low hanging fruit of user tests. Site effectiveness, Satisfaction, and Efficiency.
In conclusion, don’t ask about impressions before your evaluation of the critical path. Do test even if the conditions are not scientific. Let users use. Visitors visit. Don’t force them to have an opinion and then needle them about it. This is a carry over from the designers perspective. User’s aren’t lab rats. Your color palette isn’t of supreme importance. Listen close and you can hear a user. They are probably saying, “Just give me the dang banana already”
Is the user qualified to speak to design?
How do you get around the evaluator bias?
Can one site have multiple critical paths?
If you’ve been following the website or podcasts over the past few weeks you’ve seen Newman and I dig into the topic of web usability. We approached it, at first, from a philosophical perspective. We talked about the web design process and where web usability fits into it. We found various tools online to assist in doing user tests and we’ve found various models for how to conduct user tests.
And we started to realize something: these models tend to be obnoxiously vague. Even when they make intuitive sense, they still leave the reader with more questions than answers.
I feel like they keep creating iterations of the old Sony business model:
Step 1: Bright idea
Step 2: ????
Step 3: Profit!
It’s Mr. Sony and he’s a naughty kitty!
WTF is Step 2? Nobody I’ve read (and I’ll readily admit that I have much more to read) has articulated a quality strategy for conducting a user test. And all the user test examples I’ve seen have bared this out.
What I’ve noticed is that somebody will develop a model. For example:
I actually like this model. I think it does a good job of describing the steps and grouping the various considerations into each step. But what drives me crazy about it is that it doesn’t compel me to do anything specific. This is a 25,000 foot view of the web ux universe. And as you’ll see, it’s practically the only view these models will present:
Same thing with this model, except in this model, they’re using words and phrases I’m familiar with but use them in ways that are not intuitive and meaningful.
You can look at those two models or even the one we were initially so high on in Monday’s podcast at usability.gov until blood shoots out of your nose and you’ll never get to a better understanding of how to conduct a user test.
What we need to do is cut the chute.
We’re sailing along at 25,000 feet looking at our surroundings. If we were really sky diving we’d see a number of geological formations beneath us: flatland, rivers, mountains, and so on. That’s what’s amazing about being that high up – it’s possible to see how the mountains are connected to the rivers. You see how the rivers feed and nourish the surrounding area because the surrounding area is lush with vegetation and various critters. Finally you see that river find its delta where it meets the ocean. It’s a point-of-view that’s hard to visualize from ground-level.
What it doesn’t do is tell you why there are mountains and rivers; why life crops up around those rivers, or most importantly for this analogy: how to terraform.
This analogy, like life, revolves around water. In this case, a river.
There’s an idea that we’ve previously covered called the critical path.
This idea, for the uninitiated, is extremely simple.
Pictured: A critical path.
A critical path is the path a user must take through your website in order to complete the website’s primary goal.
Visualize it in its most common form: an e-commerce cart. The critical path looks something like this:
FRONT PAGE –> CATEGORY PAGE –> PRODUCT DESCRIPTION PAGE –> CART –> CHECKOUT –> RECEIPT
Now, obviously, it’s not necessary for somebody to start at the front page of a web site to buy something. Furthermore, this might not be the only way that a person could navigate the site that would result in a purchase. But what we just described is a from-the-rooter-to-the-tooter critical path. It’s got a little bit of everything.
If we look at this critical path more closely we can really divide it into two smaller sections based on what the pages do.
When a person buys something, they always go through this process:
Step 1: Find a product to buy
Step 2: Buy the product
Similarly, a website’s critical path can be divided into two sections: pages that help the user find a product and pages that help the user to buy.
I like to think of the pages that fit in Step 1 as FILTERS and the pages that fit in Step 2 as CONFIRMERS.
Let’s look at that critical path again:
Filters: FRONT PAGE –> CATEGORY PAGE –> (SUB-CATEGORY PAGE) –>
Confirmers: PRODUCT DESCRIPTION PAGE –> CART –> CHECKOUT –> RECEIPT
The Front Page, Category Page, (and sub-category pages if necessary) are all filters. Their purpose is to direct traffic to the Product Description page.
The Product Description page is really the beginning of the checkout process. Done right it will either sell the product using pictures, text, and multimedia; or it will confirm the shopper’s good sense to buy that particular product.
From the moment the shopper adds the product to the cart, the only job your website has is to get them into and through the checkout. This part of your website is so important that web analytics packages like Google Analytics include a tool that specifically measures this part of the process. It’s commonly known as a Conversion Funnel or Goal Conversion Funnel.
It looks like this:
The action in a conversion funnel is centered around the sales process – as it should be – because we want shoppers to buy the stuff they put in their cart.
But what about those FILTER pages? Where’s their visualization tool?
Web Usability folks have the tools to do the job but they’re rarely considered as tools to measure the effectiveness of how a user filters the information on your site. The tools are the click-maps that visualize where users click on a page. If one has the resources to do an eye-tracking test, then the heat-maps are just as valid a tool but they track the user’s eyes instead of their mouse. It’s the same underlying principal that’s been taken to another level of granularity.
What does this mean for designing a web usability test?
To me, it means that we’re out of the clouds where we talk about the three tiers of goals
Tier 1: Business and User Goals
Tier 2: Site Goals
Tier 3: Page Goals
and into a place where we connect Tier 1 to Tier 3. Once you see it, it’s so obvious it’s hard to imagine that you hadn’t seen it before.
The way you do it is by drawing a path from Step 1/Tier 1 (Goals) to Step 3/Tier 3 (Profit). Because it’s such an important path, one might even call it a critical path.
So you see, now we have the ability to design a user test because now we know the goal of user testing.
The goal of usability testing is to determine the effectiveness of your website’s critical path.
This makes all the sense in the world. The website is a machine built specifically to achieve an objective. User testing needs to be focused around measuring how well the machine achieves the objective.
Now, knowing that, how does that translate into actually designing a web usability test?
Newman will have to speak to that. Fortunately that’s what he’s preparing for Friday.
On Friday, Newman is going to break down for you the basics of creating a user test and how designing a user test around the critical path forces radical deviations from the standard questions that are often found on web usability tests. It’s sure to cause some controversy because for some jacked up reason it seems to fly in the face of conventional wisdom.
3 Questions For You
1. What web ux tests have you done and what was its primary goal?
2. Have you ever done a web ux test where evaluating the efficiency of the site’s critical path?
3. Can there be valuable non-critical paths in a website?
This week Ben and I discuss the usability design process as detailed in the Usability.gov website. We decided to focus on that because “God loves a working man” and we figure it’s the most general, benign UX method.
We also make a bunch of religious references – All in fun, nothing serious.
In order to dig into this new process and all things usability, we introduce Little Wing Marketing’s project with Java Jack’s Coffee shop. It’s a website revision project that Ben is doing and we are going to run it thru a testing process – Here, for you, live (recorded) on the Better User Experience Podcast.
Some system notes: We have changed the name of the podcast from “The Ben and Newman Show” to “The Better User Experience Podcast” and recorded the show intro using a new format. And, we’ve included sponsors – this week is Little Wing Marketing 🙂
Listen and Enjoy. Links and Images for your reference below. – Newman
Usability.gov User Centered Design Process
Java Jacks Home page design
I have a client with a new website that I’m in the process of finishing. They’re a small coffee shop and the owner has a real passion for making the perfect cup. He’s the type of guy who regularly makes trips to the farms that grow the beans he roasts. At any time, you can walk into his shop and he has 18 fresh roasted coffees available for purchase. As on-top of their game as they are with coffee, it’s also true that like most busy business owners whose day-to-day operations consume all of their time, their current website looks like a pre-Web 2.0 relic. It performs like one too. That’s the reason I’m building a new one. It’s also why I’m confident that my new design will raise sales immediately. There’s practically nowhere to go but up.
Be that as it may, I’d rather sell my clients facts, rather than confidence, so in the coming weeks, we’re going to be doing some user testing. Essentially, the website is going into its beta release.
The purpose of the user test is to get feedback about the new design to understand if there are any parts of the website that are confusing to users, and to test the site for hidden bugs that might prevent a user from getting the information they need or from making a purchase.
We will be getting people to test the website that are already familiar with the brand. Some of them will come from their Facebook page; the rest will be users who order coffee regularly from the current website.
At our disposal are a plethora of tools to record and measure how a user interacts with the website, including:
- video taping the client (over the shoulder)
- tracking mouse behavior
- recording the user’s interaction with the website
- interviewing the user
- looking at web analytics
It’s also possible to have an eye-tracking study done but cost and time factors rule that out for this test.
So what tools should I use? I’m not interested in trying them all. I just want to use the tools that are appropriate for my needs and that provide me the feedback in the most actionable way possible.
Before I start to look at the tools though, I think it’s wise to list out the questions we want to answer and constraints that we have to work under for this user test:
- Can users find the product they want to buy?
- Are users confused by the choices in the main menu?
- Can users buy the product easily? (examine the checkout process)
- Do users have enough information to make a buying decision on a coffee they have never purchased before?
- Does every part of the website function as it is intended to? (this concerns the mechanics of the website)
- Can the users find any bugs in the site?
- Two week time frame
- Users limited to Facebook fans and current users
- Want to do it as inexpensively as possible
From these questions and constraints we can make some immediate decisions:
- We don’t need a service like Ethnio which will find users to test the website.
- We will want to do interviews with the users. The choices here are either to do a personal interview which can be recorded or to have the user fill out a questionnaire. I would prefer to have a video taped interview so we can look at the user’s body language and facial expressions. However, it’s possible that we will need to use a questionnaire instead due to time limitations. In either case, we would also like to be able to follow up with the users if we have additional questions.
- We need to record how the user uses the website. This could take the form of videotaping how they use the website in the over-the-shoulder style. Since we can expect to have users comb the site while we are not physically present, capturing their screen data seems to be a better option. We should look at programs that record the user’s session and mouse behavior. It would also be helpful to be able to retrieve that data as visual information, via heatmap or some other similar visual display.
- Because we will be testing a limited number of users, it is not important to track analytics. We should have the individual feedback from each user which is more granular data than analytics by itself. There might be a tool out there that combines analytics with user testing tools. If so, it’s worth looking at but realistically it’s not necessary.
That seems to about cover it. We’ve been able to distill our needs into two essential items:
- An interview/questionnaire
- Tools that record the user’s session on the website and provide quality reporting tools
Now that we finally know what we need we can look at what’s available and decide which tools best fit our needs. In the event that multiple tools can do the job, we’ll assess the pros and cons and then make a judgement call.
I’ve done some research already into what tools are available and they are all listed in the “Tools” section of links in the sidebar. I’m sure there are more tools than what I’ve listed but I’m also reasonably sure that we can find the right tools for the job from the websites listed. For the purposes of this article, and for our experiment, we’ll confine our set of possibilities to the 10 following choices:
We can use as many of them as necessary but since we’re also trying to be economical with our time and our money, ideally we’re looking for a one-stop shop that’s free. It’s unlikely we’ll find that. But hey, a guy can hope.
Let’s take a brief look at each tool:
ClickTale offers a suite of tools that look ideal for our needs. They include:
- visitor recordings
- mouse move heatmaps
- mouse click heatmaps
- conversion funnels
- attention heatmaps
- the ability to watch a user’s activity in realtime
- form analytics
- campaign tracking
Unfortunately, all of that great tracking doesn’t come for free. They offer three plans, starting at $99/month and going up to $990/month. The fact that there’s a monthly fee indicates that this product is meant for ongoing user testing. While this is a consideration – we plan to do additional user tests in the future – it’s $290/month for the plan that gives us access to everything in the above list. It’s possible that this might be the right solution for us, but it’s likely that it will prove to be prohibitively expensive.
CrazyEgg is primarily a click-mapping tool. I’ve had extensive usage with this tool in the past. It’s great for what it does: track mouse-clicks and display the data in several easy-to-understand maps. It doesn’t record where the user moves their mouse. It doesn’t record a video of the entire session. And it costs money. Plans range from $19/month to $189/month with the $49/month plan making the most sense since it updates hourly instead of daily like the cheaper $19/month package. For our purposes, CrazyEgg might be too limited in its capabilities for us to use it, especially at $49/month. It’s likely at that price range that we will find other tools that offer more functionality. We shall see.
Feng-GUI, if it works as advertised, is a great idea and could be very helpful to web designers. The idea is that designers can upload an image of a project they want analyzed, be it a web page, a print ad, or other design. Feng-GUI “looks” at the image using an algorithm that mimics the human eye and human attention to generate a range of visual-attention maps. At $25 for 10 images, this is a dirt cheap way to get early feedback on web designs.
You could make the argument that we would benefit from submitting an image of the front page, the category page, the product description page, the cart, and the checkout pages to Feng-GUI to see what it spits back out. For $2.50 an image it’s not the kind of thing that needs a lot of thought. Just do it already.
I can’t help but think that this service is much more helpful at the beginning of the design cycle than it is at the user testing phase. But I’m biased. It’s my design. And I can’t bear the idea of the algorithm telling me that I need to redesign significant portions of the website. I want to know that before I code it all up so that users are picking nits instead of tearing the entire site to shreds.
Regardless of my feelings as a designer who is attached to his design, the fact remains that Feng-GUI provides a way to get approximate eye-tracking metrics on a design for a very cheap price. As I said at the beginning of this post, I wanted to leave eye-tracking out of this discussion because of the cost and time factors. But Feng-GUI looks like such a great tool that we’re going to have to give it a try.
At a minimum, we’ll have some interesting feedback that we can compare with the feedback we get from our users. It will be enlightening to see if it agrees with and can predict the users behavior and/or interview responses, or if it contradicts them. For the price of a cup of coffee, we can’t say no.
Inspectlet is a site that allows users the ability to record visitor sessions, display mouse-click heat maps, and tracks analytics in real time. In short, it does everything that we need to satisfy point #2 on our two-point list of needs. Its heatmap reporting isn’t as extensive as what ClickTale and CrazyEgg provide but for our purposes, because our test will be limited to 5-10 people, a heatmap of mouse movements is more of a luxury than a necessity. Unlike CrazyEgg, Inspectlet offers the additional features of being able to record the user’s session and access to real-time analytics.
Unlike ClickTale, Inspectlet is much more reasonably priced. Plans range from $7.99/month to $89.99/month.
Even better, for us, the $7.99/month plan looks perfect. It includes:
- Up to 50,000 Pageviews
- Full Real-time Analytics
- 800 Screen Captures
- Unlimited Heatmaps
- 1 Custom Metric
To top it all off, they offer the first week for free! It seems likely that we will be using Inspectlet as part of our user test. It has the right features at the right price.
But just to be sure, let’s continue to see what the rest of the list has to offer. Maybe we’ll find something even better.
KISSmetrics calls themselves “person-based analytics”. I wasn’t able to figure out exactly what that means by looking at their website. I believe it has something to do with showing Analytics to different personality types. For example, the web dev sees one thing, the sales guy another, and the stockroom sees a third. But it’s a little unclear.
KISSmetrics first popped up on my radar when WIRED did a story on them about how they track users in a way that they cannot delete. The hype was massively overblown but the company was forced to use a new method which users could disable.
What KISSmetrics really looks like they’re doing is that they’re tracking users all around the web to get a better understanding of just what exactly brought them to your website. In that sense, KISSmetrics looks like a great tool for ongoing analytics and testing. Prices start at $29/month.
Loop 11 almost owns this thing. There’s one big gaping assumption that I made when I got all sweet on Inspectlet a few minutes ago. I’m assuming that the user will know what they are supposed to do, or that they will be prompted by somebody to accomplish a task. This may be true. Whether that’s the case or not, the reality is that the user is going to have to be prompted to take specific actions on the site so we can measure how they behave.
Loop 11 provides a way to do that. Loop 11 is a tool that allows users to create a user test. They generate a link which can be sent to people in order to get them to participate in the test. This would be helpful to us because we could put the link out on the company’s Facebook page and get more than 5-10 users’ feedback on the site.
They have partnered with OpenHallway to allow their users to record the visitor’s session as well. This gives rich feedback in the form of Loop 11 reports and the individual video sessions.
The reason I said that Loop 11 almost owns the category is because there are a few features missing. The biggest one is the lack of heatmaps. One could argue that it’d be nice to have Inspectlet’s analytics but, as we’ve already covered, analytics isn’t necessary for what we’re doing. But the heatmaps, those are a necessity. In my experience with CrazyEgg, everybody, even the dumbest guy in the room understands heatmaps – without instruction. It’s such a powerful tool in that sense that it’s a must-have for us.
The massive downside is the cost. It’s $350 per test. Is it worth it? That remains to be seen. If we can conduct our own tests, then probably not.
Morae is a user testing solution similar to Loop 11 but it’s sold as a software package, has more features, and is more expensive. The entire package costs $1,495. It seems appropriate for a usability business or for a company that is dedicated to ongoing usability testing. For our purposes, it’s too expensive and it seems limited to testing users on specific machines. It’s not appropriate, unless I’m misunderstanding, for conducting a user test via the Internet.
SilverBack is like Morae in that it is tied to a specific computer. It’s much more reasonably priced at $69.95 and while it doesn’t offer as complete of a feature set as Morae, it does seem like a great tool for quick in-house user tests. A cool tool but it’s not right for what we’re doing here.
Usabilla looks really cool. It seems to be essentially a less expensive version of Loop 11. They are a website that allows you to create and conduct user tests. They have great reporting tools, including heatmaps, unlike Loop 11. But while Loop 11 has partnered with OpenHallway to offer video recording of user visits, Usabilla doesn’t offer this feature. However, after looking at OpenHallway’s website, I think it might be possible to feed the link that Usabilla generates into OpenHallway so that OpenHallway can record the user tests. Using OpenHallway will cost $19/month for their smallest package, which will work fine for our needs.
Usabilla’s price range goes from free all the way up to $139/month. Depending on how much data my client is comfortable sharing on this website, the free version could work just fine for us since it includes 1 test with 10 participants. The downside is that the results are public. The next tier up is $49/month, allows for 1 test with 50 participants and the results are private.
Finally, we have UserFly. UserFly is like OpenHallway in that it’s a simple tool that records how a user interacts with your website for late playback. Unlike OpenHallway which is priced based on hard drive space (their smallest plan allows for 90 minutes of video), UserFly is priced based on the number of captures. They offer 10 captures a month for free. Prices top out at $200/month for 10,000 captures. Who would ever look at 10,000 captures is anybody’s guess.
My gut feeling here is that OpenHallway is a better value. They also seem to play nice with Loop 11 which makes me think they can play nice with Usabilla. While UserFly is definitely worth trying – especially since we can do so for free – it really seems like a good topic for a blog post but it’s not quite what we need for this user test.
The Final Results
After all of that, it seems clear that Feng-GUI, Inspectlet, and Usabilla are the tools we should use for our user tests. With Feng-GUI we will get some great data that approximates an eye-tracking study. With Usabilla we have a tool where we can actually conduct a user test. We can write the instructions for the user and they can take the test without need for us to moderate it. It will provide us with great feedback based on the outcome of the user tests. We will use Inspectlet to record the user studies and will generate additional data and heatmaps that will show us more directly how the users behaved on the website.
The cost is reasonable too. For one month of testing it will cost:
- $25 for 10 Feng-GUI tests
- $7.99 for Inspectlet (the first week is free)
- $49 for 100 tests on Usabilla (or 10 for free)
If we found ourselves on an extremely limited budget, we could manage to conduct 10 user tests for free. We would have to do without the Feng-GUI analysis and we’d have to allow the Usabilla reports to be made public but the upside is, it wouldn’t cost us a dime. However, in this particular case, $82 seems like a completely reasonable price to pay for a month of user testing.
Count me in.