This Microsoft production is one of the vision videos that’s been making the rounds:

It’s cool, but also cold. And it’s one of the best of the bunch (Corning’s A Day Made of Glass is also very good). Others, such as the awkward imitations produced by Research In Motion (Blackberry) invite only ridicule.

Interface designer Bret Victor has produced an intelligent critique of the Microsoft video (and, by extension the whole genre). He starts by reminding us of the incredible sensory and manipulative powers of the human hand:

There’s a reason that our fingertips have some of the densest areas of nerve endings on the body. This is how we experience the world close-up. This is how our tools talk to us. The sense of touch is essential to everything that humans have called “work” for millions of years.

But what is the sensory experience of Microsoft’s future (and Corning’s, and Apple’s, and RIM’s)? It’s the feel of glass. It’s “glassy.”

Now read this: The 5 Best Toys of All Time. I think you’ll get my point.

The broader question of how to measure success is one that we raise with our own clients at the beginning of every project, as this helps us figure out the organization’s priorities and focus.  Definitions of success range from trackable statistics (“more users will see the catalog”) to anecdotal assessment (“employees will complain less about using it”).

There is no one-size-fits-all approach to measuring success.  Moreover, with the exception of online survey tools like Zoomerang or SurveyMonkey, which can be used assess usability and satisfaction, most tools today are designed to measure success from a business or technical staff’s perspective, rather than the users’.  Google’s researchers recognized this problem in assessing their own applications and developed the HEART metrics framework, a method of measuring user experience on a large scale.

The HEART framework is meant to complement what Google calls the “PULSE metrics” framework where PULSE stands for: Page views, Uptime, Latency, Seven-day active users (i.e., number of unique users who used the product at least once in the last week), and Earnings– clearly all stakeholder and/or IT concerns.  While these statistics are somewhat related to the user’s experience (which pages get looked at, which items get purchased), these can be problematic in evaluating user interface changes:

[PULSE metrics] may have ambiguous interpretation–for example, a rise in page views for a particular feature may occur because the feature is genuinely popular, or because a confusing interface leads users to get lost in it, clicking around to figure out how to escape.  A count of unique users over a given time period, such as seven-day active users, is commonly used as a metric of user experience.  It measures overall volume of the user base, but gives no insight into the users’  level of commitment to a product, such as how frequently each of them visited during the seven days.

The HEART metrics framework offers a way to more precisely measure both user attitude and behavior, while providing actionable data for making changes to a product’s user interface.  These include the following, which I’ve described very briefly here:

1. Happiness. This metric is concerned with measuring the user’s attitude toward the product, including satisfaction, visual appeal and the likelihood that the user will recommend the product to others.  The use of a detailed survey as a benchmark and then later as changes are implemented will cover this.
2. Engagement. This measures a user’s level of involvement, which will depend on the nature of the product.  For example, involvement for a web site may be as simple as visiting it, while involvement for a photo-sharing web application might be the number of photos uploaded within a given period. From a metrics standpoint, involvement can be assessed by looking at frequency of visits or depth of interaction.
3. Adoption and Retention. These metrics explore behavior of unique users more in detail, going a step beyond the seven-day active users metric.  Adoption metrics track new users starting within a given period (e.g., number of new accounts opened this month), while retention looks at how many of the unique users from the initial period are using the product at a later period.
4. Task Success.  Successful completion of key tasks is a well-known behavioral metric that relates to efficiency (time to complete at task) and effectiveness (percent of tasks completed).   This is commonly tracked on a small-scale through one-on-one usability tests, but can be expanded to web applications by seeing how closely users follow an optimal path to completion (assuming one exists), or by using A/B split or multivariate testing.

But these metrics are not helpful on their own.  They must be developed in the context of the Goals of the product or feature, and related Signals that will indicate when the goal has been met.  The authors admit that this is perhaps the hardest part of defining success, since different stakeholders may disagree about project goals, requiring a consensus-building exercise.

From my perspective, there is also the additional challenge of clients having both the forethought and resources available to track these metrics in the first place.  In many cases, measuring success requires a benchmark or baseline for comparison.  Without this in place, the new design itself must serve as a benchmark for any future changes.

As I prepare for a week chock-full of usability testing, I’m remembering some of my favorite analogies for the role of a usability study facilitator.

Carolyn Snyder, in her excellent book, Paper Prototyping, describes three roles that must be fulfilled simultaneously:

Flight attendant: keep the test participant safe and comfortable. (Although, with airline seats getting smaller and job satisfaction for flight attendants decreasing, I’d have to say this may set too low a standard.)

Sportscaster: ensure the observers understand what’s going on.

Scientist: gather data accurately while minimizing bias.

But my favorite analogy comes from Joe Dumas, who taught me much of what I know about testing. Joe said a usability test facilitator is like a duck – you must appear calm and placid, but underneath the surface you’re paddling like crazy.

The article references studies performed in a grocery store (choosing a jar of jam) and a workplace (choosing a retirement savings plan) to illustrate its point, but it struck me that this applies to online experiences, too.  A web site with a lot of content that offers too many options that are poorly organized will lead to frustrated users who will abandon your site for your competitor’s.

So how do we– marketers or user experience (UX) practitioners–craft a better choosing experience?  Authors Sheen Iyengar and Kanika Agrawal offer the following tips, to which I’ve added my UX take:

1. Cut the number of options.

Don’t worry about losing shelf space to competitors–in the end, trimming back the product line lowers costs, increases sales, and makes it easier for consumers to choose. According to the authors, “In case the poor performers aren’t evident from sales figures, focus groups and online networks can help you separate the wheat from the chaff.” My UX take: Focus on presenting the content and tasks that mean the most to your users. A combination of web analytics and user feedback (interviews or surveys) will help you figure out what should be on the site and what’s expendable.

2. Create confidence with expert or personalized recommendations.

Expert reviews and recommendations “let consumers skip over much of the information-processing component of choosing, minimizing cognitive stress and enabling them to make good choices,” according to Iyengar and Agrawal. My UX take: This tip speaks directly to the online experience. Many web sites offering highly differentiated items (books, music, clothes) benefit from recommendation tools, or automated systems (“electronic agents”) that generate suggestions based on consumers’ expressed preferences. While these tools require more of an investment on the part of the organization and sometimes the user (e.g., if a survey or profile needs to be completed), they can be worth it if your web site is one that offers a large quantity of content or inventory to peruse.

3. Categorize your offerings so that consumers better understand their options.

“For an expert, there is no completely unique product or service; rather, each offering is a distinctive combination of attributes that the expert has seen before.” The key is getting a novice to act like an expert by creating top-level categories that are easily understood. As an example, the authors cite wine retailer Best Cellars, which limits its varieties to 100 wines that are divided into eight top-level categories, such as “fizzy,” “juicy,” and “sweet.” Once the novice has chosen a category, he or she can choose a wine within that category by reading the detailed labels that accompany all the bottles. My UX take: For web site users that rely on browsing to find what they want, category names are critical. This means avoiding terminology that is either organization-centric (“Initiatives”) or vague (“Solutions”) and using what makes the most sense to users.

4. Condition consumers by gradually introducing them to more-complex choices.

“For certain kinds of decisions, you can set up consumers for success by encouraging them to learn from, and build upon, their own previous choices.” Iyengar cites a study in which two groups of car customers were asked to customize their vehicles, choosing everything from the engine to the rearview mirror. The first group started by choosing features with a high number of options, moving to those with low numbers of options. The second group started by making choices for features with a low number of options first. In the end, the first group had a less satisfying experience: “They began by carefully considering every option, but they soon grew tired and settled for the default. In the end, they wound up less satisfied with their cars than the buyers who had progressed from low choice to high choice.” My UX take: Users can go through a lot of information online, provided it’s presented to them in a way that lets them process it in logical bite-sized pieces. This means creating an information architecture that uses categories that make sense to the intended audiences, a hierarchical structure that lets users drill down and expose more information as they need it, and a supporting design that visually prioritizes information on each page.

Iyengar and Agrawal acknowledge the dilemma: “Don’t marketers have to give consumers what they want? Yes and no. We should give them what they really want, not what they say they want…They want to feel confident of their preferences and competent during the choosing process; they want to trust and enjoy their choices, not question them.” The online experience should work the same way.

You can view the web site at www.historicnewengland.org or dive right into searching and browsing the online collections — full of photos, artifacts, and reference materials having to do with 400 years of New England History.

We’re currently in the final stage of the project, conducting usability tests on the new site.

Racing with the Clock: VERY Rapid Design and Testing

Presenter: Will Schroeder of The MathWorks

Summary: Will’s premise is that in design, as in psychotherapy, the most important part of any hour is the last five minutes. So he sought to eliminate the first 45 minutes (an hour of therapy is only 50 minutes, as you may recall from the old Bob Newhart show). Will described a 2-hour design process that allowed a team of 12 people to create three parallel design concepts, review and iterate on them, and usability test them, with a successful outcome. My favorite quote from Will’s talk was, “Brainstorming is so much fun, I’m surprised it’s still legal.” Another key point was the need for show and tell: “You don’t understand [a design] until you explain it.”

The Power of Focus Groups in Design Research

Presenter: Kay Corry Aubrey of Usability Resources

Summary: Focus groups (essentially, group interviews) can be an effective way to gather qualitative data on perceptions, opinions, beliefs, and attitudes. Examples of how focus groups can inform the design process include:

• Learning about your users’ decision making process, needs, and pain points
• Determining questions for a survey or content for a card sorting exercise
• Gathering content and feature requirements

Important elements for a successful focus group include careful planning and recruiting the right participants. A skilled focus group moderator must be able to establish trust, ask good questions, listen actively, remain neutral, and manage group dynamics.

This was an excellent overview or refresher, especially for recruiting and moderating.

More info: Kay’s slides are posted on Slideshare.

Will and Kay both deserve kudos for making their slides readable. You’d think that would be expected for a bunch of usability professionals, but at least half of the presenters had slides that were illegible both in the room and in the conference proceedings.

First of all, the increasing size of the conference (450 attendees, 32 presentations in 4 simultaneous tracks this year) reflects the astounding growth of the usability profession. These are the people who conduct user research, design and evaluate interfaces to ensure they provide an effective, efficient, and satisfying experience for users. Better user experiences increase productivity, reduce costs, and increase market share for companies and organizations that use and sell technology.

I noticed some interesting trends in conference technology and culture. A few years ago, most conference attendees toted their laptops along to take notes and keep in touch with the office or clients via email. This year, I saw very few laptops — instead, nearly everyone had smartphones and similar mobile devices. I even saw an iPad or two, typically with hangers-on eyeballing the device with jealousy or skepticism.

Twitter was a little less visible this year. Last year, a twitter feed displayed conference-related tweets on a large screen for all to see. Arguments ensued (over Twitter, natch) about whether it was rude to tweet during presentations. This year the twitter feeds were no less active (see #upaboston and #miniupa), but they were not projected. Toward the end of the day, it was fun to see the final tweets about dying batteries in the aforementioned mobile devices. I’m happy to report my rollerball pen made it all the way to 6:00 without needing a recharge.

My favorite presentation of the day was Lynn Cherny’s Mining Your Data: An Easy Intro to a Tough Topic. Lynn discussed and demonstrated several methods for analyzing qualitative data — such as the answers to open-ended survey questions — and turning messy text data into numeric data for further analysis. Tools included:

• Excel’s convert text to columns feature, pivot tables, and sparkline plug-ins
• R (open source statistics software) for more sophisticated methods such as cluster analysis
• Linux command line tools (e.g., grep) for manipulating and exploring text data across multiple files
• Wordles, Many Eyes, and Concordance software for further text analysis

She inspired me to finally learn to use pivot tables — something I’ve been meaning to do for years. What a time-saver. Contact Lynn at Ghostweather for a copy of her presentation.

Watch this space for more presentation summaries.

To explain the importance of the concept, author Haley Sweetland Edwards turns to designer Eliazar Parra Cardenas, creator of Backbars, “a GreaseMonkey script to turn the headlines and comments of social link-sites into ambient bar charts (of votes/diggs/views/users…).” Cardenas explains:

“The whole point is to make textual information easier to absorb… [A well-designed site] should maximize the information that a user can understand — that you can just glance at, or take note of -– without actively thinking….

“We’ve already tried the obvious in print: putting as much text as possible in one glance (hence broadsheets), mixing in images, headlines, columns. I think the next step will be digital developments like backbars, favicons, sparklines, word coloring, spacings.”

Count me as extremely skeptical. The sites that Edwards and Cardenas hold up as examples seem both cluttered and shallow — a vote-stuffing contest for “news of the weird.”

I’m old school that way. What drives traffic are the editorial and authorial inputs that Cardenas overlooks in his list of the obvious. Not headlines, but well-written headlines. Not images, but compelling images. Not backbars, favicons, sparklines, word coloring, and spacings, but good ledes.

The New York Times isn’t making money online. But they aren’t lacking for traffic.

The chart, of Federal Spending FY 2009 YTD, is from USAspending.gov, a web site mandated by law to provide the public free, searchable information about U.S. Federal expenditures.

Seth Grimes at Intelligent Enterprise figures out the problem and its cause:

USAspending.gov produces its charts dynamically using the Google Chart API…[but] passes values to Google that are out of range. Google truncates them, just as [its] documentation explains.

Here is Grimes’ corrected chart:

Unfortunately, data misrepresentation isn’t the only problem he finds.