$1.98, Close-coupled Usability Testing

Who would have thought my humble little treatise, Maximizing Windows, would draw such fire...

Hi Tog,After reading Maximizing Windows, on your website, an engineer I know responded as follows:

Isn't it nice to...

a) have wads of people to work for you to do umpteem gazillion iterations and complete full-blown usability testing; and
b) have so much slack in the project schedule to allow you to do all of this.

Tog has been blessed to be lucky enough to work on a project that does not have any schedule pressure, and to be at a company that has wads of cash to pay for a large UI team, complete with full-blown, repetitive usability testing.

Here in the real world, lots of compromises have to be made to get a huge project out the door in absolutely the shortest time possible. (Welcome to the project! You're deliverables are already late!)

So, just how "ideal" do you think your world at Healtheon/WebMD is? What, if any, schedule, budget, quality, resource, etc., pressures were you under?

Cheers,
Jeff Kroll, Manager, Design and Usability Group, Documentum, Inc.

It's a myth that user testing must involve lots of people, lots of time, and lots of money.

Last year, the Healtheon user interface team consisted of exactly four people--a prototyper-designer, a graphic designer, a market researcher/usability professional, and myself. We split our time across five projects with the same ship date.

The projects were completed, on time, in less than seven months. The resulting Member Services enables employees to sign up for benefits during their Open Enrollment period, which, for most companies is the month of November. This was not a schedule that could slip. Even one day would have been a disaster. Nonetheless, the design went through many, many iterations. Enough to bring it from a 15% overall success rate to close to 100%.

You can do the same thing in your company with the same lack of people, the same lack of time, and the same lack of money.

The People

Having a usability professional around to do the testing for you is a marvelous luxury, but you can do it yourself. (See: User Testing on the Cheap in Tog on Interface for step-by-step instructions.)

The Plant

We had quite the lab at Healtheon during the design and testing of Member Services. It consisted of an eight by ten-foot room furnished with two desks and a single computer. Toward the end, we installed a poor-man's one-way mirror--a cubicle partition wedged between two desks. The tester would peek around the cubicle wall to see what the user was up to. (Things have improved; in our new headquarters building, we now have a real lab with a real one-way mirror.)

The Process

Jeff said it straight in a separate letter: " You've got to iterate through the design and user test cycle a lot of times before you get things right." No matter how good a designer you are, a few all-but-intractable "gotchas" almost invariably are lurking. Sometimes they will fall to your first redesign. Sometimes they will seem to hang in there forever.

Close-coupled testing lets you do all the testing you want in the allotted time and get a successful product out of it. The technique is simple. Run a test subject through the product, figure out what's wrong, change it, and repeat until everything works. Using this technique, I've gone through seven design iterations in three-and-a-half days, testing in the morning, changing the prototype at noon, testing in the afternoon, and making more elaborate changes at night.

Close-coupled testing works. Particularly in the early stages, you are not looking for subtleties. You're lucky if a user can actually get from one end of the application to the other. Problems that may have escaped your attention before a user tried the product now are glaringly obvious once the first user sits down. So fix them! And then move on to the next layer of problems.

Management Support

The most critical ingredient to design success is management support. Even though user testing need not be elaborate, even though it measurably returns money to the corporate coffers, convincing the powers-that-be that they should support it can be an uphill battle.

We were lucky at Healtheon. Our management team believed from the beginning that both design and usability testing were important. They did not shower us with money; they didn't tell any of our customers that they really ought to change their open enrollment to "sometime in the spring." But they made it clear to the company that our design and testing efforts were important to the future of the company. And they installed an engineering methodology that allowed us to continue designing and testing until late in the development cycle, abandoning the traditional "waterfall" method.

If you don't user-test, you will be hearing a loud wail of complaint from your users and your customers. If you do user-test, you will be greeted with the welcome sound of silence, enough silence that you just might be able to catch up on all that sleep you missed.

-Tog

...and a response:

I found Maximizing Windows funny and sad. I'm tired of fixing interface bugs in netscape using graphics and javascript. Oh well...

As to anyone in this industry pissing and moaning about the difficulty and expense of doing usability testing, I'm a bit scared. All the features in the world don't mean jack if my average user is confused or needs tons of training. Me, I'd take MacWrite circa 1986 over word '97 any day, hands down...

Too bad no one can put a dollar amount on the sales and productivity loss due to bad designs.

Jonathan Peterson
Technical Director,
IBM Interactive Media