The iPhone User Experience: A First Look

A collective gasp was heard around the world following the January, 2007, MacWorld Conference, when Steve Jobs pulled the wraps off the long-rumored iPhone. He proclaimed it a revolutionary product with a brand-new “multi-touch” interface as breakthrough and breathtaking as the mouse interface of the 1960s.

Is iPhone as revolutionary as claimed? Is the multi-touch interface truly breakthrough as claimed? Yes and no. Let’s take a look.

Who’s talking?

Bruce Tognazzini was hired at Apple by Steve Jobs and Jef Raskin in 1978, where he remained for 14 years, founding the Apple Human Interface Group. He has been a harsh critic of many of Apple’s later innovations, including the notorious round mouse (“farcical”) and the Macintosh Dock (see: Top 10 Reasons the Apple Dock Sucks). He is almost as stingy with his compliments as his partner, Don Norman. That makes this particular column, largely positive, most unusual.

A Brief History of Cell Phones

The original cell phones made one single break with the interface of the wired phones that had come before: The user dialed, then pressed Send, instead of dialing “live” as had been done historically. That’s it. Added later were such niceties as keyboards for message and email construction, borrowed whole, again, from the wired world. (Even the Send button was borrowed from earlier Radio-Telephone technology.)

Occasionally, bits and pieces of interface innovation have found their way into subsequent cell phones, but no one has ever revisited Bell Lab’s pushbutton phone design from the 1950s with its upside-down adding machine keyboard (with the exception of Smart Phones, based on the 1870s typewriter keyboard).

iPhone is revolutionary, not a big surprise coming from Steve Jobs. He knows how to gather a tiny team of brilliant young minds and work them half to death until they innovate beyond any reasonable expectations. He has the common sense to know what will ultimately find favor. And he has the hardened-steel man parts to take a chance and roll with it. What’s a pity is that so few others in this industry share those triple strengths.

Multi-Touch Interface History

While the iPhone as a whole may be revolutionary, the individual elements forming the interface are not so new. Bill Buxton was pushing multi-hand input back in the 1980s when the world was just waking up to the mouse, already 20 years old at the time. Several researchers were experimenting with gestural interfaces in 1990s, myself included. I was reminded of this only minutes after Steve’s speech when my partner, Jakob Nielsen, called me to say, “Jobs just announced your pinch interface!”

I hadn't thought about my particular use of the pinch gesture in years. It had been part of my Starfire Project at Sun Microsystems, a look at the future, but, when we turned the project into a film, the scene showing it was cut to keep the film within reasonable bounds. That kept it out of the 1993 film, but not out of my 1996 book, Tog on Software Design, where, on page 78, my two-fingered shrink-objects-via-pinching gesture, working exactly as Jobs described, indeed appears...

(There was nothing magical about the “20 percent” in the above text. That was just how much, in this case, “Hiroshi” was going to need to shrink the object to get the effect he wanted. You could, of course, shrink and grow objects as much as you wanted, depending on how much you pinched or spread your fingers, just as with the iPhone.)

I didn’t try to patent the object-shrink-pinch process in 1992 when I hit upon it—Sun didn’t get “patent madness” until more than a year after I had showed it around—so I can’t claim that I was even the first to come up with it because we never researched prior art.

Bill Buxton has assembled an excellent Multitouch Chronology. It reveals that Myron Krueger was using pinching as a gesture by 1982, although I haven't discovered what meaning he ascribed to it. (Myron Krueger is famous for coining the term, "artificial reality," back in 1973, a scant five years after Timothy Leary had popularized one means of achieving it.)

Innovation of the Whole

I could go down through the other “innovations” in iPhone and slowly knock them off. Yes, it’s the first cell phone with a visual display of voicemail messages, so you can randomly move among voicemails, etc., etc. However, such lists have been displayed, in an identical fashion, on enterprise-level voicemail systems and, of course, such lists have been a standard feature in email for decades.

The origins of these bits and pieces, however, is not what’s important about the iPhone. What’s important is that, for the first time, so many great ideas and processes have been assembled in one device, iterated until they squeak, and made accessible to normal human beings. That’s the genius of Steve Jobs; that’s the genius of Apple.

It’s also speaks to the limited vision of the cell phone industry. Exactly why have we never had random-access voicemail on cell phones? We’re talking about hand-held devices with more computer power than the Apollo spacecraft that took us to the moon. We’re talking about devices with screens of more than sufficient resolution. Could nobody think of displaying the messages?

The Macintosh computer did not represent a technological breakthrough either. The mouse was already 20 years old. Pointing interfaces were 20 years old. The Mac was a direct, studied “rip-off” of Apple’s expensive Lisa computer, developed concurrently, but shipped a little over a year earlier. That detracted nothing from the genius of the Mac, for what that team did was to take highly innovative technology and make it (relatively) inexpensive, attractive, and accessible.

That’s exactly what Apple has done again with iPhone. Multi-touch gestural interfaces have been hanging around in the laboratory, screaming for release, for as long as the mouse hung around. I’ve been pushing multi-touch gestural for over 20 years myself, beginning while I was still at Apple, incredulous that everyone has been ignoring it. Apple stopped ignoring it.

Fulton didn’t invent the steamboat. He just put in the hard work to make it practical. Apple didn’t invent the concept of the multi-touch interface. They’ve just, by all evidence, built the first one that, like the Mac before it, is (relatively) inexpensive, attractive, and accessible.

The iPhone User Experience

At the time I first published this, I hadn't yet been able to get my hands on an iPhone—frustrating! (You can imagine Bill Gates’s frustration. He probably had a cadre of engineers ready to take it apart, put it back together with a couple of screws missing, and paint it brown.) The following review was based on stuff on the web, including Apple’s demo site, as well as testimonials from those who had actually touched the precious object.

I got a chance to spend some time with the phone during the first 24 hours of its release. I found I had to change nothing about this review, although I did publish a follow-up with a few more suggestions for future improvement (The iPhone User Experience: A First Touch).

What strikes me about the iPhone interface in general is that it gives ordinary people access to features that have been the private purview of the young and the geeky. For example, cell phones have long had contact lists, but they were typically difficult to build, maintain, and sync.

My own Motorola phone on the Verizon network required a special cord that cost me an hour of research on the web to even discover, along with several software tricks to finally get it communicating with my Mac. I have never tried to load more than a tiny fraction of my contacts anyway, since scrolling is so laborious and resyncing is highly cumbersome.

Young vs. Old

The best user interfaces are those that bring advanced features within the easy reach of inexperienced users. Such was the genius of the SRI-Xerox-Apple interface embodied in the Mac, and such is the genius of the iPhone interface. As with other smart phones, Apple starts with a full QWERTY key layout. (You may find it reasonable to spell my name, Tog, “86664,” but old folks don’t. They can’t see the little letters beside the numbers to begin with, and they don’t want to learn.) Then, Apple has made an effort to make normally abstract features, such as Call Waiting and Conferencing, not only attractive, but dead simple.

I’ve read several reviews of the iPhone, primarily from England, talking about how the iPhone may be exciting to the young, but will fall on deaf older ears. They said the same thing about the Mac, yet one of the earliest, most rapidly-growing communities was people over 65. The same may well happen here because this is the most approachable full-featured phone I’ve ever seen. Apple will have to take some steps, however, to reduce the learning curve inherent in it's initial typing scheme to capture that market. (See the Touch Interface: Room for Improvement section of The iPhone User Experience: A First Touch.)

The Hardware

The industrial design is brilliant. Apple has created another piece of high-tech jewelry. Some fogies of advancing years have suggested the initial price point of $499 is too high. They fail to understand: The “cool” of owning this phone, particularly for the early adopters, is worth an easy $497, bringing the phone itself down to $2 even.

For those who might doubt such a high value of cool, consider the self-winding Rolex, which sports 1/10th the accuracy of a Timex at 1000 times the price. With Rolex, the technology is grossly inferior, and still people will pay thousands to own it. With the iPhone, the technology is clearly superior.

The form factor, at only a silly millimeter higher and wider than the Palm Treo, but half its thickness, is an excellent compromise of screen size and handset width. The glistening black color, surrounded by chrome, is pure sex.

The screen itself is high-density, and the lack of a physical keyboard allowed the designers to make it bigger than other smart phone displays. The 320 x 480 size is equal to the practical limits of NTSC television. Yes, NTSC, not HDTV, but we’re taking about a small, highly-portable device here and, unless you wore magnifying glasses and held it up three inches from your face, you would be hard-pressed to actually appreciate any additional resolution.

Where does the hardware fall down? Its biggest drawback is its sealed-in battery. It can play video for six hours on a charge and, while that's sufficient for entertainment on domestic flights, it falls quite a bit short for international flights. "OK, so just change the battery!" Unfortunately, it requires a crowbar and a soldering iron to change the battery. That’s a bad interface.

Some folks, including Eugenia Loli-Queru, have taken me to task for the above, based on the probable availability of snap-on external batteries, airplane power cords, and the fact that most other devices are worse. These are all good points. Nonetheless, My favored video device, a PocketDish, is simplicity itself. When the first battery dies, I just slap in a spare battery pack, and I'm good to go. (Bonus: The same programs iTunes charge for are all free.)

Still, for most people, most of the time, the iPhone's battery life is sufficient.

• Talk time: Up to 8 hours
• Standby time: Up to 250 hours
• Internet use: Up to 6 hours
• Video playback: Up to 7 hours
• Audio playback: Up to 24 hours

Apple, under Jobs, has a habit of putting out hot-looking, but fragile devices. He shipped a few million titanium PowerBook computers a few years back, forgetting to note that titanium has a rather ugly color, so they’d dipped it in some silver paint. After a matter of months, the paint “chipped, flaked, and bubbled” off people’s machines. (In my case, it wore off selectively where my palms rested when using the keyboard.)

The first iPod Nano required exquisite care to keep from scratching it. This knocked the user experience down several notches below ideal.

The iPhone appears able to withstand everything up to a nuclear incident. PC World, no friend to Apple, carried out a torture test right after the iPhone's release, dropping it on concrete pavement and taking a car key and attempting to scratch the display by scraping it repeatedly across the surface. The iPhone took a licking and kept on ticking. Apple has learned from its mistakes.

The Interface, Step-by-Step

This discussion tracks the features as demo’d on Apple’ site at http://www.apple.com/iphone/ I have taken them slightly out of order with Phone before iPod, then Internet—to make this discussion flow more smoothly. Besides, it’s supposed to be a phone, so with Phone we shall begin.

iPhone as a Phone

Calls

iPhone is able to handle perhaps 10x the number of contacts of the average cell phone without undue burden on the user. The acceleration algorithm, coupled with the “friction” algorithm, appear to let the user “throw” the list with learned speed, then have it slow on its own as it nears the target, making it simple to hit the target on the first try. Very nice.

Those of you young and technologically inclined may find this difficult to believe, but the average cell phone user cannot use many features you may find standard, such as call-waiting, call-forwarding, and conferencing. Apple has made these features completely accessible to all but those dangling their legs off the far end of the bell shaped curve.

Voicemail

Random-access voicemail I’ve already commented on. To reiterate my position: It’s about time.

SMS

SMS is a wonderful feature if you’re in Europe, where the cell phone companies don’t gouge you for using it. Notwithstanding its financial handicapping in the US, Apple has made it clear and accessible, again vastly enlarging the number of people who would feel comfortable and confident using it. This impacts the young, too, since, for the first time, they will be able to text-message their parents, knowing they’ll not only notice it, but be able to read it.

The SMS interface should include an interpreter that can expand standard messaging abbreviations, converting terms such as GBTM into “Get back to me.” This is required if people are to communicate across generations without undue burden. One option would be to have the system highlight such contractions, revealing the meaning when they are touched. This would facilitate learning.

Even better, as one of those who lives in Europe and uses SMS quite a bit, would be the reverse to help keep SMS messages with in the 160 alphanumeric characters limit; thus converting “Get back to me.” to GBTM when sending and by receiving doing what you have suggested.

Excellent idea! This would require a fairly substantial synonym list for those unfamiliar with standard SMS abbreviations, so one could type, "respond to me..." and still have it contracted to GBTM. It is also a good argument for having the system hightlight, rather than auto-expand contractions, so that users will quickly learn the new jargon on their own. (The problem today is not that people have to learn, but that they have to learn so much all-at-once. It is the steepness of the learning curve that keeps the "old folks" at bay.)

Photos

The photo interface will certainly improve upon what has come before, but not by that much. Applications on the Palm are not as elegant, but can handle the same approximate number of images before getting cumbersome. Apple’s interface, however, is smoother, sleeker, and certainly sexier. Having the orientation-sensor is a big win, as well: Shifting the orientation on a palmtop device picture-by-picture is intolerable. (I have two indexes of photos on my Palm: Those that are horizontal and those that are vertical.)

Orientation-sensing is, again, not new: Canon, for example, has used it in its cameras for years so that the photos “know” whether they are intended to be viewed horizontally or vertically, based on the orientation of the camera at the time of shooting. Again, Apple has put it to a new and valuable use.

The 2-megapixel camera, of course, is pretty much so they can say they have one, although it exceeds the resolution of many other smart phones. It needs to be 5-megapixel minimum and, in keeping with the rest of the interface, it would be nice if it “knew,” by means of GPS or other location technology, where it is, so the resulting photos could be location-labeled. Facial recognition would be a plus.

It's the quality of the lens rather than pixels that make the difference from what I have seen. My 3+ megapixel Canon Powershot is darn good. The Razr with about 1.3 megapixel is worthless. So, need to see what the iPhone actually does.

Points well-taken. The so called LED "flash" that many phones offer I also find worthless, outputting perhaps 1000th the peak light of a "real" flash and over way too long a duration.

iPhone as iPod

Music

The contrast between the Phone interfaces collectively and iPod Music interface specifically on the machine is interesting. The Phone functions have been kept clean and straightforward, accessible to all, no matter how technologically backward. The apparent goal has been to fully support current users of special features, while embracing new legions of users put off by fancy stuff like Call Waiting. Not so the iPod music features. Here, Apple has turned up the “cool,” even though accompanied with a far higher level of complexity than in the Phone functions. Here, for the first time, there are multiple and, to those of advanced years, like, say, 30, potentially confusing ways of accessing the same information.

This again demonstrates the genius of the development team: They have built appropriate interfaces for each class of expected user. It’s well known that old people don’t listen to rock music 23.5 hours a day. Why make them your design center?

Video

The interface is clean, simple, and tracks that found in other areas such as voicemail and contact lists. Again, the orientation-sensor makes it easy to watch films as intended.

What’s more interesting here is one of the fundamental principles of interface at work: The best interface in the world will not help a project that is technologically deficient, sometimes referred to as “putting lipstick on the pig.” Here, no lipstick is needed. The screen, as mentioned earlier, approximates the effective resolution of NTSC, comfortable enough to watch a standard TV show as well as movies other than special-effects fests, such as Star Wars, where nothing short of a six-foot HDTV will really do.

iPhone as Internet device

Public Wi-Fi is a mess: As you move from place to place, everyone has their hand out, trying to squeeze several thousand dollars per year out of you for the precious use of their bandwidth. (Of course, it’s priced by the hour, but do the multiplication to see how avaricious they really are.) Unlike the cell phone networks that quickly knitted together into a few nationwide carriers, Wi-Fi remains badly fragmented, making monthly or annual subscriptions impractical. Most of us with Wi-Fi enabled devices have learned just not to use email while en route or to fly Jet Blue, where they offer free airport access.

iPhone is ambidextrous, able to connect to the Internet via Wi-Fi or AT&T’s Edge network. It’s not enough. AT&T is a “cherry-picking” network, specifically aimed at the largest population clusters with big gaps in between. Edge is also spotty in Europe, leaving only Wi-Fi as a means to connect at high-speed in those areas not supporting it. In the short term, iPhone will be an Internet device that sometimes works and sometimes doesn’t, costing you a pretty penny for an intermittent connection unless you stay within the large population centers of the USA at all times.

Fundamental unreliability is a big hit, and any excuses that claim “the others are just as bad” should fall on deaf ears: After all, this is the phone that Apple proclaims is revolutionary. It seems the revolution can only truly begin when the phone’s features work virtually everywhere.

(OK, OK, I’m biased. I am owned and operated by Verizon because they truly do have the network—I left AT&T because they didn’t—yet I want an iPhone. So shoot me. Or, better yet, call me in 2009 when Apple's “exclusive contract” with AT&T comes to an end and tell me I can buy an iPhone now.)

Safari Browser

Apple has managed to make it practical to view standard web pages on a 3.5 inch screen. I’ve thought from the beginning that the drastic compromises being made to wedge reduced-content web pages into current handheld devices were an interaction dead end, and I couldn’t be happier with the job Apple has done here. If Apple doesn’t carry over this technology into some kind of slate computer, they are not nearly as bright as I think they are.

Beyond the accelerative, frictional scroll, as used with the contacts list, the browser apparently has the “smarts” to enable the user to selectively pick a single article, not by carefully enlarging it to just fit, but by indicating it and having the browser temporarily strip away everything else. I have not yet explored to see if the production iPhone can work this trick on most pages, or whether the website must pay close attention to the letter of the HTML law in every regard to make it work. If the feature is forgiving, it is a home run.

Mail

email echoes the voicemail interface. It is clean and simple. What is startling is the apparent hard separation of email, SMS, and voicemail. What I would want is a single list, defaulting to the newest and unread/unheard first. I don’t care about the medium, and neither should iPhone.

Of all the iPhone features, this is the one that seems to have completely missed the target. It would be like Blackberry having three lists: One for mail with more than 100 characters, one for mail with fewer than 100 characters, and one for mail sent from more than 3000 miles away. What sense would that make? What Apple has done, when you think about it, is just as random in terms of the user.

I should be able to read a voicemail in my list, using speech-to-text, highlighting any word that isn’t clear to hear the original. I should then be able to write an instant message, having it route via Instant Message or SMS, whichever is available and cheapest. Having then discovered the person is not responding, I should be able to convert the message into an email, recognizing, since I know the person has Yahoo mail, it will be pushed and get his attention.

I should not have to visit three different places on my phone every few minutes to see what is happening.

Maps

Here again, the system apparently displays a lack of integration. The system should “know” where I am. So far, I have not heard or read anything that would make it appear it does. Mapping information should tie in with the scheduler, so that the system can figure out that I’m 1 block or 1000 miles from my next appointment and give me appropriate notice: “You have a half hour to your next appointment down the block” vs. “You have two days to get to New York City.”

Reader Asher Gabara points out that Maps does nicely integrate map results with Calls: One can Find a Starbucks location, for example, then click on it to place a phone call instantly to that location. While a nice touch, it rather shrinks in comparison with other phones that can then give you turn-by-turn guidance to actually get there. I don't fault either the designers or the phone in this. The first model of any product cannot be expected to have every possible feature. I just point out that Maps, right now, is primarily of value to those lacking the "real deal"—a GPS.

Widgets

These apparently are the only “add-on” path for iPhone right now. I can only hope that changes. They are simple toys, and their graphics-heavy representations, though pretty, rob valuable screen space with poor return-on-acreage.

Widget type applications could work very well on a mobile device. They are task specific and very easy to make. This makes them quite adept at a certain class of application and really lowers the bar for developing creative mobile applications which I think personally will fuel an explosion of much better mobile applications in the US and Europe. For example apps like a client for a web service or perhaps ordering a Pizza etc. However no real 3rd party apps is really disappointing both personally and professionally.

Apple must provide mechanism for adding third-party apps if the iPhone is to survive in the long run. Meanwhile, I still think that Widgets are too compartmentalized. I'd rather see a Services architecture, that would allow plug-in capabilities. For example, let me select "Nearby Take-Out Restaurants" from the map, select a pizza joint, and have a menu-order screen arise from within Map itself, rather than having to find the pizza place using Map, then finding and launching another object, a Widget, to place the order. In either case, third parties could be included; in the latter, the result would be more integrated.

Toward the Future

iPhone is missing something important—a keyboard. The touch keyboard appears quite practical for a device sized like this for doing short messaging and such, but I wouldn’t want to write this whole column with it, particularly when the keyboard is obstructing half the screen. We can only hope that someone will be permitted quickly interface one to it, along with a suite of Microsoft Office-compatible applications.

It's missing something else, too, a way, when traveling, to output a high-resolution digital signal to a standard conference-room projector. Yes, yes, I know: It’s supposed to be a phone. However, it now does so much of what I drag around a laptop for when traveling, why can’t it do the rest? One thing left out and I have to drag around both.

Reader Zach White points out that pin-outs in the iPhone connector may allow third parties to add such niceties as video out and a snap-on battery. No doubt the large community of iPod add-on manufacturers are salivating even more than potential iPhone owners at the prospect of the coming feast. They will, undoubtable address each and every possible shortcoming of the iPhone that Apple permits them to. The one possible "fly in the ointment" is the locked-down nature of the iPhone software: Apple and its partner, AT&T, must approve every software addition, and how willing they will be to support the add-on community waits to be seen. Battery packs won't need software support; other add-ons well might.

There’s lots to do tomorrow, not only in expanding the capabilities of iPhone, but in more tightly integrating the features already offered, such as the three forms of mail—voicemail, email, and SMS/instant messaging. They also need to get serious about integrating mapping with other features, ensuring location-awareness and making use of that information, etc.

However, while this first iPhone may not be the be-all and end-all, neither was the original Mac. With its 128k of memory, 8 MHz processor, and primitive, inflexible interface, users would be hard-pressed to get any real work done today. Nonetheless, the original Mac was a magnificent start, and iPhone is likewise.

The Mac languished for years because Steve Jobs left Apple and the management who remained were sorely lacking in anything approaching vision. One can only assume Jobs won’t be leaving this time around. If he moves iPhone along as he would have moved Mac along during those “dark years,” the iPhone we look upon with such wonder today will seem like a simple toy three, five years from now.

Traditional cell phones are dull, limited, and at end-of-life. iPhone is glorious, and it is only the beginning.

Websites that Sell

People have been bugging me for years to do an advanced human-computer interaction course, so now I am. "Websites that

Sell" combines my 30 years of interaction design experience with my 15 years of retail sales experience. In one day, I'll show you how to turn an ailing site into a site that not only stops driving customers away, but actively closes the deal. Limited availability. Act now!

Seriously, we were promised at the beginning of the web that, soon, "bricks and mortar" stores would be a thing of the past. Then, the bubble burst, popped by the poor web sales that actually resulted. We deserved what happened: Most websites out there even today are doing their very best to drive customers away. Fortunately, the original projections were not that far off the mark, and many sites that are simply limping along now could be doing a land-office business if they only knew how to sell. I do, and I can teach you.

This is a course in "selling through design" not "marketing through designer," because marketing is not the same as sales. Sales is about keeping the customer informed and happy while successfully closing the deal. You will walk away knowing exactly how to do that.

I sold consumer electronics "live and in person" for 15 years before the invention of the personal computer, moving millions of dollars worth of product, picking up awards along the way. I also taught sales theory and practice to other salespeople, and I've been passing on these secrets to my private clients for years. Now it's your turn.

Most designers shrink from anything that might lead the web visitor to believe we are actually interested in their buying anything, believing that selling is unethical. A lot of selling is. I don't teach anything that will make you feel anything but good. I sold with an eye for future business, and anything even remotely underhanded is antithetical to future business.

Instead, I will teach you how to take care of your customer from the time they walk into your "web store" until the time they leave, credit card back in their wallet, merchandise on its way, and a smile on their face, the kind of customer experience offered by stores like Nordstrom in the real world.

This course is not a "rah, rah" session. Instead, I will teach you fundamental principles and psychological theory that can be applied directly to on-line sales, along with a lot of practical examples of what can go wrong and what to do about it.

While I'm aiming the course primarily at interaction designers, managers and marketing people have found the course useful and immediately applicable.

While the thrust is web sales, both retail and wholesale, it is, at heart, about motivating visitors and ensuring their success. Those working on sites from government to intranet, will find much of immediate, practical use.

Come join me for a whole new way of looking at "customer interaction" design. It could be the most valuable course you ever take!

The iPhone User Experience: A First Look

Who’s talking?

A Brief History of Cell Phones

Multi-Touch Interface History

Innovation of the Whole