A brief history of touchscreen technology

1980s: The decade of touch

One of the first diagrams depicting multitouch input.

Bill Buxton

In 1982, the first human-controlled multitouch device was developed at the University of Toronto by Nimish Mehta. It wasn't so much a touchscreen as it was a touch-tablet. The Input Research Group at the university figured out that a frosted-glass panel with a camera behind it could detect action as it recognized the different "black spots" showing up on-screen. Bill Buxton has played a huge role in the development of multitouch technology (most notably with the PortfolioWall, to be discussed a bit later), and he deemed Mehta's invention important enough to include in his informal timeline of computer input devices:

The touch surface was a translucent plastic filter mounted over a sheet of glass, side-lit by a fluorescent lamp. A video camera was mounted below the touch surface, and optically captured the shadows that appeared on the translucent filter. (A mirror in the housing was used to extend the optical path.) The output of the camera was digitized and fed into a signal processor for analysis.

Shortly thereafter, gestural interaction was introduced by Myron Krueger, an American computer artist who developed an optical system that could track hand movements. Krueger introduced Video Place (later called Video Desk) in 1983, though he'd been working on the system since the late 1970s. It used projectors and video cameras to track hands, fingers, and the people they belonged to. Unlike multitouch, it wasn't entirely aware of who or what was touching, though the software could react to different poses. The display depicted what looked like shadows in a simulated space.

Bill Buxton introduces the PortfolioWall and details some of its abilities.

Though it wasn't technically touch-based—it relied on "dwell time" before it would execute an action—Buxton regards it as one of the technologies that "'wrote the book' in terms of unencumbered… rich gestural interaction. The work was more than a decade ahead of its time and was hugely influential, yet not as acknowledged as it should be." Krueger also pioneered virtual reality and interactive art later on in his career.

A diagram (in Spanish!) detailing how the Video Place worked.

Touchscreens began being heavily commercialized at the beginning of the 1980s. HP (then still formally known as Hewlett-Packard) tossed its hat in with the HP-150 in September of 1983. The computer used MS-DOS and featured a 9-inch Sony CRT surrounded by infrared (IR) emitters and detectors that could sense where the user's finger came down on the screen. The system cost about $2,795, but it was not immediately embraced because it had some usability issues. For instance, poking at the screen would in turn block other IR rays that could tell the computer where the finger was pointing. This resulted in what some called "Gorilla Arm," referring to muscle fatigue that came from a user sticking his or her hand out for so long.

Enlarge / The HP-150 featured MS-DOS and a 9-inch touchscreen Sony CRT.

Wikimedia Commons

A year later, multitouch technology took a step forward when Bob Boie of Bell Labs developed the first transparent multitouch screen overlay. As Ars wrote last year:

...the first multitouch screen was developed at Bell Labs in 1984. [Bill Buxton] reports that the screen, created by Bob Boie, "used a transparent capacitive array of touch sensors overlaid on a CRT." It allowed the user to "manipulate graphical objects with fingers with excellent response time."

The discovery helped create the multitouch technology that we use today in tablets and smartphones.

1990s: Touchscreens for everyone!

IBM's Simon Personal Communicator: big handset, big screen, and a stylus for touch input.

Android Authority

In 1993, IBM and BellSouth teamed up to launch the Simon Personal Communicator, one of the first cellphones with touchscreen technology. It featured paging capabilities, an e-mail and calendar application, an appointment schedule, an address book, a calculator, and a pen-based sketchpad. It also had a resistive touchscreen that required the use of a stylus to navigate through menus and to input data.

The original MessagePad 100.

Apple also launched a touchscreen PDA device that year: the Newton PDA. Though the Newton platform had begun in 1987, the MessagePad was the first in the series of devices from Apple to use the platform. As Time notes, Apple's CEO at the time, John Sculley, actually coined the term "PDA" (or "personal digital assistant"). Like IBM's Simon Personal Communicator, the MessagePad 100 featured handwriting recognition software and was controlled with a stylus.

Early reviews of the MessagePad focused on its useful features. Once it got into the hands of consumers, however, its shortcomings became more apparent. The handwriting recognition software didn't work too well, and the Newton didn't sell that many units. That didn't stop Apple, though; the company made the Newton for six more years, ending with the MP2000.

The first Palm Pilot.

Wikimedia Commons

Three years later, Palm Computing followed suit with its own PDA, dubbed the Pilot. It was the first of the company's many generations of personal digital assistants. Like the other touchscreen gadgets that preceded it, the Palm 1000 and Pilot 5000 required the use of a stylus.

Palm's PDA gadget had a bit more success than IBM and Apple's offerings. Its name soon became synonymous with the word "business," helped in part by the fact that its handwriting recognition software worked very well. Users used what Palm called "Graffiti" to input text, numbers, and other characters. It was simple to learn and mimicked how a person writes on a piece of paper. It was eventually implemented over to the Apple Newton platform.

PDA-type devices didn't necessarily feature the finger-to-screen type of touchscreens that we're used to today, but consumer adoption convinced the companies that there was enough interest in owning this type of device.

Near the end of the decade, University of Delaware graduate student Wayne Westerman published a doctoral dissertation entitled "Hand Tracking, Finger Identification, and Chordic Manipulation on a Multi-Touch Surface." The paper detailed the mechanisms behind what we know today as multitouch capacitive technology, which has gone on to become a staple feature in modern touchscreen-equipped devices.

The iGesture pad manufactured by FingerWorks.

Westerman and his faculty advisor, John Elias, eventually formed a company called FingerWorks. The group began producing a line of multitouch gesture-based products, including a gesture-based keyboard called the TouchStream. This helped those who were suffering from disabilities like repetitive strain injuries and other medical conditions. The iGesture Pad was also released that year, which allowed one-hand gesturing and maneuvering to control the screen. FingerWorks was eventually acquired by Apple in 2005, and many attribute technologies like the multitouch Trackpad or the iPhone's touchscreen to this acquisition.

2000s and beyond

With so many different technologies accumulating in the previous decades, the 2000s were the time for touchscreen technologies to really flourish. We won't cover too many specific devices here (more on those as this touchscreen series continues), but there were advancements during this decade that helped bring multitouch and gesture-based technology to the masses. The 2000s were also the era when touchscreens became the favorite tool for design collaboration.

2001: Alias|Wavefront's gesture-based PortfolioWall

Car Design News

As the new millennium approached, companies were pouring more resources into integrating touchscreen technology into their daily processes. 3D animators and designers were especially targeted with the advent of the PortfolioWall. This was a large-format touchscreen meant to be a dynamic version of the boards that design studios use to track projects. Though development started in 1999, the PortfolioWall was unveiled at SIGGRAPH in 2001 and was produced in part by a joint collaboration between General Motors and the team at Alias|Wavefront. Buxton, who now serves as principal research at Microsoft Research, was the chief scientist on the project. "We're tearing down the wall and changing the way people effectively communicate in the workplace and do business," hesaid back then. "PortfolioWall's gestural interface allows users to completely interact with a digital asset. Looking at images now easily become part of an everyday workflow."

Bill Buxton introduces the PortfolioWall and details some of it abilities.

The PortfolioWall used a simple, easy-to-use, gesture-based interface. It allowed users to inspect and maneuver images, animations, and 3D files with just their fingers. It was also easy to scale images, fetch 3D models, and play back video. A later version added sketch and text annotation, the ability to launch third-party applications, and a Maya-based 3D viewing tool to use panning, rotating, zooming, and viewing for 3D models. For the most part, the product was considered a digital corkboard for design-centric professions. It also cost a whopping $38,000 to get the whole set up installed—$3,000 for the presenter itself and $35,000 for the server.

The PortfolioWall allowed designers to display full-scale 3D models.

Car Design News

The PortfolioWall also addressed the fact that while traditional mediums like clay models and full-size drawings were still important to the design process, they were slowly being augmented by digital tools. The device included add-ons that virtually emulated those tangible mediums and served as a presentation tool for designers to show off their work in progress.

Another main draw of the PortfolioWall was its "awareness server," which helped facilitate collaboration across a network so that teams didn't have to be in the same room to review a project. Teams could have multiple walls in different spaces and still collaborate remotely.

The PortfolioWall was eventually laid to rest in 2008, but it was a prime example of how gestures interacting with the touchscreen could help control an entire operating system.