Multi-Touch touchscreen technology may be at issue in patent litigation. In such cases, a multi-touch touchscreen expert witness may be utilized. The evolutionary history of this technology may be an important fact during multi-touch touchscreen patent litigation.
Because of Android smartphones and the iPhone and iPod touch, nearly everyone knows about multi-touch user interfaces.
What would you do without the two-finger pinch-to-zoom gesture on a touchscreen(i)?
Multi-touch means that a touchscreen or tablet can register the positions of more than just one fingertip or touch at once.
It is easy to think that multi-touch came into the world in 2007 with the introduction of devices like the iPhone: that before, touchscreens and tablets could only register a single finger touch, like the touchscreen check-in kiosks found in airport terminals.
But multi-touch goes back much earlier than 2007: back before the days of hand-held PDAs with touchscreens in the late ’90s(ii) , before tablet computers in the early 90’s(iii) , even before the days of the earliest personal computers like the legendary Apple II in 1977.
As a development engineer with long-time experience with touchscreens and tablets serving as a touchscreen expert witness, I have been asked from time to time to review the history of touchscreens and tablets with (and without) multi-touch.
First, a few highlights from the history of “ordinary” touchscreens.
In 1973 and 1977, at the CERN particle accelerator center, Frank Beck and Bent Stumpe developed a transparent touchscreen to go over a display(iv) . The goal was to come up with a better control interface for the accelerator, where a less-confusing operator interface was very important for operating the accelerator.
Before that, in 1965 Eric Arthur Johnson at the Royal Radar Establishment in England built a touchscreen for air traffic control(v) . It used capacitive coupling, like most touchscreens in smartphones today, although it could only detect one touch at a time.
Although when you get down to it, you might trace touchscreens and tablets all the way back to Elisha Gray’s Telautograph in the 1890’s(vi) .
But just how long before 2007 did multi-touch happen?
Here are a few highlights of multi-touch touchscreen prior art:
Jeff Han made a splash — and gave a TED talk — demonstrating a large multi-touch display in 2005(vii) . His multi-touch technology sensed fingertips based on optics, specifically “frustrated total internal reflection” where light is reflected off a fingertip pressed on glass.
Before that, in 2002 Jun Rekimoto of Sony Computer Science Laboratories demonstrated multi-touch using what is now called “projected capacitance”, along with some impressive demonstrations of multi-touch gestures(viii) . I like this one in particular, because it was made with components like plywood and some stretched wires, along with electronics that could be built by a somewhat more advanced electronics hobbyist.
Before that, from about 2000 to 2005, a company called Fingerworks sold multi-touch devices with a broad variety of multi-touch gestures(ix) . These had quite a faithful following of users.
FingerWorks closed up shop in 2005, after the company was acquired by Apple.
Another multi-touch tablet — one that some people have cited as the first multi-touch tablet — dates from 1984(x) . It also used “projected capacitance” to sense multiple fingertips. Seonkyoo Lee was a graduate student at the University of Toronto, in a group that did research on bi-manual (two-handed) user interfaces, so the interest in creating a multi-touch device was understandable.
But before that, Nimish Mehta developed a multi-touch system in 1982(xi) , using optical tracking, not completely unlike Jeff Han’s system some 20 years later.
An even earlier one dates from 1976. A group led by Roy Kaplow (one of my thesis advisers at MIT) designed a multi-touch surface over a display(xii) . This was for a completely re-definable touch “keyboard”, where not only the keys/buttons/icons could be changed on the fly, but also the complete layout, for example making whatever touchable graphical images were desired.
…. And multi-touch gestures, like pinch-and-zoom?
(Before we get into that, note that pinch-and-zoom is not the only finger gesture used for continuously zooming in and out. For example, on the Nokia N900 smartphone you could zoom in and out continuously just by moving one finger in a circular “screw” motion in one direction or the other.)
If you look at Rekimoto’s video demonstrations from 2002(xiii) , you’ll see not only two-finger gestures to zoom and rotate, but also four (or more) fingers at once to zoom and rotate two things independently. Rekimoto also showed multi-touch gestures to move multiple objects by “herding” them along with your hands or even your arms on the tablet.
Before that, Tognazzini, a researcher at Sun Microsystems made a short film with actors in 1992(xiv) showing “a day in the life of a knowledge worker in the far-off distant year, 2004”. It showed a world of wireless networked tablets with built-in video, multi-touch user interfaces, and also a two-finger zoom and rotate gesture. The zoomable picture played an important part in the dramatic plot.
Another reference I find interesting is Krueger’s Videoplace . It combined live video images of the user (like shadows) with graphical objects the user could interact with in real-time. It shows two-handed gestures like the pinch-and-zoom we know today.
So, with so much that had been done before, just what is really new about multi-touch touchscreens and gestures today? Or more particularly, what things could be novel enough to be patentable today?
As patent practitioners know, it all depends on exactly what a patent’s claims say, and how that compares with the details of prior art. A qualified multi-touch or multi-touch gestures expert witness on touchscreens and tablets, and their long history, may be helpful in understanding the claims and knowing what the prior art might be.
About the Author: Jean Renard Ward is highly experienced, MIT-educated expert witness in patent litigation. Mr. Ward’s areas of design and development expertise include multi-touch/touchscreen and tablet hardware, capacitive touch and proximity sensors, styli/electronic pens, haptics; gestures, user interfaces (UIs), touchscreen graphics, and accessibility user interfaces (blind/visually-impaired); digital rights management (DRM), digital encryption and authentication (PKI), and malware detection; programming/coding (C/C++/Java, other systems), source-code analysis and reverse-engineering, and firmware. Clients include Google, Samsung, Ericsson, Lenovo, Motorola, Nokia, and Lucent Technologies. Mr. Ward has been Granted multiple US patents. He received his degree in Computer Science and Electrical Engineering Degree from M.I.T. Mr. Ward can be contacted at Rueters-Ward Services; Phone: (617) 600-4095; Cell: (781) 267-0156; Email:mailto:jrward@alum.mit.edu Website: www.ruetersward.com.
_____________________________________
i Picture: Apple: iPhone User Guide for iPhone OS 3.1 Software, 2009.
ii For example, US Robotics’ Palm Pilot 5000 PDA, released in 1996,
or the Apple’s Newton PDA released in 1993.
iii See Robert Carr, “The Power of PenPoint”, 1991, and
“Microsoft Windows for Pen Computing Programmer’s Reference”, 1992.
iv Bent Stumpe, “A new principle for an X-Y touchscreen”, CERN,16 March, 1977.
v E.A. Johnson, “Touch Display – A Novel Input/Output Device for Computers”,
Electronic Letters, October 1965.
vi Elisha Gray, U.S. Patent 386,815, “Telautograph”, July 31, 1888.
Jefferson Y. Han, “Low-Cost Multi-Touch Sensing through Frustrated Total Internal
Reflection”, UIST, October 2005. There are also video demonstrations uploaded
on YouTube.com.
viii Jun Rekimoto, “SmartSkin: An Infrastructure for Freehand Manipulation on Interactive
Surfaces”, CHI April 2002.
ix “About FingerWorks”, www.fingerworks.com, August 2000.
x Seonkyoo Lee, “A Fast Multiple-Touch-Sensitive Input Device”, U. Toronto, 1984.
xi Nimish Mehta, “A Flexible Machine Interface”, University of Toronto, 1982.
xii Roy Kaplow et al, “A computer-terminal, hardware/software system with enhanced
user input capabilities: the enhanced-input terminal system (EITS)”, SIGGRAPH ‘76.
xiii These have been uploaded on YouTube.com: search for “Rekimoto” and “SmartSkin”.
xiv Bruce Tognazzini, “STARFIRE: A Vision of Future Computing”, 1992.
xv Myron W. Krueger et al, “VIDEOPLACE: An Artificial Reality”, CHI, April 1985.
