Touchscreen

"Touch sensitive" redirects here. For the album, see Touch Sensitive.

A touchscreen is an input device normally layered on the top of an electronic visual display of an information processing system. A user can give input or control the information processing system through simple or multi-touch gestures by touching the screen with a special stylus/pen and-or one or more fingers. Some touchscreens use ordinary or specially coated gloves to work while others use a special stylus/pen only. The user can use the touchscreen to react to what is displayed and to control how it is displayed; for example, zooming to increase the text size.

The touchscreen enables the user to interact directly with what is displayed, rather than using a mouse, touchpad, or any other intermediate device (other than a stylus, which is optional for most modern touchscreens).

Touchscreens are common in devices such as game consoles, personal computers, tablet computers, electronic voting machines, and smartphones. They can also be attached to computers or, as terminals, to networks. They also play a prominent role in the design of digital appliances such as personal digital assistants (PDAs)es and some books (E-books).

The popularity of smartphones, tablets, and many types of information appliances is driving the demand and acceptance of common touchscreens for portable and functional electronics. Touchscreens are found in the medical field and in heavy industry, as well as for automated teller machines (ATMs), and kiosks such as museum displays or room automation, where keyboard and mouse systems do not allow a suitably intuitive, rapid, or accurate interaction by the user with the display's content.

Historically, the touchscreen sensor and its accompanying controller-based firmware have been made available by a wide array of after-market system integrators, and not by display, chip, or motherboard manufacturers. Display manufacturers and chip manufacturers worldwide have acknowledged the trend toward acceptance of touchscreens as a highly desirable user interface component and have begun to integrate touchscreens into the fundamental design of their products.

History

The prototype x-y mutual capacitance touchscreen (left) developed at CERN in 1977 by Bent Stumpe, a Danish electronics engineer, for the control room of CERN’s accelerator SPS (Super Proton Synchrotron). This was a further development of the self-capacitance screen (right), also developed by Stumpe at CERN in 1972.

E.A. Johnson described his work on capacitive touchscreens in a short article published in 1965 and then more fully—with photographs and diagrams—in an article published in 1967. The applicability of touch technology for air traffic control was described in an article published in 1968. Frank Beck and Bent Stumpe, engineers from CERN, developed a transparent touchscreen in the early 1970s, based on Stumpe's work at a television factory in the early 1960s. Then manufactured by CERN, it was put to use in 1973. A resistive touchscreen was developed by American inventor George Samuel Hurst, who received US patent #3,911,215 on October 7, 1975. The first version was produced in 1982.

In 1972, a group at the University of Illinois filed for a patent on an optical touchscreen that became a standard part of the Magnavox Plato IV Student Terminal. Thousands were built for the PLATO IV system. These touchscreens had a crossed array of 16 by 16 infrared position sensors, each composed of an LED on one edge of the screen and a matched phototransistor on the other edge, all mounted in front of a monochrome plasma display panel. This arrangement can sense any fingertip-sized opaque object in close proximity to the screen. A similar touchscreen was used on the HP-150 starting in 1983; this was one of the world's earliest commercial touchscreen computers. HP mounted their infrared transmitters and receivers around the bezel of a 9" Sony Cathode Ray Tube (CRT).

In 1985, Sega released the Terebi Oekaki, also known as the Sega Graphic Board, for the SG-1000 video game console and SC-3000 home computer. It consisted of a plastic pen and a plastic board with a transparent window where the pen presses are detected. It was used primarily for a drawing software application.

In the early 1980s, General Motors tasked its Delco Electronics division with a project aimed at replacing an automobile's non essential functions (i.e. other than throttle, transmission, braking and steering) from mechanical or electro-mechanical systems with solid state alternatives wherever possible. The finished device was dubbed the ECC for "Electronic Control Center", a digital computer and software control system hardwired to various peripheral sensors, servos, solenoids, antenna and a monochrome CRT touchscreen that functioned both as display and sole method of input. The ECC replaced the traditional mechanical stereo, fan, heater and air conditioner controls and displays, and was capable of providing very detailed and specific information about the vehicle's cumulative and current operating status in real time. The ECC was standard equipment on the 1985–89 Buick Riviera and later the 1988–89 Buick Reatta, but was unpopular with consumers partly due to the technophobia of some traditional Buick customers, but mostly because of costly to repair technical problems suffered by the ECC's touchscreen which being the sole access method, would render climate control or stereo operation impossible.

Multi-touch technology began in 1982, when the University of Toronto's Input Research Group developed the first human-input multi-touch system, using a frosted-glass panel with a camera placed behind the glass. In 1985, the University of Toronto group including Bill Buxton developed a multi-touch tablet that used capacitance rather than bulky camera-based optical sensing systems (see History of multi-touch).

In 1986, the first graphical point of sale software was demonstrated on the 16-bit Atari 520ST color computer. It featured a color touchscreen widget-driven interface. The ViewTouch point of sale software was first shown by its developer, Gene Mosher, at Fall Comdex, 1986, in Las Vegas, Nevada to visitors at the Atari Computer demonstration area and was the first commercially available POS system with a widget-driven color graphic touchscreen interface.

In 1987, Casio launched the Casio PB-1000 pocket computer with a touchscreen consisting of a 4x4 matrix, resulting in 16 touch areas in its small LCD graphic screen.

Until 1988 touchscreens had the bad reputation of being imprecise. Most user interface books would state that touchscreens selections were limited to targets larger than the average finger. At the time, selections were done in such a way that a target was selected as soon as the finger came over it, and the corresponding action was performed immediately. Errors were common, due to parallax or calibration problems, leading to frustration. A new strategy called "lift-off strategy" was introduced by researchers at the University of Maryland Human – Computer Interaction Lab and is still used today. As users touch the screen, feedback is provided as to what will be selected, users can adjust the position of the finger, and the action takes place only when the finger is lifted off the screen. This allowed the selection of small targets, down to a single pixel on a VGA screen (standard best of the time).

Sears et al. (1990) gave a review of academic research on single and multi-touch human–computer interaction of the time, describing gestures such as rotating knobs, adjusting sliders, and swiping the screen to activate a switch (or a U-shaped gesture for a toggle switch). The University of Maryland Human – Computer Interaction Lab team developed and studied small touchscreen keyboards (including a study that showed that users could type at 25 wpm for a touchscreen keyboard compared with 58 wpm for a standard keyboard), thereby paving the way for the touchscreen keyboards on mobile devices. They also designed and implemented multitouch gestures such as selecting a range of a line, connecting objects, and a "tap-click" gesture to select while maintaining location with another finger.

In 1990 the University of Maryland Human – Computer Interaction Lab demonstrated a touchscreen slider, which was later cited as prior art in the lock screen patent litigation between Apple and other touchscreen mobile phone vendors (in relation to U.S. Patent 7,657,849).

In c. 1991–92, the Sun Star7 prototype PDA implemented a touchscreen with inertial scrolling. In 1993, the IBM Simon—the first touchscreen phone—was released.

An early attempt at a handheld game console with touchscreen controls was Sega's intended successor to the Game Gear, though the device was ultimately shelved and never released due to the expensive cost of touchscreen technology in the early 1990s. Touchscreens would not be popularly used for video games until the release of the Nintendo DS in 2004. Until recently, most consumer touchscreens could only sense one point of contact at a time, and few have had the capability to sense how hard one is touching. This has changed with the commercialization of multi-touch technology.

Technologies

There are a variety of touchscreen technologies with different methods of sensing touch.

Resistive

Main article: Resistive touchscreen

A resistive touchscreen panel comprises several layers, the most important of which are two thin, transparent electrically-resistive layers separated by a thin space. These layers face each other with a thin gap between. The top screen (the screen that is touched) has a coating on the underside surface of the screen. Just beneath it is a similar resistive layer on top of its substrate. One layer has conductive connections along its sides, the other along top and bottom. A voltage is applied to one layer, and sensed by the other. When an object, such as a fingertip or stylus tip, presses down onto the outer surface, the two layers touch to become connected at that point: The panel then behaves as a pair of voltage dividers, one axis at a time. By rapidly switching between each layer, the position of a pressure on the screen can be read.

Resistive touch is used in restaurants, factories and hospitals due to its high resistance to liquids and contaminants. A major benefit of resistive touch technology is its low cost. Additionally, as only sufficient pressure is necessary for the touch to be sensed, they may be used with gloves on, or by using anything rigid as a finger/stylus substitute. Disadvantages include the need to press down, and a risk of damage by sharp objects. Resistive touchscreens also suffer from poorer contrast, due to having additional reflections from the extra layers of material (separated by an air gap) placed over the screen. This is the type of touchscreen used by Nintendo in the DS family, the 3DS family, and the Wii U GamePad.

Surface acoustic wave

Main article: Surface acoustic wave

Surface acoustic wave (SAW) technology also uses ultrasonic waves that pass over the touchscreen panel. When the panel is touched, a portion of the wave is absorbed. This change in the ultrasonic waves registers the position of the touch event and sends this information to the controller for processing. Surface acoustic wave touchscreen panels can be damaged by outside elements. Contaminants on the surface can also interfere with the functionality of the touchscreen.

Capacitive

Capacitive touchscreen of a mobile phone

Main article: Capacitive sensing

A capacitive touchscreen panel consists of an insulator such as glass, coated with a transparent conductor such as indium tin oxide (InSnO). As the human body is also an electrical conductor, touching the surface of the screen results in a distortion of the screen's electrostatic field, measurable as a change in capacitance. Different technologies may be used to determine the location of the touch. The location is then sent to the controller for processing.

Unlike a resistive touchscreen, one cannot use a capacitive touchscreen through most types of electrically insulating material, such as gloves. This disadvantage especially affects usability in consumer electronics, such as touch tablet PCs and capacitive smartphones in cold weather. It can be overcome with a special capacitive stylus, or a special-application glove with an embroidered patch of conductive thread passing through it and contacting the user's fingertip.

The largest capacitive display manufacturers continue to develop thinner and more accurate touchscreens, with touchscreens for mobile devices now being produced with 'in-cell' technology that eliminates a layer, such as Samsung's Super AMOLED screens, by building the capacitors inside the display itself. This type of touchscreen reduces the visible distance (within millimetres) between the user's finger and what the user is touching on the screen, creating a more direct contact with the content displayed and enabling taps and gestures to be more responsive.

A simple parallel plate capacitor has two conductors separated by a dielectric layer. Most of the energy in this system is concentrated directly between the plates. Some of the energy spills over into the area outside the plates, and the electric field lines associated with this effect are called fringing fields. Part of the challenge of making a practical capacitive sensor is to design a set of printed circuit traces which direct fringing fields into an active sensing area accessible to a user. A parallel plate capacitor is not a good choice for such a sensor pattern. Placing a finger near fringing electric fields adds conductive surface area to the capacitive system. The additional charge storage capacity added by the finger is known as finger capacitance, CF. The capacitance of the sensor without a finger present is denoted as CP in this article, which stands for parasitic capacitance.

Surface capacitance

In this basic technology, only one side of the insulator is coated with a conductive layer. A small voltage is applied to the layer, resulting in a uniform electrostatic field. When a conductor, such as a human finger, touches the uncoated surface, a capacitor is dynamically formed. The sensor's controller can determine the location of the touch indirectly from the change in the capacitance as measured from the four corners of the panel. As it has no moving parts, it is moderately durable but has limited resolution, is prone to false signals from parasitic capacitive coupling, and needs calibration during manufacture. It is therefore most often used in simple applications such as industrial controls and kiosks.

Projected capacitance

Back side of a Multitouch Globe, based on Projected Capacitive Touch (PCT) technology

Schema of projected-capacitive touchscreen

Projected Capacitive Touch (PCT; also PCAP) technology is a variant of capacitive touch technology. All PCT touch screens are made up of a matrix of rows and columns of conductive material, layered on sheets of glass. This can be done either by etching a single conductive layer to form a grid pattern of electrodes, or by etching two separate, perpendicular layers of conductive material with parallel lines or tracks to form a grid. Voltage applied to this grid creates a uniform electrostatic field, which can be measured. When a conductive object, such as a finger, comes into contact with a PCT panel, it distorts the local electrostatic field at that point. This is measurable as a change in capacitance. If a finger bridges the gap between two of the "tracks", the charge field is further interrupted and detected by the controller. The capacitance can be changed and measured at every individual point on the grid (intersection). Therefore, this system is able to accurately track touches. Due to the top layer of a PCT being glass, it is a more robust solution than less costly resistive touch technology. Additionally, unlike traditional capacitive touch technology, it is possible for a PCT system to sense a passive stylus or gloved fingers. However, moisture on the surface of the panel, high humidity, or collected dust can interfere with the performance of a PCT system. There are two types of PCT: mutual capacitance and self-capacitance.

Mutual capacitance

This is a common PCT approach, which makes use of the fact that most conductive objects are able to hold a charge if they are very close together. In mutual capacitive sensors, a capacitor is inherently formed by the row trace and column trace at each intersection of the grid. A 16-by-14 array, for example, would have 224 independent capacitors. A voltage is applied to the rows or columns. Bringing a finger or conductive stylus close to the surface of the sensor changes the local electrostatic field which reduces the mutual capacitance. The capacitance change at every individual point on the grid can be measured to accurately determine the touch location by measuring the voltage in the other axis. Mutual capacitance allows multi-touch operation where multiple fingers, palms or styli can be accurately tracked at the same time.

Self-capacitance

Self-capacitance sensors can have the same X-Y grid as mutual capacitance sensors, but the columns and rows operate independently. With self-capacitance, the capacitive load of a finger is measured on each column or row electrode by a current meter. This method produces a stronger signal than mutual capacitance, but it is unable to resolve accurately more than one finger, which results in "ghosting", or misplaced location sensing.

Use of styli on capacitive screens

Capacitive touchscreens don't necessarily need to be operated by a finger, but until recently the special styli required could be quite expensive to purchase. The cost of this technology has fallen greatly in recent years and capacitative styli are now widely available for a nominal charge, and often given away free with mobile accessories.

Infrared grid

Infrared sensors mounted around the display watch for a user's touchscreen input on this PLATO V terminal in 1981. The monochromatic plasma display's characteristic orange glow is illustrated.

An infrared touchscreen uses an array of X-Y infrared LED and photodetector pairs around the edges of the screen to detect a disruption in the pattern of LED beams. These LED beams cross each other in vertical and horizontal patterns. This helps the sensors pick up the exact location of the touch. A major benefit of such a system is that it can detect essentially any input including a finger, gloved finger, stylus or pen. It is generally used in outdoor applications and point of sale systems which can not rely on a conductor (such as a bare finger) to activate the touchscreen. Unlike capacitive touchscreens, infrared touchscreens do not require any patterning on the glass which increases durability and optical clarity of the overall system. Infrared touchscreens are sensitive to dirt/dust that can interfere with the IR beams, and suffer from parallax in curved surfaces and accidental press when the user hovers his/her finger over the screen while searching for the item to be selected.

Infrared acrylic projection

A translucent acrylic sheet is used as a rear projection screen to display information. The edges of the acrylic sheet are illuminated by infrared LEDs, and infrared cameras are focused on the back of the sheet. Objects placed on the sheet are detectable by the cameras. When the sheet is touched by the user the deformation results in leakage of infrared light, which peaks at the points of maximum pressure indicating the user's touch location. Microsoft's PixelSense tables use this technology.

Optical imaging

Optical touchscreens are a relatively modern development in touchscreen technology, in which two or more image sensors are placed around the edges (mostly the corners) of the screen. Infrared back lights are placed in the camera's field of view on the other side of the screen. A touch shows up as a shadow and each pair of cameras can then be pinpointed to locate the touch or even measure the size of the touching object (see visual hull). This technology is growing in popularity, due to its scalability, versatility, and affordability, especially for bigger units.

Dispersive signal technology

Introduced in 2002, by 3M, this system uses sensors to detect the piezoelectricity in the glass that occurs due to a touch. Complex algorithms then interpret this information and provide the actual location of the touch. The technology claims to be unaffected by dust and other outside elements, including scratches. Since there is no need for additional elements on screen, it also claims to provide excellent optical clarity. Also, since mechanical vibrations are used to detect a touch event, any object can be used to generate these events, including fingers and stylus. A downside is that after the initial touch the system cannot detect a motionless finger.

Acoustic pulse recognition

The key to this technology is that a touch at any one position on the surface generates a sound wave in the substrate which then produces a unique combined sound after being picked up by three or more tiny transducers attached to the edges of the touchscreen. The sound is then digitized by the controller and compared to a list of pre-recorded sounds for every position on the surface. The cursor position is instantly updated to the touch location. A moving touch is tracked by rapid repetition of this process. Extraneous and ambient sounds are ignored since they do not match any stored sound profile. The technology differs from other attempts to recognize the position of touch with transducers or microphones in using a simple table look-up method, rather than requiring powerful and expensive signal processing hardware to attempt to calculate the touch location without any references. As with the dispersive signal technology system, a motionless finger cannot be detected after the initial touch. However, for the same reason, the touch recognition is not disrupted by any resting objects. The technology was created by SoundTouch Ltd in the early 2000s, as described by the patent family EP1852772, and introduced to the market by Tyco International's Elo division in 2006 as Acoustic Pulse Recognition. The touchscreen used by Elo is made of ordinary glass, giving good durability and optical clarity. APR is usually able to function with scratches and dust on the screen with good accuracy. The technology is also well suited to displays that are physically larger.

Construction

There are several principal ways to build a touchscreen. The key goals are to recognize one or more fingers touching a display, to interpret the command that this represents, and to communicate the command to the appropriate application.

In the most popular techniques, the capacitive or resistive approach, there are typically four layers:

1. Top polyester coated with a transparent metallic conductive coating on the bottom
2. Adhesive spacer
3. Glass layer coated with a transparent metallic conductive coating on the top
4. Adhesive layer on the backside of the glass for mounting.

When a user touches the surface, the system records the change in the electric current that flows through the display.

Dispersive-signal technology which 3M created in 2002, measures the piezoelectric effect—the voltage generated when mechanical force is applied to a material—that occurs chemically when a strengthened glass substrate is touched.

There are two infrared-based approaches. In one, an array of sensors detects a finger touching or almost touching the display, thereby interrupting light beams projected over the screen. In the other, bottom-mounted infrared cameras record screen touches.

In each case, the system determines the intended command based on the controls showing on the screen at the time and the location of the touch.

Development

The development of multipoint touchscreens facilitated the tracking of more than one finger on the screen; thus, operations that require more than one finger are possible. These devices also allow multiple users to interact with the touchscreen simultaneously.

With the growing use of touchscreens, the marginal cost of touchscreen technology is routinely absorbed into the products that incorporate it and is nearly eliminated. Touchscreens now have proven reliability. Thus, touchscreen displays are found today in airplanes, automobiles, gaming consoles, machine control systems, appliances, and handheld display devices including the Nintendo DS and multi-touch enabled cellphones; the touchscreen market for mobile devices is projected to produce US$5 billion in 2009.

The ability to accurately point on the screen itself is also advancing with the emerging graphics tablet/screen hybrids. PVDF plays a major role in this innovation due its high piezoelectric properties.

TapSense, announced in October 2011, allows touchscreens to distinguish what part of the hand was used for input, such as the fingertip, knuckle and fingernail. This could be used in a variety of ways, for example, to copy and paste, to capitalize letters, to activate different drawing modes, and similar.

Ergonomics and usage

Touchscreen accuracy

Users must be able to accurately select targets on touchscreens, and avoid accidental selection of adjacent targets, to effectively use a touchscreen input device. The design of touchscreen interfaces must reflect both technical capabilities of the system, ergonomics, cognitive psychology and human physiology.

Guidelines for touchscreen designs were first developed in the 1990s, based on early research and actual use of older systems, so assume the use of contemporary sensing technology such as infrared grids. These types of touchscreens are highly dependent on the size of the user's fingers, so their guidelines are less relevant for the bulk of modern devices, using capacitive or resistive touch technology. From the mid-2000s onward, makers of operating systems for smartphones have promulgated standards, but these vary between manufacturers, and allow for significant variation in size based on technology changes, so are unsuitable from a human factors perspective.

Much more important is the accuracy humans have in selecting targets with their finger or a pen stylus. The accuracy of user selection varies by position on the screen. Users are most accurate at the center, less so at the left and right edges, and much less accurate at the top and especially bottom edges. The R95 accuracy varies from 7 mm in the center, to 12 mm in the lower corners. Users are subconsciously aware of this, and are also slightly slower, taking more time to select smaller targets, and any at the edges and corners.

This inaccuracy is a result of parallax, visual acuity and the speed of the feedback loop between the eyes and fingers. The precision of the human finger alone is much, much higher than this, so when assistive technologies are provided such as on-screen magnifiers, users can move their finger (once in contact with the screen) with precision as small as 0.1 mm.

Hand position, digit used and switching

Users of handheld and portable touchscreen devices hold them in a variety of ways, and routinely change their method of holding and selection to suit the position and type of input. There are four basic types of handheld interaction:

1. Holding at least in part with both hands, tapping with a single thumb
2. Holding with one hand, tapping with the finger (or rarely, thumb) of another hand
3. Holding the device in one hand, and tapping with the thumb from that hand
4. Holding with two hands and tapping with both thumbs

Use rates vary widely. While two-thumb tapping is encountered rarely (1-3%) for many general interactions, it is used for 41% of typing interaction.

In addition, devices are often placed on surfaces (desks or tables) and tablets especially are used in stands. The user may point, select or gesture in these cases with their finger or thumb, and also varies the use.

Combined with haptics

Touchscreens are often used with haptic response systems. A common example of this technology is the vibratory feedback provided when a button on the touchscreen is tapped. Haptics are used to improve the user's experience with touchscreens by providing simulated tactile feedback, and can be designed to react immediately, partly countering on-screen response latency. Research from the University of Glasgow Scotland [Brewster, Chohan, and Brown 2007 and more recently Hogan] demonstrates that sample users reduce input errors (20%), increase input speed (20%), and lower their cognitive load (40%) when touchscreens are combined with haptics or tactile feedback [vs. non-haptic touchscreens].

"Gorilla arm"

Extended use of gestural interfaces without the ability of the user to rest their arm is referred to as "gorilla arm." It can result in fatigue, and even repetitive stress injury when routinely used in a work setting. Certain early pen-based interfaces required the operator to work in this position for much of the work day. Allowing the user to rest their hand or arm on the input device or a frame around it is a solution for this in many contexts. This phenomenon is often cited as a prima facie example of what not to do in ergonomics.

Unsupported touchscreens are still fairly common in applications such as ATMs and data kiosks, but are not an issue as the typical user only engages for brief and widely spaced periods.

Fingerprints

Fingerprints and smudges on an iPad touchscreen

Touchscreens can suffer from the problem of fingerprints on the display. This can be mitigated by the use of materials with optical coatings designed to reduce the visible effects of fingerprint oils, or oleophobic coatings as most of the modern smartphones, which lessen the actual amount of oil residue (which includes alcohol), or by installing a matte-finish anti-glare screen protector, which creates a slightly roughened surface that does not easily retain smudges, or by reducing skin contact by using a fingernail or stylus.

See also

• Dual-touchscreen
• Pen computing
• Energy harvesting
• Flexible keyboard
• Gestural interface
• Graphics tablet
• Graphics tablet-screen hybrid
• Lock screen
• List of Touch Solution manufacturers
• Tablet PC
• Touch switch
• Touchscreen remote control
• Multi-touch
• Omnitouch
• SixthSense

External links

• Howstuffworks - How do touchscreen monitors know where you're touching?
• What are the different types of touchscreen technology
• From touch displays to the surface: A brief history of touchscreen technology
• Annotated bibliography of references to handwriting recognition and pen computing