The Tremendous VR and CG Systems—of the 1960s

Ivan Sutherland has blazed a truly unique trail through computing over the past six decades. One of the most influential figures in the story of computing, he helped to open new pathways for others to explore and dramatically extend: interactive computer graphics, virtual reality, 3D computer graphics, and asynchronous systems, to name but a few.

The Computer History Museum recently made public its two-part oral history with Sutherland. These interviews present a wonderful opportunity to learn more about Sutherland’s life in computing, in his own words. The interview transcripts can be downloaded here and here. And the two-part video of the interview is available here and here.

Bob Sproull, a lifelong colleague of Sutherland and himself a major figure in computing, served as instigator, interviewer, and editor for these oral histories, and he involved me, Marc Weber, and Jim Waldo in the effort. The Museum is making this oral history interview public during the 60th anniversary year of Sutherland’s breakthrough in interactive computer graphics, the program Sketchpad, for which he earned his Ph.D. from MIT in 1963.

Ivan Sutherland’s Unique Perspective

There is a phrase, popular in 17th and 18th century England, that occurs to me when thinking about Ivan Sutherland: “A man of many parts.” The phrase refers to an individual who had made serious contributions to a domain, while also possessing multiple, and often diverse, talents and pursuits. The description fits Sutherland well, although it also misses something important: There is a commonality in his many accomplishments, a shared wellspring for his many parts.

To get at this wellspring, start with geometry. From his youth, Sutherland possessed an unusually keen spatial, geometric intuition. In his mind and at his hands, he experienced an immediacy in perceiving how things fit and worked together. Perspective drawing involves a set of techniques to represent a three-dimensional scene on the two-dimensional plane of a sheet of paper or a stretch of canvas. These renderings can proceed in different ways, determined by the number of vanishing points employed. Together the vanishing points define the viewpoint of the observer. One-point, two-point, and three-point perspectives are all very different, providing distinct ways to understand the represented scene.

This switching of viewpoints, the ability to look at something from a fresh and unexpected angle, and then to integrate this new perspective with those that came before, seems to me the link between Sutherland’s unusual spatial intuition with his diverse contributions in computing. It’s an ability to find a new viewpoint on a subject, to look at it from this novel perspective, and then to explore how this vantage might change the subject itself through fresh solutions and directions.

The Early Influence of Edmund Berkeley

As recounted in his oral history interviews, Sutherland’s life in computing was profoundly shaped by interactions he and his older brother, Bert, had with two central figures in the early history of computing: Edmund Berkeley and Claude Shannon. Bert, who went on to a remarkable career in computing himself, distinguished by his roles as a research manager at Xerox PARC and at Sun Laboratories, told his story in his own oral history with the Computer History Museum in 2017.

The Sutherland brothers, through a connection of their mother’s, began visiting Edmund Berkeley in New York City from their home in Scarsdale while Ivan was still in grade school. At the time, Berkeley was establishing himself as a leading author, publisher, and consultant for the new world of digital computers. In Berkeley’s offices, the Sutherland brothers encountered his light-seeking robot squirrel “Squee,” now in the collection of the Computer History Museum, which also holds some of Berkeley’s papers.

As a boy, Ivan Sutherland visited Edmund Berkeley’s office and met his robot squirrel “Squee.”Mark Richards/Computer History Museum

Back at home, the Sutherland brothers worked on their own versions of light-seeking robots, using surplus parts their engineer-father helped them to source in New York City. The Sutherland parents were happy to indulge their sons’ inquiries. In this clip from his oral history, Ivan describes a surplus Sperry P4 military gunsight computer that his father installed in the family kitchen:

Ivan Sutherland on the surplus Navy computer that his parents installed in the kitchen.www.youtube.com

As an undergraduate engineering student at Carnegie Tech (today’s Carnegie Mellon University), and then as a graduate student at Caltech (before he moved to MIT after one year), Sutherland continued to design and build more advanced light-seeking robots. The reason? Aesthetics, he explains in his oral history. For Sutherland, engineering design has a strong aesthetic dimension. Beauty and simplicity gave the practice of engineering an affective pull. “In fact, I think that engineering and art are very closely related,” he explains.

In Berkeley’s offices, the Sutherland brothers also had the opportunity to work with his new creation, Simon, a very simple and inexpensive computer. Unlike the giant mainframes of this era, which relied on thousands of vacuum tubes, Simon was compact and animated by a handful of inexpensive relays—simple electrical on/off switches. Nevertheless, the machine was able to perform mathematical and logical operations.

Edmund Berkeley’s Simon was a simple and inexpensive digital computer that used electrical relays.Mark Richards/Computer History Museum

Further, Simon was programmable, using instructions encoded on a punched paper tape. During his high school years in the 1950s, Ivan Sutherland devised a program for Simon that allowed it to perform division, quite a feat for the humble machine. “I’m quite proud of having written a division routine for a two-bit computer when I was in high school,” he explains in the oral history. “So I can almost literally say I’ve been in the computer business nearly all my life.”

Through Berkeley, the Sutherland brothers were introduced to Claude Shannon, renowned for his development of information theory. While a maestro of abstraction, Shannon was also a keen builder. During a visit to Shannon’s office at the Bell Telephone Laboratories in northern New Jersey, he showed the brothers his creation Theseus.

Theseus consisted of a small maze of movable metal panels affixed to the top of a metal box containing magnets and relay electronics like Berkeley’s Simon. Through the action of the relays and magnets, a toy mouse was able to find its way through the maze and then “remember” the successful route. While the Sutherland brothers were duly impressed, their attempts to re-create this early effort in machine problem-solving and artificial intelligence proved unsuccessful.

Breakthrough at MIT

After graduating from Carnegie Tech in 1959, Ivan Sutherland headed to Caltech for graduate studies in electrical engineering. There, as he recounts in his oral history, he was invited to attend a lunch with Marvin Minsky and Oliver Selfridge, two central figures in digital computing at MIT and the new field of artificial intelligence. Over the meal, Sutherland listened to Minsky’s and Selfridge’s enthusiastic reports of computer developments at MIT and its Lincoln Laboratory. Adding to Sutherland’s excitement about MIT was the fact that Claude Shannon had moved there. Sutherland quickly decided to continue his graduate work at MIT, and Shannon agreed to advise him.

At MIT, Sutherland met with Wesley A. Clark, the designer and impresario of an immensely powerful experimental computer, the TX-2, at MIT’s Lincoln Laboratory. Clark had incorporated into the TX-2 two critical innovations in computer technology: high-speed switching transistors and large-capacity magnetic core memories. The machine would provide valuable lessons about the use, capabilities, and potential of these new technologies.

A transistorized flip-flop logic module from the TX-2.Computer History Museum

Clark believed the TX-2 had the potential to make real a new kind of computing. As Sutherland explains in his oral history, “Wes took TX-2 and treated it as a window into the future of what computing might be if everybody had one of his own.” Sutherland proposed to use the TX-2 to create software for generating engineering drawings. Without hesitation, Clark gave him access to the machine. In this clip from his oral history, Sutherland recalls how he came to use the TX-2:

Ivan Sutherland on using Lincoln Lab’s TX-2 to create Sketchpad.www.youtube.com

In January 1963, Sutherland completed his Ph.D. on the system he created on the TX-2, called Sketchpad. With it, a user was able to interactively and in real time make line drawings on the computer’s CRT screen, using a light pen for direct input on the display. Sketchpad afforded many different capabilities for working with these line drawings, including the automatic completion of shapes, resizing, and the ability to copy and repeat elements.

In his dissertation, Sutherland described how to use Sketchpad to make animated cartoons, including this winking girl.Ivan Sutherland/Computer History Museum

Sketchpad represented much more than just a new way to create line art. As Sutherland put it in his thesis, “The Sketchpad system makes it possible for a man and a computer to converse rapidly through the medium of line drawings. Heretofore, most interaction between men and computers has been slowed down by the need to reduce all communication to written statements that can by typed; in the past, we have been writing letters to rather than conferring with our computers.…The Sketchpad system…opens up a new era of man-machine communication.” Sutherland’s source code for Sketchpad is available here, and his 1994 lecture about the history of Sketchpad can be viewed here.

Innovation in the Military

After MIT, Sutherland fulfilled his ROTC commitments by serving in the U.S. Army, first at the National Security Agency, where he continued to work on computer graphics, and then as the second director of the Information Processing Technology Office of ARPA, the Advanced Research Projects Agency of the U.S. Department of Defense. Only in his mid-20s, Sutherland succeeded the MIT psychologist J.C.R. Licklider, who had established the office and its leading role in supporting computer science and artificial intelligence research in the United States.

While Sutherland continued many of Licklider’s projects at ARPA, he added new projects of his own. Critically for Sutherland, he supported a new effort by Clark, who by then had moved from MIT to Washington University, in St. Louis. Clark had created an innovative small computer intended for an individual user called the LINC, especially suited to the real time needs of biomedical research, and he moved the project and team to St. Louis. (Clark discussed the history of the LINC in a 1986 talk.)

Now, Clark envisioned an entirely new approach to computer design. In it, computers would be built up from distinct units, each unit providing an entire function. In this way, computers could be assembled in a flexible and bespoke manner, built with just what was needed for a particular use and no more. Clark called the approach macromodule, and Sutherland funded the research.

This new approach was a radical departure in digital computing design. In the mainstream, all the operations of computers were coordinated by following the regular beat of a single electronic signal, the “clock.” For the macromodule approach, an alternate, asynchronous approach to the orchestration of computer operations was required. The practical challenges and the theoretical potentials of asynchronous systems became a central passion and focus for Sutherland thereafter.

The researchers in Clark’s macromodule effort built a variety of different units, such as this addition module donated to the Computer History Museum by Ivan Sutherland.

One of the modules built as part of Wesley Clark’s macromodule project. This one performed addition.Computer History Museum

After his appointment at ARPA, Sutherland accepted a tenured engineering position at Harvard University. There, he expanded his graphical ambitions from the two-dimensional abilities of Sketchpad to three-dimensional graphics and a new interface for experiencing them. He established a laboratory of graduate and undergraduate students alike, aimed at creating views of 3D scenes—drawn with lines—as well as a display worn on the head for viewing those scenes. The display would present different views of the 3D scene depending on the direction that the user looked. By the close of the 1960s, they had a working system. This project is frequently cited as an early milestone in virtual reality. Sutherland discussed this project and its relation to virtual reality in this 1996 lecture.

The head-mounted display from Ivan Sutherland’s Harvard project.Computer History Museum

In 1968, Sutherland moved to the University of Utah, and he cofounded a new startup to pursue 3D computer graphics. Sutherland’s key partner in both activities was David C. Evans, an accomplished computer researcher. Evans was establishing a computer science department at Utah focused on 3D computer graphics, the same focus as the company he was starting with Sutherland. The new company, Evans & Sutherland, moved quickly to produce workstations for creating 3D graphics, beginning with the LDS-1 and then moving on to the very successful Picture System. Other products and efforts became essential to computer animation and to military pilot training.

Sutherland and Evans fostered a remarkably productive and creative community of students at Utah, who went on to cofound Adobe, Pixar, and Silicon Graphics, among other companies. Some of these figures discussed this remarkable environment in a 1994 meeting.

Sutherland’s experiences through his time in Utah comprise just the first half of his story in computing and engineering. Beyond it lies another startup, a faculty career at Caltech, a revolution in VLSI microchip design, a walking-robot project at Carnegie Mellon, venture capital investing, a consulting firm that became the basis for Sun Laboratories, and fresh contributions to asynchronous systems that continue to this day at Portland State University. For these stories, Sutherland’s new oral history interviews are an incredible source, as are this event with the Sutherland brothers in 2004 and this retrospective lecture by Ivan Sutherland at the Computer History Museum in 2005.

Editor’s note: This post originally appeared on the blog of the Computer History Museum.