The central theme of our current interface research at the Brown University Computer Graphics Lab is that gesture-based interfaces are, in many cases, faster and more direct than traditional windows, icons, menus and pointers (WIMP) style interaction. This theme is especially true when dealing with interaction techniques for 3D modeling applications. Not only are they fast and more direct, but gesture-based interfaces allows for users to have maximum screen space when 3D modeling because the clutter of menus and widgets such as buttons and sliders are removed. The advantages and potential for gesture-based interaction has led us to develop Jot, a framework that allows Graphics Lab members to research novel gesture-based interaction techniques for the various tasks and operations used in 3D modeling.
The Jot framework is based on our previous work, called Sketch, a system for creating imprecise 3D models. Sketch's interface uses only a three button mouse (no menus or 2D interface widgets are used). Sketch interprets lines drawn by the user on the image plane of a 3D view as operations and parameters. These operations include primitive creation, primitive manipulation, and camera manipulation. For example, gestures that create primitives provide enough information to select which primitive to create, its dimensions and its place in 3D. Creating a cube, for example, requires the user to draw 3 gesture lines one for each of the principle axes, each line meeting at a single point. The cube is generated with it's length, width, and height corresponding to the three gesture lines and its place in 3D based on the intersection point. Primitives such as cylinders, cones, pyramids, and extrusions can also be instantiated. The primitive manipulation interface allows for automatic object constraint by gesturally drawing a motion constraint over the object before manipulating it. For example, to constrain an object's movement to a given axis, a straight line is drawn indicating what axis to constrain the object to, and when the user moves the object it will only move along that axis. Other gestures constrain objects to move along surfaces, rotate around a given principle axis, or scale and deform to fit a new gesture contour.
The beauty of the Jot framework is that it acts as a building block for other related gesture-based interaction research. The framework has been used for a wide spectrum of interface research for modeling applications, from concept design to design of detailed feature-based parametric parts. As members of the our graphics lab research other venues of gesture-based interaction, the techniques and ideas are incorporated into Jot such that it can be expanded and extended for future use. Three of the current extensions that we are working on include ErgoSketch, SmartSketch, and NetSketch. All three projects are based on the Jot framework and will be described in this article.
ErgoSketch stems from the idea that although Jot makes it easy to create 3D models, there are some tasks that mouse or pen-based techniques do not naturally support. For example, the task of rapidly examining objects in one's hand and annotating in three-dimensions are better suited to higher degree-of-freedom interaction techniques. As a result of this deficiency, we developed a system that incorporates both the initial interaction techniques of Jot and techniques for doing operations that require a switch of interaction metaphors. ErgoSketch users sit at an ActiveDesk (a variant of the responsive workbench developed by Input Technologies, Inc. of Toronto) and can perform the 2D gesture-based 3D modeling operations Jot provides. In addition, they can pick up, examine, and annotate objects with a virtual pen. To transition between entering 2D gestures and manipulating an object, the user puts down the pen and picks up a tracked proxy object attached to the Active Desk. This action signals the system to switch to stereoscopic viewing mode. The object the user was constructing can now be rapidly examined in 3D. In addition, while holding the virtual object with one hand, a user can use his other hand to pick up a second tracker which acts as a virtual pen for annotation of the held object. When the user places the initial tracker back on the table, the system switches back to monoscopic mode and the user can go back to creating and editing objects. It is this seamless transition between 2D and 3D interaction that we find most interesting.
The idea behind SmartSketch is that, in many cases, users do not need to start from such basic primitives as cubes, cones, and cylinders when creating 3D models. For example, a user is modeling the interior of certain rooms in a house to show customers the different furniture arrangements possible. The model requires such primitives as tables, chairs, desks, and the like. It would be inefficient to have to create believable furniture objects from the basic primitive set that Jot provides. Therefore, SmartSketch allows user to create 3D models by using context-sensitive primitives based on the specific application domain. These "advanced" primitives are created using Jot's gesture-based interface metaphor but with new semantic meanings applied to the creation of objects. In the furniture layout application, the gesture for creating a cube is semantically changed such that a chair, desk, window, or wall is created depending on the context in which the gestures were drawn. For example, if the strokes drawn take the shape of a bounding box for a chair, then a chair will be created with those dimensions. However, if the strokes drawn take the shape of a bounding box for a desk, then a desk is created. The context in which the shape is drawn is used to resolve any ambiguity. For example, one set of gestures drawn on a floor object might create a chair, while the same gestures drawn on a desk could create a lamp. We are currently exploring a number of other application domains such as exterior landscaping and city planning.
Another important aspect of 3D modeling that we feel is interesting is the ability to have multiple people working together in a shared virtual space in real-time. NetSketch is a project in its early stages that allows multiple participants to work in a shared virtual space for 3D modeling. Since the Jot framework is built for the single user only, we felt an important extension is interfaces for group interaction. NetSketch will allow us to research the many issues that come up when combining gesture-based interfaces with a collaborative modeling application. We are currently researching how to allow a given participant to show his camera view to others, how to let participants interact with the same object, and visualization techniques and communication mechanisms that allow participants to better understand what others are doing in the virtual space.
Jot is an important and exciting framework for gesture-based interface research. Its flexibility and robustness has allowed for many extensions and new projects to develop in our lab. As Jot gets larger and more ideas are explored, we hope that our body of interaction research will not only help to advance gesture-based interfaces in 3D modeling but other applications as well so interfaces are easier, faster, and natural to use.
 Forsberg, A. S., LaViola, J. J., Markosian, L., Zeleznik, R. C. (1997) "Seamless Interaction in Virtual Reality", IEEE Computer Graphics and Applications, 17(6):6-9, November/December 1997.
 Zeleznik, R.C., Herndon, K., Hughes, J. (1996) "Sketch: An Interface for Sketching 3D Scenes." Proceedings of SIGGRAPH'96, 163-170.