Introduction
The central theme of our current interface
research at the Brown University Computer Graphics Lab is that gesture-based
interfaces are, in many cases, faster and more direct than traditional windows,
icons, menus and pointers (WIMP) style interaction. This theme is especially
true when dealing with interaction techniques for 3D modeling applications. Not
only are they fast and more direct, but gesture-based interfaces allows for
users to have maximum screen space when 3D modeling because the clutter of menus
and widgets such as buttons and sliders are removed. The advantages and
potential for gesture-based interaction has led us to develop Jot, a framework
that allows Graphics Lab members to research novel gesture-based interaction
techniques for the various tasks and operations used in 3D modeling.
The
Jot framework is based on our previous work, called Sketch, a system for
creating imprecise 3D models[2]. Sketch's interface uses only a three button
mouse (no menus or 2D interface widgets are used). Sketch interprets lines drawn
by the user on the image plane of a 3D view as operations and parameters. These
operations include primitive creation, primitive manipulation, and camera
manipulation. For example, gestures that create primitives provide enough
information to select which primitive to create, its dimensions and its place in
3D. Creating a cube, for example, requires the user to draw 3 gesture lines one
for each of the principle axes, each line meeting at a single point. The cube is
generated with it's length, width, and height corresponding to the three gesture
lines and its place in 3D based on the intersection point. Primitives such as
cylinders, cones, pyramids, and extrusions can also be instantiated. The
primitive manipulation interface allows for automatic object constraint by
gesturally drawing a motion constraint over the object before manipulating it.
For example, to constrain an object's movement to a given axis, a straight line
is drawn indicating what axis to constrain the object to, and when the user
moves the object it will only move along that axis. Other gestures constrain
objects to move along surfaces, rotate around a given principle axis, or scale
and deform to fit a new gesture contour.
The beauty of the Jot framework
is that it acts as a building block for other related gesture-based interaction
research. The framework has been used for a wide spectrum of interface research
for modeling applications, from concept design to design of detailed
feature-based parametric parts. As members of the our graphics lab research
other venues of gesture-based interaction, the techniques and ideas are
incorporated into Jot such that it can be expanded and extended for future use.
Three of the current extensions that we are working on include ErgoSketch,
SmartSketch, and NetSketch. All three projects are based on the Jot framework
and will be described in this article.
ErgoSketch
ErgoSketch stems from the idea that although Jot makes it easy to create
3D models, there are some tasks that mouse or pen-based techniques do not
naturally support. For example, the task of rapidly examining objects in one's
hand and annotating in three-dimensions are better suited to higher
degree-of-freedom interaction techniques. As a result of this deficiency, we
developed a system that incorporates both the initial interaction techniques of
Jot and techniques for doing operations that require a switch of interaction
metaphors[1]. ErgoSketch users sit at an ActiveDesk (a variant of the responsive
workbench developed by Input Technologies, Inc. of Toronto) and can perform the
2D gesture-based 3D modeling operations Jot provides. In addition, they can pick
up, examine, and annotate objects with a virtual pen. To transition between
entering 2D gestures and manipulating an object, the user puts down the pen and
picks up a tracked proxy object attached to the Active Desk. This action signals
the system to switch to stereoscopic viewing mode. The object the user was
constructing can now be rapidly examined in 3D. In addition, while holding the
virtual object with one hand, a user can use his other hand to pick up a second
tracker which acts as a virtual pen for annotation of the held object. When the
user places the initial tracker back on the table, the system switches back to
monoscopic mode and the user can go back to creating and editing objects. It is
this seamless transition between 2D and 3D interaction that we find most
interesting.
SmartSketch
The idea behind SmartSketch is
that, in many cases, users do not need to start from such basic primitives as
cubes, cones, and cylinders when creating 3D models. For example, a user is
modeling the interior of certain rooms in a house to show customers the
different furniture arrangements possible. The model requires such primitives as
tables, chairs, desks, and the like. It would be inefficient to have to create
believable furniture objects from the basic primitive set that Jot provides.
Therefore, SmartSketch allows user to create 3D models by using
context-sensitive primitives based on the specific application domain. These
"advanced" primitives are created using Jot's gesture-based interface metaphor
but with new semantic meanings applied to the creation of objects. In the
furniture layout application, the gesture for creating a cube is semantically
changed such that a chair, desk, window, or wall is created depending on the
context in which the gestures were drawn. For example, if the strokes drawn take
the shape of a bounding box for a chair, then a chair will be created with those
dimensions. However, if the strokes drawn take the shape of a bounding box for a
desk, then a desk is created. The context in which the shape is drawn is used to
resolve any ambiguity. For example, one set of gestures drawn on a floor object
might create a chair, while the same gestures drawn on a desk could create a
lamp. We are currently exploring a number of other application domains such as
exterior landscaping and city planning.
NetSketch
Another
important aspect of 3D modeling that we feel is interesting is the ability to
have multiple people working together in a shared virtual space in real-time.
NetSketch is a project in its early stages that allows multiple participants to
work in a shared virtual space for 3D modeling. Since the Jot framework is built
for the single user only, we felt an important extension is interfaces for group
interaction. NetSketch will allow us to research the many issues that come up
when combining gesture-based interfaces with a collaborative modeling
application. We are currently researching how to allow a given participant to
show his camera view to others, how to let participants interact with the same
object, and visualization techniques and communication mechanisms that allow
participants to better understand what others are doing in the virtual space.
Conclusion
Jot is an important and exciting framework for
gesture-based interface research. Its flexibility and robustness has allowed for
many extensions and new projects to develop in our lab. As Jot gets larger and
more ideas are explored, we hope that our body of interaction research will not
only help to advance gesture-based interfaces in 3D modeling but other
applications as well so interfaces are easier, faster, and natural to use.
References
[1] Forsberg, A. S., LaViola, J. J.,
Markosian, L., Zeleznik, R. C. (1997) "Seamless Interaction in Virtual Reality",
IEEE Computer Graphics and Applications, 17(6):6-9, November/December 1997.
[2] Zeleznik, R.C., Herndon, K., Hughes, J. (1996) "Sketch: An Interface
for Sketching 3D Scenes." Proceedings of SIGGRAPH'96, 163-170.