Pen and Multi-touch Computing

One of the primary research themes in the IS&UE lab is creating new and innovative pen-centric user interfaces and applications. The main focus of our work in pen computing is mathematical sketching, the ability to associate handwritten mathematics and drawings together to create illustrations that assist users in learning and understanding science and mathematical concepts. We are also interested in learning how users are affected by the pen-centric interfaces we develop and whether the educational applications we create are pedagogically effective. In addition, we are exploring how multi-touch interfaces can be used in mathematics and in improving workflow efficiency.

Mathematical Sketching

Diagrams and illustrations are frequently used to help explain mathematical concepts. Students often create them with pencil and paper as an intuitive aid in visualizing relationships among variables, constants, and functions, and use them as a guide in writing the appropriate mathematics to solve the problem. However, such static diagrams generally assist only in the initial formulation of the required mathematics, not in "debugging" or problem analysis. This can be a severe limitation, even for simple problems with a natural mapping to the temporal dimension or problems with complex spatial relationships.

Mathematical sketching is a novel, pen-based, gestural interaction paradigm for mathematics problem solving, designed to overcome these limitations. Mathematical sketching derives from the familiar pencil-and-paper process of drawing supporting diagrams to facilitate the formulation of mathematical expressions; however, with mathematical sketching, users can also leverage their physical intuition by watching their hand-drawn diagrams animate in response to continuous or discrete parameter changes in their written formulas. Diagram animation is driven by implicit associations that are inferred, either automatically or with gestural guidance, from mathematical expressions, diagram labels and drawing elements.

Rapid Prototyping Gesture Recognition

Recently, $-family recognizers have become prevalent for sketch recognition, and for good reason. They are easily implemented and accurate. However, there is a trade off between accuracy and execution time. We have explored techniques of minimizing this tradeoff using minimalist recognizers and intelligent prototype selection.

Understanding User Performance with Pen and Touch-Based Interfaces

The pen-and-paper interface metaphor that pen-based computers provide make pen-based interfaces both natural and expressive. However, these interfaces can also be difficult to use without proper feedback and guidance in understanding how to perform the various gestures and pen-based commands for a particular application. This project focuses on understanding we can develop tools to assist users when working with pen-based UIs.

Multi-touch Computing

The focus of this project is to explore multi-touch input. We are currently investigating how this input modality can be used for interacting with 3D objects on a 2D display, and how it can be combined with pen-input for entering and manipulating handwritten mathematical expressions.