Pen and Multi-touch Computing

One of the primary research themes in the IS&UE RCE is creating new and innovative pen-centric user interfaces and applications. The main focus of our work in pen computing is mathematical sketching, the ability to associate handwritten mathematics and drawings together to create illustrations that assist users in learning and understanding science and mathematical concepts. We are also interested in learning how users are affected by the pen-centric interfaces we develop and whether the educational applications we create are pedagogically effective. In addition, we are exploring how multi-touch interfaces can be used in mathematics and in improving workflow efficiency.

Mathematical Sketching

Diagrams and illustrations are frequently used to help explain mathematical concepts. Students often create them with pencil and paper as an intuitive aid in visualizing relationships among variables, constants, and functions, and use them as a guide in writing the appropriate mathematics to solve the problem. However, such static diagrams generally assist only in the initial formulation of the required mathematics, not in "debugging" or problem analysis. This can be a severe limitation, even for simple problems with a natural mapping to the temporal dimension or problems with complex spatial relationships.

Mathematical sketching is a novel, pen-based, gestural interaction paradigm for mathematics problem solving, designed to overcome these limitations. Mathematical sketching derives from the familiar pencil-and-paper process of drawing supporting diagrams to facilitate the formulation of mathematical expressions; however, with mathematical sketching, users can also leverage their physical intuition by watching their hand-drawn diagrams animate in response to continuous or discrete parameter changes in their written formulas. Diagram animation is driven by implicit associations that are inferred, either automatically or with gestural guidance, from mathematical expressions, diagram labels and drawing elements.

Understanding User Performance with Pen-Based Interfaces

The pen-and-paper interface metaphor that pen-based computers provide make pen-based interfaces both natural and expressive. However, these interfaces can also be difficult to use without proper feedback and guidance in understanding how to perform the various gestures and pen-based commands for a particular application. This project focuses on understanding we can develop tools to assist users when working with pen-based UIs.