## Pen and Multi-touch Computing

One of the primary research themes in the IS&UE RCE is creating new and innovative pen-centric user interfaces and applications. The main focus of our work in pen computing is mathematical sketching, the ability to associate handwritten mathematics and drawings together to create illustrations that assist users in learning and understanding science and mathematical concepts. We are also interested in learning how users are affected by the pen-centric interfaces we develop and whether the educational applications we create are pedagogically effective. In addition, we are exploring how multi-touch interfaces can be used in mathematics and in improving workflow efficiency.

### Mathematical Sketching

Diagrams and illustrations are frequently used to help explain mathematical concepts. Students often create them with pencil and paper as an intuitive aid in visualizing relationships among variables, constants, and functions, and use them as a guide in writing the appropriate mathematics to solve the problem. However, such static diagrams generally assist only in the initial formulation of the required mathematics, not in "debugging" or problem analysis. This can be a severe limitation, even for simple problems with a natural mapping to the temporal dimension or problems with complex spatial relationships.

Mathematical sketching is a novel, pen-based, gestural interaction paradigm for mathematics problem solving, designed to overcome these limitations. Mathematical sketching derives from the familiar pencil-and-paper process of drawing supporting diagrams to facilitate the formulation of mathematical expressions; however, with mathematical sketching, users can also leverage their physical intuition by watching their hand-drawn diagrams animate in response to continuous or discrete parameter changes in their written formulas. Diagram animation is driven by implicit associations that are inferred, either automatically or with gestural guidance, from mathematical expressions, diagram labels and drawing elements.

• #### Corner Finding

IStraw is a new corner finding technique based on an analysis of the ShortStraw algorithm. Our analysis reveals several limitations in ShortStraw and we develop techniques to overcome them. We also present an extension to our corner finding approach for dealing with ink strokes that contain curves and arcs. An evaluation of our approach shows significant accuracy improvements over ShortStraw for polyline ink strokes with and without curves using an all-or-nothing accuracy metric while still maintaining ShortStraw's computational complexity.

##### Publications
• Xiong, Y. and LaViola J. "A ShortStraw-Based Algorithm for Corner Finding in Sketch-Based Interfaces", Computers and Graphics, 34(5):513-527, October 2010.
• Xiong, Y. and LaViola J. "Revisiting ShortStraw - Improving Corner Finding in Sketch-Based Interfaces", Proceedings of the Sixth Eurographics/ACM Symposium on Sketch-Based Interfaces and Modeling 2009, 101-108, August 2009.
• #### AlgoSketch

AlgoSketch is a pen-based algorithm sketching prototype with supporting interactive computation. AlgoSketch lets users fluidly enter and edit 2D handwritten mathematical expressions in the form of pseudocode-like descriptions to support the algorithm design and development process. By utilizing a novel 2D algorithmic description language and a pen-based interface, AlgoSketch users need not work with traditional, yet complex 1D programming languages in the early parts of algorithm development. Based on preliminary user feedback, we believe AlgoSketch has the potential to be used to design and test new algorithms before more efficient code is implemented. In addition, it can support users who may not be familiar with any advanced programming languages.

This work was performed in collaboration with Brown University.

##### Publications

Visualization of three-dimensional vector operations can be very helpful in understanding vector mathematics. However, creating these visualizations using traditional WIMP interfaces can be a troublesome exercise. VectorPad is a pen-based application for three-dimensional vector mathematics visualization. VectorPad allows users to define vectors and perform mathematical operations upon them through the recognition of handwritten mathematics. The VectorPad user interface consists of a sketching area, where the user can write vector definitions and other mathematics, and a 3D graph for visualization. After recognition, vectors are visualized dynamically on the graph, which can be manipulated by the user. A variety of mathematical operations can be performed, such as addition, subtraction, scalar multiplication, and cross product. Animations show how operations work on the vectors. A short, informal user study evaluating the user interface and visualizations of VectorPad was performed. VectorPad's visualizations were generally well liked; results from the study show a need to provide a more comprehensive set of visualization tools as well as refinement to some of the animations.

### Understanding User Performance with Pen-Based Interfaces

The pen-and-paper interface metaphor that pen-based computers provide make pen-based interfaces both natural and expressive. However, these interfaces can also be difficult to use without proper feedback and guidance in understanding how to perform the various gestures and pen-based commands for a particular application. This project focuses on understanding we can develop tools to assist users when working with pen-based UIs.

• #### Evaluation of Techniques for Visualizing Mathematical Expression Recognition Results

An experimental study was performed that evaluates four different techniques for visualizing the machine interpretation of handwritten mathematics. Typeset in Place puts a printed form of the recognized expression in the same location as the handwritten mathematics. Adjusted Ink replaces what was written with scaled-to-fit, cleaned up handwritten characters using an ink font. The Large Offset technique scales a recognized printed form to be just as wide as the handwritten input, and places it below the handwritten mathematical expression. The Small Offset technique is similar to Large Offset but the printed form is set to be a fixed size which is generally small compared to the written expression.

The experiment explores how effective each technique is with assisting users in identifying and correcting recognition mistakes with different types and quantities of mathematical expressions. The evaluation is based on task completion time and a comprehensive post-questionnaire used to solicit reactions on each technique. The results of our study indicate that, although each technique has advantages and disadvantages depending on the complexity of the handwritten mathematics, subjects took significantly longer to complete the recognition task with Typeset in Place and generally preferred Adjusted Ink or Small Offset.

This work was performed in collaboration with Brown University.

• #### GestureBar

GestureBar is a novel, approachable UI for learning gestural interactions that enables a walk-up-and-use experience which is in the same class as standard menu and toolbar interfaces. GestureBar leverages the familiar, clean look of a common toolbar, but in place of executing commands, richly discloses how to execute commands with gestures, through animated images, detail tips and an out-of document practice area. GestureBar's simple design is also general enough for use with any recognition technique and for integration with standard, non-gestural UI components. GestureBar was evaluated in a formal experiment showing that users can perform complex, ecologically valid tasks in a purely gestural system without training, introduction, or prior gesture experience when using GestureBar, discovering and learning a high percentage of the gestures needed to perform the tasks optimally, and significantly outperforming a state of the art crib sheet. The relative contribution of the major design elements of GestureBar is also explored. A second experiment shows that GestureBar is preferred to a basic crib sheet and two enhanced crib sheet variations.

This work was performed in collaboration with Brown University.

• #### An Empirical Study in Pen-Centric User Interfaces: Diagramming

A user study was performed that aimed at helping understand the applicability of pen-computing in desktop environments. The study applied three mouse-and-keyboard-based and three pen-based interaction techniques to six variations of a diagramming task. The key finding was that while the mouse and keyboard techniques generally were comparable or faster than the pen techniques, subjects ranked pen techniques higher and enjoyed them more. The results from a formal user study suggests there is a broader applicability and subjective preference for pen user interfaces than the niche PDA and mobile market they currently serve.

This work was performed in collaboration with Brown University.