ISUE
Interactive Systems & User Experience Lab
One of the primary research themes in the IS&UE lab is developing and evaluating 3D user interfaces for virtual, augmented, and mixed reality. In particular, we are focused on exploring how to bring 3D user interface techniques and concepts into mainstream video games by leveraging the existing body of work in 3DUI and VR and devising new strategies and methodologies for bringing spatial 3D interaction to gamers. Additionally, we are interested in the continued learning and understanding of how humans interact with and are affected by 3D interfaces.
With the release of a variety of new motion controllers for both PC and console gaming, 3D user interfaces are becoming commonplace in modern games. The focus of this work is to explore how to best utilize 3D spatial interaction in the video game domain by examining existing interaction techniques and creating novel ones as well as understanding how these interfaces affect users.
3D object selection is highly demanding when, 1) objects densely surround the target object, 2) the target object is significantly occluded, and 3) when the target object is dynamically changing location. Most 3D selection techniques and guidelines were developed and tested on static or mostly sparse environments. In contrast, games tend to incorporate densely packed and dynamic objects as part of their typical interaction. With the increasing popularity of 3D selection in games using hand gestures or motion controllers, our current understanding of 3D selection needs revision. We present a study that compared four different selection techniques under five different scenarios based on varying object density and motion dynamics. We utilized two existing techniques, Raycasting and SQUAD, and developed two variations of them, Zoom and Expand, using iterative design. Our results indicate that while Raycasting and SQUAD both have weaknesses in terms of speed and accuracy in dense and dynamic environments, by making small modifications to them (i.e., flavoring), we can achieve significant performance increases.
We are exploring full body navigation for video games through a variety of means. In RealNav, we developed a reality-based locomotion control system of the quarterback in American football. Focusing on American football drives requirements and ecologically grounds the interface tasks of: running down the field, maneuvering in a small area, and evasive gestures such as spinning, jumping, and the "juke."
The locomotion interface is constructed by exploring data interpretation methods on two commodity hardware configurations. The choices represent a comparison between hardware available to video game designers, trading off traditional 3D interface data for greater hardware availability. Configuration one matches traditional 3D interface data, with a commodity head tracker and leg accelerometers for running in place. Configuration two uses a spatially convenient device with a single accelerometer and infrared camera. Data interpretation methods on configuration two use two elementary approaches and a third hybrid approach, making use of the disparate and intermittent input data combined with a Kalman filter. Methods incorporating gyroscopic data are used to further improve the interpretation.
We are further studying how fully body interfaces might be developed for video games; using a Wizard of Oz approach and the commercial video game Mirror's Edge, players are presented with several different tasks such as running, jumping, and climbing. Following our protocol, participants were given complete freedom in choosing the motions and gestures to compete these tasks. Our experiment results show a mix of natural and constrained gestures adapted to space and field of view restrictions. We present guidelines for future full body interfaces.
RealDance is a prototype video game for exploring spatial 3D interaction for dance-based gaming and instruction. Our interface uses four Nintendo Wii remotes and is independent of buttons, floor position, cameras, or sensor bars so the user is untethered, allowing for natural, full-body motion. Our range of detectable movements includes stationary poses, punches, kicks, claps and stomps, which are scored in the context of the dance routine.
The focus of this project is to explore how technologies that have traditionally been found in virtual reality but are now becoming mainstream in the commerical marketplace affect user performance in video games. Specifically, we are interested in whether 3D stereo as well as head and hand tracking improve a player's ability to learn to play video games and achieve better scores. In addition, we are exploring overall user experience when players use these technologies.
This is a systematic study on the recognition of 3D gestures using spatially convenient input devices. Specifically, we examine the linear acceleration-sensing Nintendo Wii Remote coupled with the angular velocity-sensing Nintendo Wii MotionPlus. We created a 3D gesture database, collecting data on 25 distinct gestures totalling 8500 gestures samples. The experiment explores how the number of gestures and the amount of gestures samples used to train two commonly used machine learning algorithms, a linear and AdaBoost classifier, affect overall recognition accuracy. We examined these gesture recognition algorithms with user dependent and user independent training approaches and explored the affect of using the Wii Remote with and without the Wii MotionPlus attachment.
The Bespoke 3DUI XNA Framework is an open-source software platform for research in 3D user interaction. The Bespoke 3DUI XNA Framework distinguishes itself from other platforms, in that it provides 3D user interface machinery in a game development framework. This combination leverages lowcost, widely available game technologies, enabling researchers to investigate 3DUI techniques, and providing game developers a foundation for prototyping 3DUIs in commercial video games.
In recent years the popularity of music and rhythm-based games has experienced tremendous growth. However, almost all of these games require custom hardware to be used as input devices, and these devices control only one or two similar instruments. One Man Band, a prototype video game for musical expression uses novel 3D spatial interaction techniques using accelerometer-based motion controllers. One Man Band provides users with 3D gestural interfaces to control both the timing and sound of the music played, with both single and collaborative player modes. One Man Band further investigates the ability to detect different musical gestures without explicit selection of mode, giving the user the ability to seamlessly transition between instrument types with a single input device.
With the rise in popularity of 3D spatial interaction in console gaming, such as the Nintendo Wii, it is important to determine whether existing menuing technique findings still hold true when using a 3D pointing device such as the Wii Controller. This project compares linear menus with two other menu techniques: radial menus and rotary menus. We measure effectiveness through task completion time and the number of task errors. User study results indicate that radial menus are an effective menu technique when used with a 3D pointing device.
Non-isomorphic rotational mappings have been shown to be an effective technique for rotation of virtual objects in 3D desktop environments. However, it is unclear how non-isomorphic 3D rotation techniques transition into immersive virtual environments. This project aims to determine how various components, such as head tracking and stereo, affect user performance when rotating virtual 3D objects.
This work was performed in collaboration with Brown University.
The Turn Table peripheral was designed for the FIEA PC video game Sultans of Scratch.
In the game the player must emulate scratching a left and right disc, in addition to slamming a slider knob, in order to match on screen queues to earn points in a tag team style match between other players. The peripheral is composed of an Arduino microprocessor board that relays information about disc rotation and slider location to the video game. Disc rotation is capture with two optical mice that provide counter and counter-clockwise change in rotation values. This paired with a potentiometer's voltage output for the slider's location, enables the game to make the player feel like an actual DJ.
We present a prototype system for interactive construction and modification of 3D physical models using building blocks. Our system uses a depth sensing camera and a novel algorithm for acquiring and tracking the physical models. The algorithm, Lattice-First, is based on the fact that building block structures can be arranged in a 3D point lattice where the smallest block unit is a basis in which to derive all the pieces of the model. The algorithm also makes it possible for users to interact naturally with the physical model as it is acquired, using their bare hands to add and remove pieces. We present the details of our algorithm, along with examples of the models we can acquire using the interactive system. We also show the results of an experiment where participants modify a block structure in the absence of visual feedback. Finally, we discuss two proof-of-concept applications: a collaborative guided assembly system where one user is interactively guided to build a structure based on another user's design, and a game where the player must build a structure that matches an on-screen silhouette.
This is a proof-of-concept implementation of a handheld stereo projection display system for virtual worlds. We utilize a single pico projector coupled with a six DOF tracker to generate realtime stereo imagery that can be projected on walls or a projection screen.
This is a novel, real-time, markerless vision-based tracking system, employing a rigid orthogonal configuration of two pairs of opposing cameras. Our system uses optical flow over sparse features to overcome the limitation of vision-based systems that require markers or a pre-loaded model of the physical environment. We show how opposing cameras enable cancellation of common components of optical flow leading to an efficient tracking algorithm. Experiments comparing our device with an electromagnetic tracker show that its average tracking accuracy is 80% over 185 frames, and it is able to track large range motions even in outdoor settings.
The Virtual Experience Test (VET) is a survey instrument used to measure holistic virtual environment experiences based upon the five dimensions of experiential design: sensory, cognitive, affective, active, and relational. Experiential Design (ED) is a holistic approach to enhance presence in virtual environments that goes beyond existing presence theory (i.e. a focus on the sensory aspects of VE experiences) to include affective and cognitive factors.
To explore the 3D file browsing technique design space, we analyzed the existing literature and developed three representative 3D file browsing techniques that cover many of their characteristics. Block3D uses a priority weighting scheme to elevate and display files in a grid-based structure. Cluster3D uses sets of animated racks to display files. LTreeCube3D visualizes files and directories using groups of semi-transparent cubes within a larger cube-like structure. An experiment was conducted exploring the effect these 3D file browsing technique have on users in a manual file searching task. The results indicate that users completed the manual file search task significantly faster using Block3D than both LTreeCube3D and Cluster3D. Although subjective ranking showed users preferred the Block3D technique, user feedback also showed merits of the other techniques.
