Jackknife: A Reliable Recognizer with Few Samples and Many Modalities

Jackknife is a general purpose gesture recognizer for C++, C#, and JavaScript that is designed to work with a variety of input devices including Kinect, WiiMote, Leap Motion, as well as touch and stylus devices. Our recognizer can even be used to detect hand gestures passing through inaudible sound waves due to the doppler effect. If the input can be represented as a sequence of points through time, there is a good chance that our recognizer can be used to detect input patterns with high accuracy. Jackknife is also designed for gesture customization. This means that a user or developer only needs to provide one or two examples of a gesture pattern to work, whereas other recognizers require lots of training data, which may be quite difficult to collect.

Reference Source: Jackknife Repository
JK2017 Dataset: Jackknife Repository
Paper: PDF


Some Technical Details

Jackknife is a collection of dynamic time warping (DTW) based techniques tailored for gesture customization. That is, Jackknife is designed to work well with only one or two samples per gesture class for many different input devices.

As an overview of the underlying methods and process, we first resample an input gesture to a fixed number of points, similar to $1 [1]. We then measure the similarity of the given input to each gesture class template using DTW, where the local cost function measures the inner product of corresponding gesture path direction vectors, per Penny Pincher [2]. Motivated by the complexity-invariant distance (CID) [3], we further augment the DTW score with a set of new correction factors designed specifically for gestures. Finally, the input sample is said to belong to the gesture class whose template yields the best score. When dealing with continuous data, we also find an appropriate rejection criteria by selecting the threshold which maximizes the F1 score of synthetically generated distributions. Negative samples are generated by splicing together template gestures and positive samples are generated with gesture path stochastic resampling (GPSR) [4].

Further details, including information on various optimizations, can be found in the Jackknife paper.