CC BY
37LEAP MOTION AS EXPRESSIVE GESTURAL INTERFACE
Turgali B.K.
Turgali Bauyrzhan Kuanishuly - Master, DEPARTMENT OF COMPUTER ENGINEERING AND TELECOMMUNICATIONS, INTERNATIONAL INFORMATION TECHNOLOGY UNIVERSITY, ALMATY, REPUBLIC OF KAZAKHSTAN
Abstract: motion-capture is a popular tool used in musically expressive performance systems. Several motion-tracking systems have been used in laptop orchestras, alongside acoustic instruments, and for other musically related purposes such as score following. Some examples of such systems include camera tracking, Kinect for Xbox, and computer vision algorithms. However, these systems lack the ability to track at high resolutions and primarily track larger body motions. The Leap Motion, released in 2013, allows for high resolution tracking of very fine and specific finger and hand gestures, thus presenting an alternative option for composers, performers, and programmers seeking tracking of finer, more specialized movements. Current third party externals for the device are noncustomizable; MRLeap is an external, programmed by one of the authors of this paper, that allows for diverse and customizable interfacing between the Leap Motion and Max/MSP, enabling a user of the software to select and apply data .streams to any musical, visual, or other parameters. This customization, coupled with the specific type of motion-tracking capabilities of the Leap, make the object an ideal environment for designers of gestural controllers or performance systems.
1. INTRODUCTION
Motion-capture devices are frequently used as interfaces for musical expression; the emotive gestures that one makes to interact with motion-capture systems make them effective in this role [1, 2, 3, 4]. Laptop Orchestras have created and implemented musical interfaces for Xbox Kinect, Computer Vision, and many other motiontracking devices. Prior to developing the discussed object for Leap Motion [5], the authors had experience both designing and creatively implementing a system intended for score following [6, 7, 8, 9]. This tracking used a Playstation Camera and blobtracking algorithms to determine the location of a player's hands in a determined space. Such tracking is limited to larger movements and in the type of data that can be obtained since smaller and subtler gestures cannot be adequately captured. A large gap in motion-capture devices released prior to the Leap Motion is in the ability to track fine motor gestures, like those a pianist would make during performance. The Leap, which is designed to function as a touchless mouse, provides a bridge in this technology gap and allows for hands-free motion-capture performance with high resolution and fine motor tracking.
2. THE LEAP MOTION
2.1. Leap Motion Development Objectives
The Leap Motion was designed by Leap Motion, Inc. and released in 2013. Leap Motion is a USB peripheral designed to create an invisible air space surrounding a computer screen that can be interacted with; it is essentially a touchless mouse. However, it is not designed to replace a mouse and keyboard and is intended to function alongside devices currently in use by a computer.
Interaction Area
2 feet above the controller, by 2 feet wide on each side (150° angle), by 2 feet deep on each side (i20a angle)
Fig. 1. Leap Motion field of view
2.2. Leap Motion as Musical Interface
The MRLeap object, which allows the Leap to interface with Max/MSP, is designed for users to extract generated data for musical or otherwise creative purposes. The fine motor control that the Leap is capable of tracking allows for a type of gestural control not present in motiontracking systems that target larger movements, such as the Xbox Kinect or computer-vision. Therefore, the external can lead to the creation of new interfaces for musical expression that are gesturally controlled, but quite different from large-movement dependent gestural controllers. Several data parameters are available (these are discussed in detail later in the paper) and the user is presented with a graphical display that provides information regarding which data streams are arising from which movements. The Leap tracks many different types of movements, so the diversity of data types is quite extensive. This diversity indicates that as a musically expressive device, the Leap is quite versatile and can be used artistically in several different ways, depending on the data output a given user is accessing.
3. MRLEAP DATA CAPTURE
Similar to the concept of a frame in video applications, the MRleap deals with the concept of analysis frames where subsequent and more detailed tracking information is embedded into the frame data structure. This top down architecture, allows for strong associative linking between tracked objects and their validity throughout their respective life cycles. Each element is aware of its parent ID and thus the user can easily compare validity between frames.
3.1. Frame Object
All tracking data is contained in a frame object, which includes basic frame information, motion data about the frame itself, hand data, and pointable data. Frame data includes the current frame ID, number of hands currently recognized in the frame, timing information, and fps. Motion data is calculated between the current frame and a specified earlier frame (default 1 frame in the past) and includes information about frame translation, rotation, and scale.
3.2. Hand Object
The hand object is part of the frame object and contains information about palm position, palm velocity, palm rotation, scale, translation, hand sphere (a sphere fit to the curvature of the hand), as well as information about pitch, yaw, and roll. As in the frame object, the hand object provides information about the current hand ID, frame ID the hand is found in, time since a hand has been recognized (in ms), and how many pointables are attached to the hand object.
3.3. Pointable Objects
Pointable objects are attached to a hand object. Pointables are defined as either a finger or a tool object. If a valid hand is found, the external will check for pointables and assign them as either a finger object or tool object. A tool is described as a pointable object that is longer, straighter, and thinner than a finger. The tracking data available for pointables are identical for finger and tool objects. Pointable direction, tip position, tip velocity, pointable dimensions (width, height), and touch zone are available. Like the hand and frame objects, pointables are also aware of themselves and their parent structures and provide their ID, hand ID, frame ID, and time they have been visible (in ms). Tool objects might be useful to track the movements of a conductor, or the instrument bodies, such as clarinet or oboe players. To facilitate easier tracking and increase the chances of reliable following, a pencil or straw might have to be attached to the instrument to make it "thin" enough for tracking.
3.4. Gesture Data
Alternatively, to getting raw data about the state of a current object, basic gestures are also reported by the external. The following gesture types are currently available:
• Circle: reports circle center, state (valid, starting, in progress, completed), turns, direction; as well as gesture ID, associated pointable ID, hand ID, and frame ID;
• Key tap and Screen tap: reports tap position and tap direction; same ID information as seen above;
• Swipe: report swipe position (state, start position, and current position) and swipe direction (speed, position); same ID information as seen above;
4. CREATIVE APPLICATIONS
4.1. Performance Capabilities
The Leap Motion is versatile and can be used as a gestural controller. The Leap not only tracks hands but any other objects that have reflective surfaces, allowing users of the objects quite a bit of freedom and customization when designing their own implementation of this device. The high-resolution tracking and relatively small area of tracking makes the Leap suitable for small and detailed performative gestures. The Leap cannot track at extreme distances from its origin; this attribute should be considered not as a limitation but as a performance tool and indication of which type of gestures and motions the Leap Motion tracks most effectively. Additionally, as many of the current motion-tracking technology available has a lower resolution and is designed to track a larger area (such as the Kinect), this makes the Leap a desirable alternative controller for anyone seeking to program an instrument that implements very fine, detailed performance motion. The Leap is also extremely portable and relatively inexpensive which makes the device desirable as a controller in performance and potentially obtainable by performance groups wishing to perform composers' works.
4.2. Works Composed for MRLeap - Piano Suite
The first work to implement MRLeap for Leap Motion successfully in a live performance setting was Hyperkinesis by Alyssa Aska [10], premiered on February 28, 2014. The piece is written for piano and Leap Motion, and uses tracking of the hands only. The hardware was a success in rehearsal and performance and received favourable response from the performer. The use of the Leap in performance was intuitive for the performer. The only issues encountered occurred prior to performance, including Max/MSP program crashes. However, these crashes appeared to be due to CPU usage; this result highlights why the MRLeap's ability to choose which data to track is such an important feature. Users with less powerful computers may otherwise be unable to interface with the Leap, as tracking all of the information draws considerable processing power. The work performed is part of a larger piano suite, containing five movements, each corresponding to one of the gestures of the Leap Motion. Aska's piano suite is designed to research gestural controller interactions with live instruments in performance, and test the viability of both the Leap motion and the MRLeap external. Successful performance reports to date indicate that the Leap works well
when paired with piano in performance so long as computer usage is carefully considered. Additionally, performing with the Leap and a live musical instrument presents logistical issues of Leap placement onstage. For the sake of these works a custom stand is being constructed which allows the Leap to be placed below but near C4 on a keyboard. Future performances of these works for piano and Leap motion are already programmed throughout the year, with the second performance occurring April 23rd, 2014.
5. FUTURE DEVELOPMENTS
Future improvements on the MRleap external are planned in the areas of rigid hand modeling to keep track of "dead" pointables. This would allow estimating the current pointable position even if the tracking algorithms have lost the object. Customizable gestures as well as user-defined gestures are being explored for future updates as well. Additionally, the authors hope that a future update to the SDK will allow for chaining multiple Leap Motion controllers. This idea arose due to the tracking field of the Leap Motion; in order to track the entire length of a piano multiple Leap Motion devices would be required. Future compositional and creative uses planned include a work for Accordion and Leap Motion to be premiered in Italy in July 2014.
6. CONCLUSIONS
The Leap Motion is an extremely new, very effective expressive device aimed at capturing small and detailed performance motions. MRLeap, designed by Martin Ritter in 2014, provides a solution to facilitating instrument design and interface with the Leap Motion by users wishing maximum customization and control. The external allows users to access all of the parameters tracked by the Leap Motion, and isolate specified parameters for performance and/or instrument design. The device has proven successful in concert performances to date, and due to the customizability, MRLeap has the potential for being a widespread tool that any person seeking to design an interface for musical expression may benefit from.
References
1. Rodriguez David and Rodriguez Ivan. "VIFE_alpha v.01 - Real-time Visual Sound Installation perfomed by Glove-Gesture." Paper, New Interfaces for Musical Expression.Vancouver, B.C. May 26-28, 2005.
2. Morales-Manzanares Roberto, Morales Eduardo F., Dannenberg Roger and Berger Jonathan. "SICIB: An Interactive Music Composition System Using Body Movements." In Computer Music Journal. Vol. 25. № 2, 2001. Pp. 25-36.
3. Kiefer Chris, Collins Nick and Fitzpatrick Geraldine. "Phalanger: Controlling Music Software With Hand Movement Using A Computer Vision and Machine Learning Approach". Paper, New Interfaces for Musical Expression. Pittsburgh, PA., 2009.
4. Bevilacqua Frédéric, Zamborlin Bruno, Sypniewski Anthony, Schnell Norbert, Guédy Fabrice and Rasamimanana Nicolas. "Continuous realtime gesture following and recognition", accepted in Lecture Notes in Computer Science (LNCS), Gesture in Embodied Communication and Human-Computer Interaction. Springer Verlag, 2009.
5. Leap Motion Controller Website. [Electronic resource]. URL: https://www.leapmotion.com/ (date of acces: April 15, 2014).
6. Ritter M., Hamel K., & Pritchard B. "Integrated Multimodal Score-Following Environment", Proceedings of International Computer Music Conference. Perth, Australia, 2013.
7. Litke D & Hamel K. "A Score-based Interface for Interactive Computer Music," Proceedings of the International Computer Music Conference. Copenhagen, Denmark. August 2007.
8. Cont Arshia. "On the creative use of score following and its impact on research" In: Sound and music computing. Padova, Italy, 2011.
9. Orio N., Lemouton S., Schwarz D. & Schnell N. "Score-following: State of the Art and New Developments", Proceedings of New Interfaces for Musical Expression. Montreal. Canada, 2003.
10.Aska A. [Electronic resource]. URL: http://www.alyssaaska.com/ (date of acces: April 15, 2014).
A COMPREHENSIVE LEAP MOTION DATABASE FOR HAND GESTURE RECOGNITION Turgali B.K.
Turgali Bauyrzhan Kuanishuly - Master, DEPARTMENT OF COMPUTER ENGINEERING AND TELECOMMUNICATIONS, INTERNATIONAL INFORMATION TECHNOLOGY UNIVERSITY, ALMATY, REPUBLIC OF KAZAKHSTAN
Abstract: the touchless interaction has received considerable attention in recent years with benefit of removing the burden of physical contact. The recent introduction of novel acquisition devices, like the leap motion controller, allows obtaining a very informative description of the hand pose and motion that can be exploited for accurate gesture recognition. In this work, we present an interactive application with gestural hand control using leap motion for medical visualization, focusing on the satisfaction of the user as an important component in the composition of a new specific database.
Keywords: hand gesture recognition, touchless interaction, leap motion, support vector machine.
I. INTRODUCTION
The user has dreamt for a long time to interact with a more natural and intuitive machine rather than with conventional means. For this reason, the gesture is the richest means of communication that can be used for human computer interaction [2].
The technology can prove to be interesting for applications requiring interacting with the computer in particular environments, like an operating room where sterility causes a big issue. The system is designed to allow a touchless HMI, so that surgeons will be able to control medical images during surgery. This paper presents a study of hand gesture recognition through the extraction, processing and interpretation of data acquired by the LM. This leads us to design recognition and classification approaches to develop a gesture library useful to the required system control. We introduce several novel contributions. We collected a new dataset formed by specific dynamic gestures related to recommended commands, and then we created our own data format. We propose a three-dimensional structure for the combination of spatial features like the arithmetic mean, the standard deviation, the root mean square and the covariance, in order to effectively classify dynamic gestures.
II. LEAP MOTION DEVICE
The LM is a compact sensor released in July 2013 by the Leap Motion Company [1]. The device has a small dimension of 80 x 30 x 12.7 mm. It has a brushed aluminum body with a black glass on its top surface, which hides three infrared LEDs, used for scene illumination, and two CMOS cameras spaced 4 centimeters apart, which capture images with a frame rate of 50 up to 200 fps. This depends on whether USB 2.0 or 3.0 is used. The LMC allows a user to get information about objects located in a device's field of view (a format similar to an inverted pyramid, whose lower area length measures 25mm and the top one 600mm, with 150° of field of view).
