paranoid blog

iPinchMe - Control your Mac using simple Hand Gestures

iPinchMe is a small tool, that uses your Mac’s builtin iSight to capture your finger movements and executes user definable actionscripts when you perform simple hand movements.

Download iPinchMe
Requirements: OS X 10.5 or later, USB Webcam
Download Link: ipinchme.zip

Disclaimer:
This software is provided “as is” and in a HIGHLY EXPERIMENTAL state. I take no responsibility for whatever happens when you use it

Here is a short introduction on how to use it:

1. Download and unpack the file.
2. Start the program.
3. While the prgram is initializing, hold your palm over the red box, so that it is fully covered. Instead of your hand, you can also use a different object. The only requirement is that the object’s color is not to close to the background color.
4. Once the initialization is finished, iPinchMe will track your hands movements, as can be seen by the blue/yellow ellipse drawn over your hand.
5. Try to move your hand a little and watch out for messages like “Last trigger: left” or  “Last trigger: down”, which mean that iPinchMe thinks you moved your hand to the left or to downwards respectively. It will probably take a little time until you are used to how fast or how slow you will have to move your hand in order for the movement to be recognized.
6. By moving your hand to one of the three boxes on top you can set iPinchMe’s state. This state can either be “green”, “blue” or “yellow”. More about that in the section about binding Applescripts to the gestures.

General Notes:
You should be in front of a static and more or less homogeneous background. Be sure to have enough illumination in your room. Otherwise iPinchMe will recognize false pinches or none at all, wildly triggering AppleScripts.Remember: iPinchMe will not work like magic! Computer Vision is a tough and computationally expensive problem…

How to bind other Apple Scripts to the pinches:

Upon the first start, iPinchMe will create the following file: ~/Library/Application Support/iPinchMe/ipinchme.script which contains an Apple Script function that is called from iPinchMe.

It will look like this:

on handle_event(event_msg,state )

end handle_event

Depending on what gesture you perform and what state is activated, the strings event_msg and state will have a different content. Currently supported are the following gestures:

1. event_msg = “left” (swift left movement)
2. event_msg = “up” (swift up movement)
3. event_msg = “right” (swift right movement)
4. event_msg = “down” (swift down movement)

The state variable’s content will either be “none”, “green”, “yellow” or “blue”. To bind one of these gestures to an Apple Script, for instance if you would like iTunes to play the next song when you perform the “right” gesture, you will have to add something like this into the body of the handle_event method:

if event_msg is equal to “right” then
tell application “iTunes” to next track
end if

With the different possible states you can now modify iPinchMe’s behavior to - for instance - execute iTunes related scripts only when the state is set to “blue”. My current ipinchme.script looks like this:

on handle_event(event_msg, state)

if state is equal to “blue” then

if event_msg is equal to “right” then
tell application “iTunes” to activate
tell application “iTunes” to next track
end if

if event_msg is equal to “left” then
tell application “iTunes” to activate
tell application “iTunes” to previous track
end if

if event_msg is equal to “down” then
tell application “iTunes” to activate
tell application “iTunes” to playpause
end if

end if

end handle_event

With the above setup you can switch between songs using the left/right gesture and play/pause using the down gesture:

That’s it. I hope that you will be able to play around and experiment a little now. If there are some problems, do not hesitate to leave a comment on my blog.