Here is a short preview of the upcoming version of iPinchMe, called iPinchVU. It uses the handvu library for hand tracking:Drawing a box with your hand opens iTunes, moving left/right triggers play/pause or switches to the next track. I really hope to release this version soon, but I am incredibly busy right now. Anyways, thanks to everyone who is reading this and commenting. Without knowing that there are people out there appreciating this work, I would have stopped a long time ago.
I am proud to release a little side project of mine: MacTheClapper.
Description: MacTheClapper is a small program running in your system menu that listens on your Mac’s builtin microphone and executes a user defined applescript when it detects a double clap (or finger snap) sound.
Howto: Download the application and unzip. Start the program. A small “MC” icon will appear in your system menu. Click on the icon and choose “Settings”. A dialog will popup with wich you may configure MacTheClapper. Everything should be mostly self explanatory. Be sure to quit MacTheClapper and restart in case you changed the AppleScript command. Once you have configured MacTheClapper, click “Start” in the MacTheClapper menu. Clapping twice loud enough should now trigger the AppleScript command.
That’s it for now. Drop a comment if you liked the program or have some trouble using it.
For all of you who don’t know what the “clapper” is:
I’ve decided to stop experimenting and just use some mature open source code. Instead of using my own homebrew hand tracking code, the next version of iPinchMe will use HandVU. Results seem to be promising. Hopefully, I will find some time these days to release the new version. As HandVU is GPL-licensed, the next version of iPinchMe will be released together with all of its source code under a GPL license.
Creating programs like iPinchMe is a lot of fun, but ultimately one has to ask this question: What’s the point?
So, since I do not really know what I want to do next with iPinchMe, I am asking all of you (in case there actually are people reading this):
- How would you like to control applications with hand gestures?
- What is the interface of your dreams?
- Will gesture based input be a viable alternative to keyboard and mouse one day or just be confined to certain niches?
- What’s a killer-app for hand gesture based interfaces?
Tell me what you think!
P.S: To get your juices flowing, you may want check out this overview article at Venturebeat.
Edit: New Version available.
I decided to stop using background differencing and use a skin histogram based approach combined with the camshift algorithm. Since everybody was screaming Fluidtunes, Coverflow, etc. I am testing my current development version mainly with iTunes:Coverflow can now be controlled by a sort of waving motion. Songs are selected with the pinch gesture. Hopefully, this version of iPinchMe will soon be available for download.
Edit: New Version available.
I am proud to announce that the first beta version of iPinchMe is now available for download:
Link: iPinchMe - Beta
Requirements: Mac OS X 10.5 or later, Intel based Mac with builtin iSight.
This software is provided “as is” and in a HIGHLY EXPERIMENTAL state. I take no responsibility for whatever happens when you use it
Here is a short introduction on how to use it:
- Download and unpack the file.
- Start the program.
- While the prgram is initializing, step away from the camera, so that iPinchMe can learn the background. You can have it relearn the background, if you press the “Reinitialize” button (sometimes you have to press it a few times).
- Step back in front of the camera. The prgram should now recognize you as the foreground and draw a contour around you.
- Perform a pinch gesture (as shown in the iPhoto videos). Upon releasing the pinch, Expose should be triggered.
You should be in front of a static and more or less homogeneous background with as few dark areas as possible. Be sure to have enough illumination in your room. Otherwise iPinchMe will recognize false pinches or none at all, wildly triggering AppleScripts.Remember: iPinchMe will not work like magic! Computer Vision is a tough and computationally expensive problem…
How to bind other Apple Scripts to the pinches:
Upon the first start, iPinchMe will create the following file: ~/Library/Application Support/iPinchMe/ipinchme.script which contains an Apple Script function that is called from iPinchMe.
It will look like this:
if event_msg is equal to “one_pinch_release” then
tell application “System Events” to key code 101
Depending on what gesture you perform, the string event_msg will have a different content. Currently supported are the following gestures:
- event_msg = “one_pinch_down” (pinching with one hand)
- event_msg = “two_pinch_down” (pinching with two hands)
- event_msg = “one_pinch_release” (stop pinching with one hand)
- event_msg = “two_pinch_release” (stop pinching with both hands)
- event_msg = “one_pinch_release_left” (stop pinching with one hand on the left side of the captured camera image)
- event_msg = “one_pinch_release_right” (stop pinching with one hand on the right side of the captured camera image)
To bind one of these gestures to an Apple Script, for instance if you would like iTunes to play the next song when you perform the one_pinch_down gesture, you will have to add something like this into the body of the handle_event method:
if event_msg is equal to “one_pinch_down” then
tell application “iTunes” to next track
Apart from binding apple scripts to gestures, iPinchMe can also send keyboard events to other applications. This feature is however in a quite experimental state. So you should be really careful when to activate it!!! Only use it when you know what you are doing!!!!
Controlling Google Earth: Tick the checkbox “Send Keys” and then make sure that Google Earth is running in the foreground. In the same way you can also control TuxRacer or other apps that react on the left/right/up/down arrow keys. Look at the videos below, in order to get an impression on how move your hands.
I hope that you will have some fun with this small tool. In case you have questions, comments, bug reports, etc., just use the comments section. Ideas for future versions are always welcome.
Another possible application for iPinchMe:(I sure hope to find serious applications as well:-)
Seems that iPinchMe is able to control Google Earth:
Here’s a preview of the upcoming tool “iPinchMe”, it will allow you to bind applescripts to certain pinch gestures. Check out the video for a short demonstration. In the demo, a single pinch is bound to an AppleScript which send the “leftarrow key down” command to the currently active application, in this cae iPhoto. The result is a kind of hands free image browsing solution. If anyone likes this sort off stuff and/or has some ideas that could be incorporated in the final app: Just drop me a comment.