Everything you need to know to begin developing multi-touch applications
As I continue making progress on the hardware systems and fabrication of a new multi-touch surface I come across a lot of people who are more interested in developing programs for the surface than the engineering side of things. I usually just send a pile of links to these people, but for future reference I thought it’d be nice to catalog everything I’ve come across and found useful over the last couple years.
http://nuigroup.com
Importance: This is the central hub for a loosely organized group of students and hackers from all over the world building multi-touch surfaces, software and experimenting with new techniques.
Members of this community were responsible for the most popular open-source feature tracking software out there: Community Core Vision (CCV). They also sponsor a good number of students every summer for the Google Summer of Code to develop new features and push boundaries.
Pages of interest
NUI Group Wiki
Community forums
Collection of MT knowledge (PDF)
http://tuio.org/
Importance: Homepage of the ‘glue’ that holds MT architecture together. Learn everything you need to know about how to develop multi-touch applications here.
TUIO is the protocol which enables multi-touch information to be sent from a tracking program to your actual application. Think of this metaphor: your multi-touch surface is like your internet router, modem and internet cables – just the hardware that you have to plug in to get started. Once your hardware is set up then you can send and receive information using packets of information – for the internet thats TCP/IP, also known as internet traffic. In a multi-touch system packets of information are sent using TUIO. Its just a live stream of data thats constantly reporting the location and state of all the objects (fingers or otherwise) that are being detected by your webcam.
Pages of interest
Software page w/ links to all known frameworks, APIs and tracking software
Showcase of open-source projects that you can download and learn from
http://ccv.nuigroup.com/
Importance: Open-source tracking software that is used to actually detect your fingers, identify them and then generate TUIO packets containing that information.
In every multi-touch system you will find to very important things – the hardware itself and the computer vision software that makes that hardware useful. CCV looks at webcam imagery and uses computer vision technology (specifically OpenCV) to pick out and identify useful features, like fingers. It then sends that information out to your machine automatically using the TUIO protocol. One thing you have to do is learn what all the sliders and checkboxes mean in the program, so I suggest running the program and just playing with everything so you can see what effect the parameters have on the resulting image.
Pages of interest
The basics of using CCV
NUI Group forum thread on building your own mini MT surface
Once you’ve got your feet wet in learning about multi-touch programming you’ll need a device that you can use to test and debug your application without having to use the big surface on campus. This forum thread will guide you through building a miniature MT surface that several people at UNK have had great success with. You just take some transparent material (a picture frame without its backing works nicely) and stick it on top of a cardboard box, then stick a webcam inside the box.
Once you’ve got the actual box put together you have to run and configure CCV properly to detect your fingers. The most important thing to remember is that this surface does not actively illuminate your fingers – so you need to Invert your image in CCV so that it’s looking for dark spots instead of light spots.
http://peauproductions.com
Importance: Collection of practical hardware articles and seller of modified PS3Eye webcams and miscellaneous other hardware.
Although there are three free PS3Eye webcams available in the Kelly Grant cart in OTOL 117 you may want to buy your own someday. You can find other suppliers, but the important thing is that Peau Productions can actually modify the camera for you for use with infrared.
Originally the owner of this site was known for modifying and reselling PS3Eye webcams (removing the stock IR filter and adding an M12 camera mount with bandpass optics) and now he’s grown his business to include other MT-specific products. He also has a great collection of articles and information on his site about different materials and parts of multi-touch systems and how well they work.
http://sparkon.net/
This site is pretty new, so its still got a little ways to go before it contains a LOT of useful info, but this site is actually more of a directory of projects than a directory of information for beginners. Just surf around and find projects that look interesting and follow their links – many times you’ll find links to source code so you can start picking it apart to learn.
Java-specific resources
Although you can use any language you want to develop MT applications I know a lot of you are just starting out in programming and are most comfortable in the Java + eclipse environment. So here’s a collection of links that might help you out:
Pages of interest
Making a multi-touch applet using TUIO (CSIS 150 lab)
MT4j – multi-touch framework for Java
TUIO documentation for Java library
With or without the above resources, the best thing you can do to learn new information is to try!
Sometimes a bottom-up approach can be very confusing and off-putting, and this may be one of those situations. If you’ve gotten to this point and you’re still confused then maybe you need a different approach. Try imagining a program that you can play with on a multi-touch surface – it can be extremely simple, like a single white box that you can touch and move around.
- Create this program without even thinking about using multi-touch, only the mouse.
- Once this is working, download the TUIO library for whatever language you are developing in and import it into your program.
Refer to the TUIO documentation for how to use it in your program and maybe check out examples of other programs that you can steal functions or variables from - Get access to a surface either by coming to campus or building an MT Mini and hook up its webcam and whatever else it might have to your computer.
- Run CCV and start tweaking the sliders and buttons until you’ve got a nicely filtered image.
Whenever you touch your surface a white circle should show up with a number next to it. - Now run your program while CCV is running and try to get things to work together!