Yay

I just tried out a new webcam lens today, and I'm happy to say it's way brighter now. I no longer need to keep the gain cranked on the highest possible setting and the sensitivity is better. Turns out the lens I was using from my 'Infrared Webcam' still had an IR filter coating on it.

Images are now like this even for light touching

It's bright enough now that fingerprint grease smudges show up enough to be detected. I'm gonna need to work on some adaptive background removal soon.

Multitouch Screen Progress 11

Work continues on the touchlib software. I added a wrapper for the DSVideoLibrary which performs a bit better on my machine than video wrapper (though I'm still not sure why exactly) and also adds support for firewire cams. Unfortunately CPU usage is still higher than I would like (30-40% just to capture the video at 640x480 @ 30 fps, though 20% of that is wasted waiting for mutexes). Anyone out there an expert in optimizing multithreaded apps? I would probably have to get a firewire cam or maybe a nice PtGrey camera if I want lower CPU usage. Those cams can get a bit pricey so I'll have to hold off on that for now.

Otherwise, things are working pretty well. I haven't touched the hardware for a while. I need to optimize some of my blob-tracking routines (right now the code can't handle more than 8 blobs at once) and I've also started thinking seriously about what kind of software I want to work on to showcase. I'd definately like to make at least one 'game' app for the screen. I realized some things that are unique to working with this kind of multitouch screen when it comes to designing games:

• Everyone can see what the other players are doing
• Everyone can interact at once
• There is no distinction between players (ie, we can't identify finger presses by player) unless we explicitly define screen areas for each player
• Multi-finger gestures can become more difficult to detect because of the last point

Because of these points, the screen may lend itself more to cooperative games where distinguishing between players and secrecy don't matter.

I think for now I will do some kind of air hockey / pong game first, which should be pretty easy to write.

Probably the more exciting applications are single user. I have my 3d modelling app, Claymod which I would like to convert to be multi-touch. Multitouch would let you directly squeeze, pinch and nudge the 3d models. I also have a music sequencer / synthesizer application which I did a lot of work on but isn't in a usable state yet. This might also make a cool multi-touch application, especially for live use. Both use Irrlicht as a low-level GUI, so the work of making them multi-touch enabled is partially shared.

White and Nerdy

I haven't seen a wierd al video in a while, but this one reminds me why I love him.

My fellow nerds will appreciate:

Videowrapper added to touchlib

I've integrated VideoWrapper into touchlib. VideoWrapper is a very easy to use, high-level video capture library that includes support for Directshow, and several manufacturer specific camera sdk's. These include Videre Design, Point Grey and Canon. Check these out if you are looking for a more professional video capture solution that provides high frame rates and resolutions. VideoWrapper is also extensible with DLL's so adding new SDK's is a breeze. I still have a bug to work out, but once that is fixed I should have my camera running at it's full frame rate and resolution.

First Videos

Here's a couple videos of me using my screen. FYI, my camera is still running at 320x240 @ 15 fps, so any slowness or innacuracy you can attribute to that. ;)

Touchscreen Progress 10

I got my screen back up and running again after juicing it up with 3x the LED's (old screen had 14 LEDs, new one has 42). This time I mounted my LEDs on a strip of plastic instead of sanding them and gluing them to the screen with UV Glue. Interestingly, when I first hooked it all up it wasn't any brighter (except at the edges) than using 14 LEDs. I figured that it must be that the light is not getting dispersed enough - my LEDs are a very tight angle (16 degrees). To fix this I sanded the heads flat which seemed to help a lot. Next thing I might try is making them concave with my dremel instead of flat which might help dispersion even more. The moral of the story - wider angle led's are better. Now that the LED's aren't glued to the screen it's a lot less fragile and easier to tear down and mess with.

I also ended up getting rid of my paper + plastidip surface and going back to the design outlined in Jeff Han's paper. I put a diffuser (I sanded a thin piece of plastic) to act as the screen. The diffuser helps reduce background noise and also eliminates one of the big problems I was having with the paper surface which was specular glare from the projector on the bottom of the acrylic screen. The other problems with the paper are that it gets wrinkly once the rubber is on it and it's not the nicest tactile surface for heavy use. Still, I think it might be possible to find a similar surface. A surface has to have the following properties:

• It has to be flexible enough to register individual finger presses and not be too hard to press
• Has to be rigid enough to keep it's shape (not too stretchy - esp if coated).
• It has to be usable as a rear projection surface
• Has to scatter IR (can't be IR transparent).
• has to feel good to touch and touch and drag

Any surface can be coated with the plasti-dip to ensure that it couples well with the acrylic screen.

Sensitivity is a lot better now for me, but it's still not up to the level that you can see in Han's multitouch video. Improving that will be my next priority. I have a new IR filter which should cut down on background noise. I'll also make some improvements to the blob detection code (detect more blobs but reject more based on criteria).

Multitouch screen links of the day

• DIY Multitouch - A bunch of links and info on various multitouch techniques.
• Khronos Projector - Uses infrared + a stretchy lycra surface and detects the depressions on the surface.

I've also been toying with the idea of making a 'glove' with infrared lights in the finger tips. Ideally the LED's would light up as you press down on the screen, and would get brighter as you press harder. XBox controllers have pressure sensitive buttons in them - maybe I can cannibalize those parts and try something out.

Work continues on the software. I've got some ideas for applications I'd like to try. I'm in the middle of revamping my screen to add more LEDs and testing different screen surfaces / projection methods so I won't be able to test anything for the moment though.

More Multitouch links

• Smartskin - uses capacitance testing to detect conductive objects, such as skin. Sounds like the best choice for a small form factor. Could fit over or be integrated with an LCD monitor.
• Malleable touch surface - I'm guessing this is what the Hyperfabric uses. It's another computer vision approach.

Multitouch roundup

Futnuh posted a nice vid of our multitouch code in action on his site (near the bottom). Check it out.

Here's one of the demos featured on Han's multitouch screen, in more detail. Rigid shape manipulation.

Hyperfabric art installation. Wish I knew how it worked more. Stretchy fabric is one thing I was going to try as a screen surface.

Warcraft 3 multitouch. Nice. I'd like to be able to get this running on my screen (even if it's only single touch)

More touchscreen progress

Here's some more examples of interactive table goodness (thanks Futnuh). We now have directshow video capture working, thanks to Hebnern. I had Futnuh's interactive smoke demo running on my table yesterday (pics forthcoming) and it was sweet. It's fun to aim two smoke blobs at each other and watch them collide. The biggest issue I have to overcome is finding a way to make the surface more sensitive. More LED's, better optics, and different 'compliant surface' set ups are things to try. I'll be testing out some more things soon, and hopefully getting a chance to work on some demos of my own. I've learned a lot putting this first prototype together so hopefully my next one will be even better.

Success!

Woo! Today was a great day for my touchscreen. I got it all put together, complete with some aluminum pieces to protect the LED's. I also got the calibration software working (though it still needs some fixes). Check out my flickr set for pics. There is still a lot of work to do, but for the first time I have the screen projecting and detecting in synch.

Entertaible

Futnuh brought my attention to this interesting item: the Entertaible. The article says it uses IR Led's and receptors around the edges. I'd be interested to know more about how it works. I'm guessing the resolution is limited to the number of LED's and so it's probably pretty low. Cool stuff though. I'd like to make some similar software where multiple people can play games on a table like surface. The combination of real game pieces and an LCD table is also interesting (but it has been done before).