Touchscreen Progress 9

Just posted another flickr shot of my screen in it's current state. Work continues on the software - I'm currently putting together the config app (which will handle calibration).

Touchscreen progress 8

I finished coating a large piece of paper with plasti-dip last night and this morning I tried it out on my screen. I posted a short but rather large video here (hi-res). The big bright spot is actually caused by the projector and the other glare might be from an open window next to it. I will need to experiment to see how to remove the projector glare spot. Maybe a diffuser or a more Matte coat on the bottom of the screen would help. Overall though I am pretty pleased. The finger spots are bright and it works well enough as a projection surface too. It didn't come out quite as transparent as my first test did, but that's probably because I used better quality paper. Now that I have this done, I can seriously work on a configuration application and try to get the projected image synced up with the detected finger presses.

plasti-dip experiments

I conducted an experiment using paper coated with plasti-dip as a possible screen surface. It went pretty well - I think I will go ahead and try a screen sized piece next. I get a nice bright spot (just as bright as using your fingers on the bare screen) plus you don't have to press quite as hard and it isn't dependant on whether you have dry fingers or not. Also it seems like there is minimal hysterisis (sticking). As a bonus, the plasti-dip makes the paper more transparent which makes it brighter as a projection surface. Spreading this stuff out might be a problem - they sent me the canned version instead of the spray version and it starts drying fairly quickly. I may take Darran's suggestion and use a flat paint roller.

I also finished putting together the stand - I chose to use PVC. I'm going for a flat 'table' style screen. I'll post some pics soon.

I also realized that my projector is running at TV resolution since I'm using S-video cables. I need to find the proper computer cable for it online somewhere which should be fun considering that this is a fairly old projector. Ebay here I come...

Touchscreen Progress 7

I found this interesting article last night. Here's the important part:

The amount of light that can be drawn out with FTIR is dependent on many things: wavelength, airgap thickness, refractive index of each medium, and angle of incidence.

Interesting stuff. Here's my experiment using paper + plastic wrap as a screen surface. Note that there is a fair amount of light in the room (it's right next to window) but the paper flattens out the background noise nicely and acts as projection surface. On the downside, the contrast is not very high. A better filter on my camera might also help.. I'm still using photo negatives.

I ordered some Plasti-dip (spray rubberizer) which should be arriving today. I'll be doing some more tests.

Touchscreen Progress 6

I set up a flickr account for posting some pictures of my multitouch progress. Just a few pics up there so far, but I will be adding to it.

In other news - progress continues on the software. Next step is to set up a configuration app which will let you customize your settings and 2d mappings. This is a bit hard since none of us have reached the point where we have a fully working screen. The last big hurdle we will face is figuring out what to use for the projection screen or using a 'compliant' surface of some kind. Also, I may need an IR filter for my projector (an IR remover not an IR pass). Futnuh has found a screen material which is fairly transparent to IR which is good news. The IR filter on the projector is necessary because even if the projection screen is transparent to IR, your hand ends up becoming lit up by IR which is no good. I'm going to continue doing tests to find a suitable compliant surface because I think it has the potential to solve a bunch of problems. Right now you have to press pretty hard (or have moist hands) to get a nice IR dot on the camera. Also, I end up having to use the screen in the dark because all the lights in my house put out a lot of IR. Ideally, the screen should be usable in any environment. A good overlay would be able to produce a nice spot without having to press too hard or have moist hands and could potentially block out the room IR, producing a nice flat, easy to analyze image. Plus it could act as the projection surface at the same time. I think it's just a matter of finding the right material.

In other news, I think I found a camera which should be ideal for our purposes. It's the Philips SPC900NC. It's a CCD camera that purports to offer up to 90 frames per second (for 320x240 no doubt). It looks like it can still do 640x480 @ 60 frames per second which is great. It really is a great webcam compared to the cheaper ones I've been playing with lately - super sharp image, lots of options and very impressive frame rates. The lense assembly is not too hard to remove once you know how - it's a little scary at first because you realize that the thing has absolutely no visible screws anywhere. But really all you have to do is pop off the grey lense cap and unscrew the lense assembly. And now for the funny part. I tried to remove the IR filter from the lense assembly. It unfortunately has the kind of filter which is pretty much impossible to remove. The back lense is glued to the IR filter (which is also a lense). I tried scraping off the IR coating but it absolutely wouldn't come off. Then I tried burning it off with a torch and it flaked off - but the lense cracked and it was pretty much impossible to get the thing back together. At that point I got that sinking feeling when you know you just flushed 99$ down the toilet. However, I realized that all I needed was new lense assembly that doesn't have an IR filter on it. So, I unscrewed the lense from my much cheaper (20$) 'nightvision webcam' and guess what, it screws on the philips camera and it works fine! plus there is no IR filter. I wish I hadn't destroyed the original lense before realizing this, but I guess that's life.

Comp USA is offering 20$ off the camera right now, so if you live in the states, pick one up.

another multi-touch concept..

Just thought I'd share another idea using the FTIR multi-touch technology: the multi-touch touchpad/mouse. Basically it works the same as touchpads on laptops do. I imagine that it is fairly small, maybe 4-5 inches square. You can make little sweeping motions to move the mouse cursor around as with touchpads, but you can also touch with multiple figers in order to produce scaling and rotational information. Basically instead of just producing 2d movement information (as a regular touchpad) it would produce a transformation matrix containing translation, scale and rotational information. You could even scale along an axis by touching with 3 fingers and keeping one stationary while moving the other two. It would be easier to construct and have a smaller (portable) form factor. You don't need a projector. You could probably construct it using a webcam fairly close to the acrylic - maybe a fisheye lense would help. So many applications feature transformation (move/rotate/scale) - 3d modelling aps, 2d vector art apps, paint apps, music apps, etc. It might be a really nice, intuitive input device.

svn updated

I checked in a bunch of code changes to the SVN - some major refactoring to make things cleaner, logical and flexible. The next thing we should work on is getting background removal, smoothing and whatever other filters we want to include.

Touchscreen Progress 5

I ran some tests on my screen with the LED's attached and it's looking pretty good so far. My one concern is background brightness and also having to press hard or use moist fingers to get a nice spot. I'm going to try adding some additional LED's to see if that helps and also look into getting an IR pass filter, but I'm wondering if using a 'compliant surface' (as Han says) might not be better. This could virtually eliminate background noise as the surface could block outside IR sources and could act as a screen for the video at the same time. I'll need to run some tests. I am currently eyeing up a fairly cheap projector screen on ebay that ends soon. Hopefully it will stay affordable.

Blob detection and tracking - first test

Click Here for the first test of my blob detection code in action. Notice it picks up some noise towards the end - this should be fixed when I add some adaptive background removal. Additionally there are some threshold levels that can be played with that affect the detection. Overall it works well - we'll need some more complicated tests like Jeff does on his presentations - like multiple fingers from each hand moving near each other. Those are the harder cases to resolve. Thanks again for the test video, futnuh.

Touchscreen Progress 4

The software is coming along nicely - I got the connected components algorithm working for blob detection (needs to be refined a little bit, but otherwise it's working well) and I also got what I think is a good algorithm for the blob ID tracking. Basically the algorithm is to figure out all the possible tagging scenarios given the current set of blobs on the screen and picks the one that would introduce the least total error (the smallest change from the last frame). It seems to work pretty well so far, but I've only tested it by tracking the lights in my house using my webcam.

I also got my LED's glued onto my screen. The UV glue ended up working fine in regular sunlight. Some internet sites warned against doing this but I had no problems. It took an hour or two to cure in partly cloudy weather. With a real UV lamp it should cure in under 30 seconds which would be pretty cool if I actually had one. Once I get them soldered up I can start doing some real tests.

Touchscreen Progress 3

I sanded flat my LED's yesterday - hopefully my UV glue will arrive today or tomorrow so I can get them attached. I also ordered some aluminum extrusions to use for framing the screen (off ebay - there is an ebay store that sells small quantities and will cut them to size for you). I also got a 'first hack' of the blob tracking software working. I will work on improving it incrementally. I also may end up writing my own blob detection code too since the CV blobs lib is returning some strange results at times and it shouldn't be a very hard algorithm at all given the data (just a connected components algorithm).

Touchscreen Progress 2

Yesterday I did my first trial wiring up of the LEDs - all went well. I'm doing groups of 7 IR LED's with a 15 ohm resistor in a series running off the 12v rail on an old PC power supply. This will no doubt be part of my tutorial when I write all this stuff up. Right now I am kind of stumped on how to enclose this thing. Ideally I'd like some kind of aluminum bracket style frame which goes around the edge of the acrylic and protects the LED's. Any ideas? Ideally I'd like to find some kind of pre-built picture frame to do the job, but so far I haven't found anything suitable.

Mindmap updated

I updated the mindmap for my FTIR touch screen project. It can be found here. I added a couple new links and some info. I started work on the software end. I was going to drop using OpenCV's Blob detection library since it didn't seem to be working correctly. However, I happened to glance over the docs for it and I realized what my problem was - the blob tracking lib expects greyscale input and I was feeding it color input. With that change in place it seems to be working well. This should save me some work but I still need to write some code for tracking blobs from frame to frame to ensure that they are recognized as being the same blob.

FTIR touchscreen progress

Well, I got the acrylite sheet cut and mostly polished today. My leds didn't arrive :(~~~ but they should be here monday. I also ordered some UV glue which should also be here this week. I can't wait until I have this thing assembled. While I'm waiting I guess I should start on the software end. I want to write my own blob detection and tracking code.

I also recieved another web cam I ordered, advertised as an 'infrared webcam'. Basically it just seems to have no IR filter and it comes with some IR LED's. This is fine since I jacked up the focus on my creative webcam when I did the IR hack.

BTW, I do plan on putting up some pics and instructions if I can get this thing working (prolly some software too). I plan on supporting OSC (open sound control) and I will use OpenCV for capturing the video.

Got my projector

Just got the projector I ordered off ebay. I still have to verify that it's working with my PC, but I was able to power it on and bring up the menu, which is a good sign. I'm still waiting for my LED's but they should be here tomorrow or saturday. I'm also still debating about the best way to polish the edges of my acrylic sheet (whether it's wise to use a flame torch or not) and trying to find some 400 and 600 grit sandpaper. Hopefully I will solve that problem today.

Multitouch screens!

I'm sure everyone has seen that awesome video on youtube of the multitouch screen being developed at NYU (just search for multitouch and I'm sure you'll find it). Multitouch is great because it promises to be a much more expressive and intuitive interface to the PC. Simple gestures could replace the need for icon-packed toolbars (which leads to faster editing). I assumed the technology would be way out of reach for the average person to implement but after reading a few pages on how the multi-touch is accomplished, I'm thinking it might be possible to do using off the shelf technology and some programming.

Here's the list of things you need:

Acrylic sheet for the screen
IR-Hacked web cam (a 10$ off the shelf webcam will work - just remove the IR filter)
Projector (potentially optional if you just want to make a touch pad with no screen)
some IR Led's
some kind of stand to set things up on (I may build a simple one out of PVC or something)
software

As you can see the most expensive thing is the projector

Basically how it works: IR light projected from the LED's in the sides of the screen is reflected off your finger tips wherever you touch the screen. This is picked up by the webcam and the software analyzes the webcam image and determines where your fingers are. The screen is rear-projected using the projector. And that's it! So far I have the IR webcam hack working and was able to get OpenCV working to capture images from the Webcam. Next step is picking up an acrylic sheet.

I set up a mindmap for the project here which includes some helpful links and other ramblings. I'll be adding to it as time goes on.

In other news - I also got a few things done on the game engine. I made a few more event-generation functions and also Niko gave me permission to use some of his collision code in OgreOpcode. Thanks Niko!

Collision detection finally working

I finally have collision detection working in my little game engine and it seems to be working great. I had to borrow some code from Irrlicht in order to get it working how I wanted - hopefully Niko will let me make it a part of OgreOpcode. Now that this big task is out of the way I have a bunch of smaller, easier tasks to finish (like having collisions cause scripting events).

Once I get a small working game framework together I may release it to spare others the pain I had to go through. Making games shouldn't be difficult!