Touchlib svn updates

I added a new Flash application to touchlib. It's a game called Pegs and it uses the great Box2d physics engine. There's really nothing about it that leverages the fact that it's multitouch, but it's fun. I think Box2d could be a nice platform for writing simple games. Check out Pegs for examples on how to do basic things like collision detection and using Sprites in place of the simple wireframes. The PhysicsCanvas class should be your basis for multitouchable physics apps. I will be adding additional features to it as needed. For now, you can override it's methods to get a working physics system with very little work.

I also added a basic demo of Box2d physics with touchlib support. You can grab and manipulate multiple shapes at once. Nui (chris moore) is reportedly working on an update to that with the rest of the Box2d demos. It's called Phys1 in the AS3\src directory. Code is in AS3\app\demos\Phys\, if my memory serves me.

Diffused Illumination setup pics

I uploaded a few pictures of my diffused illumination setup to flickr, just in case anyone wanted to see how it's done. It's dead simple really, especially if you buy a pre-made IR illuminator like I did. For a diffuser I used mylar architect paper which costs about $5 and can be found in larger art supply stores. The camera is the PtGrey Firefly MV with a camera IR filter taped on. The mirror is a first surface mirror - a bit larger than I ended up needing. I probably could have gone with something 1/2 the size.

Difused Illumination test

I tried out the diffused illumination method yesterday. Results were pretty encouraging - I got reliable touches without having to press hard like I did with the FTIR. Also you can use any part of your finger or hand to trigger it (or other objects such as a pointer or paintbrush..). I think this is the approach I will go with for the next table I build.

Touchlib in nuigroup wiki

The nuigroup.com wiki is embarrassingly sparse right now. I made an effort to get some basic documentation for touchlib in there. Two pages of note:

• http://nuigroup.com/wiki/Touchlib/
• http://nuigroup.com/wiki/Touchlib_Config_Information/

If you're a touchlib user, feel free to add your knowledge to the Wiki. There's lots of people who are interested in the technology who need your help getting up to speed.

Diffused Illumination vs FTIR

This post is just a quick recap for those that haven't been following all the discussions. What I am terming 'diffused illumination' is shining an infrared light source through the screen (either behind the screen with the camera and projector or behind the user as you might be able to do using daylight as your light source). The screen also acts as your diffuser which causes any objects behind it to appear blurry. Only objects very close to the screen can be resolved. Diffused illumination illuminates the whole hand and there are a couple different ways you can analyze the image - you can use some kind of shape based approach to pick out the fingers or try something like the approach illustrated in a past blog post. This is the technique employed by the CityWall and Microsoft's Surface. The advantage of the Diffused Illumination approach is that you can use things other than fingers such as pencils, brushes or pointers. The dryness of your fingers doesn't matter and dragging operations should work a lot better. Also it lends itself to using fiducial tags which are markers you can apply to objects that can be recognized by the system and tracked. This is how Microsoft's Surface can track cell phones, credit cards, juice cups, etc. Tags also lend themselves to board games with tangible peices. Diffused illumination may also be more tolerant to bright environments and is probably the only technique that can work in daylight. DI is also a lot easier to make - you can just buy an infrared light source. No need to solder LED's or calculate resistor values. I still need to do some more testing (I should be getting a new infrared light source later this week) so I will post how it works out.

Fiducial detection, first hack

I took my first stab at adding fiducial detection to touchlib today. I cam up with a simple algorithm that takes advantage of OpenCV's tree of contours (cvFindContours) which we are already using for blob detection right now. Basically it first detects whether a blob is square (which tells touchlib that this is a fiducial and not a finger press) then it adds up the detected shapes inside that square using a formula to get a unique identifier.

Diffused Illumination

I've started taking a look at what would be necessary to start implementing support in touchlib for what I am hereby naming the 'diffused illumination' approach. Someone else proposed Rear Illuminated but it can also be front illuminated so we need a name that works for both. The picture is a test run doing a 'front illuminated' setup. I just pointed my screen towards the window and recorded a test movie. I've developed a simple algorithm that I think will work well for front and rear illuminated setups (front illuminated setups just need an extra step to invert the colors). Basically the algorithm is to blur the image then subtract the blurred image from the original. I believe this is how Photoshop's high pass filter works. I then also blur the image again to get rid of noise. What you are left with is the parts of the image that are sharp and in focus and the blurry parts are removed, which is exactly what we want.

Touchlib flash apps now work with Flash CS3

There were a few errors compiling the flash apps under flash CS3. I fixed those now, so get the latest update from the SVN. This probably means that they will no longer open under the Flash 9 Alpha so if you are still using that it's time to upgrade.

Long time..

It's been a long time since I've been able to post anything, but I really haven't had too much to blog about. I'm working a bit on strategy game in Flash 9 - think of it as Master of Orion meets Weird Worlds meets Monopoly. I'll post more when I have something worth showing. My goal is to keep it pretty simple so that I actually finish it. I also have been thinking about getting back to working on my music app Genome since both Buzz and Buze leave something to be desired under Windows Vista (and that something is 'not crashing constantly').

I am still working on building a multitouch screen for my workplace - I should have the last bit of hardware this week (a first surface mirror for projection), so I am hoping I will be able to start putting it all together. I am interested in trying out some different things like seeing how well the CityWall approach works and maybe trying to elevate the LED's slightly like the Tangent table. I ordered a big piece of silicone rubber but it sticks to my acrylic too much to be useful. I wonder if a thinner piece attached to a more rigid piece of plastic would stick less. I'd also like to try some of the plasti-dip on a thicker piece of the mylar drafting paper. I think that might work well.

Touchlib

Just a note to anyone trying out the binaries: You'll also need to have the .NET framework installed apparently (I'm guessing visual studio 2005 added this dependency.. anyone know how to remove it?). You'll also need DirectX installed if you expect to get any video capturing using the DSVideoLib option (which is the default). I'll update the touchlib page with this info soon. I also have some slightly newer binaries to post with some fixes and additions.

As you know, I am working on a complete package for developing apps using Touchlib, FLOSC, and Flash 9. I'm currently working on the low level stuff like generating events for multitouch things. Debugging is proving to be a pain in the ass since I moved my camera and table to a seperate machine so I have to walk to another room when I want to test something. This is combined with the fact that Flash 9 doesn't have breakpoints and code stepping working (yet). So, I decided I've had enough and I'm gonna write some code to record the OSC packets to an XML file that I can replay again and again while I'm debugging. I'm also coding some built in stuff for visualizing finger presses and displaying debug info. This way I won't have to get up and I'll at least be able to use Flash's trace window.

Once I get this stuff working, my first app is gonna be the Jeff Han style photo app since it should be fairly easy to do in flash and everyone seems to want it. Plus it'll allow me to write code for for the 'scale, rotate, translate' gesture which will no doubt become a multitouch standard.

Preliminary OSC support added to touchlib

I checked in the first version of a small demo app built on touchlib that outputs presses using the TUIO and OSC protocols. Pictured on the left is one of the reference apps which responds to the TUIO protocol taking messages from my new OSC app. This allows you to distribute the processing of touch messages and applications across two machines (OSC operates over regular internet cables). It can also run on one machine. This will also let you write multitouch applications in things like Flash (using FLOSC or Flash 9) and Processing - anything that responds to OSC. This should lower the bar for those looking to write some apps. Note: these changes now require you to have the OSCPack files downloaded in order to compile the OSC example app. Also set the OSCPACK_HOME environment variable to the root dir where you have OSCPack installed.

You can download the pre-compiled executable files and associated DLL's here. If you have a working hardware setup (including projector), you'll need to run the configapp.exe to calibrate your stuff first. Those without a full FTIR hardware setup may not get much use out of this. Note: there may be some bugs with the OSC.exe app. I will take a look at fixing these soon.

Get the latest updates from Google Code.

Stuff, Zelda:TP Thoughts

Things have been crazy hectic at my new job. As a result, I've been too stressed out to work on Genome or Touchlib after work. Hopefully things will calm down soon and I'll be able get some stuff done again.

I have been playing a little Zelda: Twilight Princess to calm down a bit. So far it has been a really enjoyable game. I was a little hesitant to buy it after FFXII since I didn't want another game that forces you to play for hours or requires you to cover the same territory over and over again. The last Zelda game (Windwaker) suffered from this problem in that the sailing sections ended up being very time consuming and were pretty boring to me. I'm happy to report that Zelda: TP doesn't suffer from any of those flaws - the level design is very tight and there is little or no need to re-cover old terrain so far. Also, you can save your game wherever you want. The puzzles are clever but not so tough that you can't figure them out on the first or second try. It's a great example of how games should be designed. Your goal is always clear, you receive constant feedback when you do something right, actions that your character can perform are always highlighted on screen, dying is only a temporary set-back (you can restart at the beginning of that room), hints are available if you need them, and it's usually obvious when you need to use a particular tool in your arsenal. Also, the levels themselves act as tutorials for learning how to use your tools when you acquire them - they gradually show you all the different techniques you can execute. It's a far cry from the original Zelda when you had to bomb the fourth rock from the right on a specific map screen (a rock that looks exactly like all the other rocks...). Also, all the items provide different abilities rather than just being stronger versions of the same weapon as most FPS games are. This way the game is always throwing at you fresh variations on the gameplay. Overall, I highly recommend it.

Today's Work

I got a bunch of stuff done on Genome's tracker module. You now scroll patterns larger than the screen, and you can box select with the mouse. I'm planning on putting my own spin on the tracker in an attempt to bring it into the 21st century. For one, I'm making it horizontal instead of vertical (mirroring pretty much every other music program out there). Mouse interaction will be much better than in the past (read: Buzz). You will still be able to do everything with the keyboard, but there's no reason why you shouldn't be able to do everything with the mouse too. Gone is the fact that deleting a cell moves all the cells beneath it up. I always felt that this was annoying behavior. Sometimes it does come in handy for moving blocks of notes around, however a lot of the time you just want to delete something without messing up your timing for everything else. Instead, I will leverage box selection and allow users to move around blocks of notes with the keyboard or mouse, or I will add another keyboard short cut (like ctrl-delete) to do the old behavior.

On the touchlib front, on suggestion from some other #ftir'ers I checked out Oscpack and FLOSC. I was able to get both up and running fairly easily so I think I may take a shot at adding OSC support to Touchlib. I will use the TUIO protocol which is already used by the reactivision software. A handful of programs already support this protocol - Audiomulch for one - so it could expand the available applications that can work with touchlib. Also, it will alow for development in Flash (with FLOSC) which is exciting to me as a Flash Developer. This could be a good option for people to start making apps without having to mess with C++.