Archive for the ‘MultiTouch’ Category
I realize I haven’t posted any updates lately, but I am sort of waiting to have one *big* update. This won’t be that big update, but I’d like to at least give a quick rundown of what I have been up to since the last update.
There isn’t a whole lot to say here, but there is one big one:
Dynamic SubDevice Creation
An option that has been created is for dynamic subdevice creation. The idea is that because there are a limited number of TUIO subdevices to send events through, there is an inherent limit to the number of unique TUIO cursors or blobs that can be represented or pushed through the X input system at any given time. This means that if there are 5 subdevices that the TUIO driver is using to send events through, and 6 active blob or cursor ids, only 5 of those will have events being sent through a subdevice; the 6th has to wait for a new subdevice to be available before it can start sending X events. The dynamic subdevice creation option allows new subdevices to be created when this limit is reached.
In the last post I mentioned changing the input layer of certain multitouch applications so that they would be XI2 compatible; shortly thereafter I had success using the mpx patch posted here. The patch adds several new callbacks that can that can be registered with freeglut and used to receive multi pointer events. For the most part they just add an extra device id parameter to the usual mouse_entered, mouse_pressed, etc. events. Using this patch it was fairly simple to port the popular tuiosmoke application to be XI2 compatible. I haven’t actually posted the patch for this yet, but I will by the end of this project. I would also be willing to write a quick post on using the new freeglut callbacks if anyone is interested.
TMD (TUIO Monitor Daemon)
I have written a daemon that will connect to X and listen for any hierarchy events – namely, Master Add and Slave Add events. Upon receiving a slave add event, the daemon will see if it a TUIO subdevice, and if it is, create a new master pointer and reattach the new slave device to the new master pointer. This was written to relieve the pain of manually making hierarchy changes so that TUIO subdevices were under their own master pointer.
More recently I have been working on a GUI version of xinput that will have most of the features that the CLI app has. Past hierarchy changes, it will be very useful for changing the properties of devices, and I hope it will also make it easier to sift through them. I am developing this right now – I expect it will be available in beta form mid next week.
So there it is, a *very* quick rundown of what I have been up to. I haven’t gotten any feedback yet from anyone who has tried any of my work out, so if you have, PLEASE let me know. If anyone is interested in reading about one certain topic or another, let me know and I can make a post on it – I haven’t gotten any feedback, so it’s difficult to know if there is anything that anyone wants to read about (if anything).
If you aren’t too interested in reading all the development info and want to see results, keep an eye on the project news feed. Late-August I expect to have a video showcasing the driver and interaction with XI2 applications.
I hope you are all convinced that I have been up to something over here :).
This is part 2 of my previous post which discussed the current progress of my GSoC project and also included a video that showed the driver in action.
In part 1 I ended off by talking about MDs and cursors. To create new MDs or cursors you need to use the XI2 api, which can’t (or, atleast, shouldn’t) be done in the driver. From a users point of view, this makes sense. If I plug a mouse in, I would be pretty pissed if the driver decided that it wanted to make a new MD and attach itself to it without me telling it to do that. Granted, this driver is a special case (especially because we don’t have xblobevents yet), but I still think it is important to use everything as it was meant to be used. If we start making exceptions everywhere, we will end up with a bunch of special purpose applications, which is what we were trying to avoid in the first place.
To this end, I have written the driver such that it leaves MDs untouched. In the video, though, the second half shows multiple cursors moving around at once. To do this, I was required to manually move the subdevices that the driver creates to their own master devices. Let me give you quick example.
When starting X, it will parse my xorg.conf, and come across this section:
Section "InputDevice" Identifier "mt0" Driver "tuio" Option "SubDevices" "3" EndSection
When reaching this, it will create a new device that uses the tuio driver, and set the “SubDevices” option for the device. Once this device is started, it will create the number of devices specified by “SubDevices” to be used to route events through (it also routes events through itself).
Once the tuio driver is loaded and its subdevices are created, the output of `xinput list –short` will look something like this:
[ryan@namffuh ~]>> xinput list --short ⎡ Virtual core pointer id=2 [master pointer (3)] ⎜ ↳ Virtual core Xtst pointer id=4 [slave pointer (2)] ⎜ ↳ mt0 id=6 [slave pointer (2)] ⎜ ↳ Macintosh mouse button emulation id=8 [slave pointer (2)] ⎜ ↳ TPPS/2 IBM TrackPoint id=9 [slave pointer (2)] ⎜ ↳ mt0 subdev 0 id=12 [slave pointer (2)] ⎜ ↳ mt0 subdev 1 id=13 [slave pointer (2)] ⎜ ↳ mt0 subdev 2 id=14 [slave pointer (2)] ⎣ Virtual core keyboard id=3 [master keyboard (2)] ↳ Virtual core Xtst keyboard id=5 [slave keyboard (3)] ↳ ThinkPad Extra Buttons id=7 [slave keyboard (3)] ↳ AT Translated Set 2 keyboard id=10 [slave keyboard (3)] ↳ Video Bus id=11 [slave keyboard (3)]
“mt0” is the device as specified in the xorg.conf, and the “mt0 subdev #” devices are the devices create by the “mt0” device. At this point all input events will be routed through the “Virtual core pointer” MD and thus through the one pointer that is active. To create new cursors/MDs, xinput can be used:
[ryan@namffuh ~]>> xinput create-master "TUIO MD 0" [ryan@namffuh ~]>> xinput create-master "TUIO MD 1" [ryan@namffuh ~]>> xinput create-master "TUIO MD 2" [ryan@namffuh ~]>> xinput list --short ⎡ Virtual core pointer id=2 [master pointer (3)] ⎜ ↳ Virtual core Xtst pointer id=4 [slave pointer (2)] ⎜ ↳ mt0 id=6 [slave pointer (2)] ⎜ ↳ Macintosh mouse button emulation id=8 [slave pointer (2)] ⎜ ↳ TPPS/2 IBM TrackPoint id=9 [slave pointer (2)] ⎜ ↳ mt0 subdev 0 id=12 [slave pointer (2)] ⎜ ↳ mt0 subdev 1 id=13 [slave pointer (2)] ⎜ ↳ mt0 subdev 2 id=14 [slave pointer (2)] ⎣ Virtual core keyboard id=3 [master keyboard (2)] ↳ Virtual core Xtst keyboard id=5 [slave keyboard (3)] ↳ ThinkPad Extra Buttons id=7 [slave keyboard (3)] ↳ AT Translated Set 2 keyboard id=10 [slave keyboard (3)] ↳ Video Bus id=11 [slave keyboard (3)] ⎡ TUIO MD 0 pointer id=15 [master pointer (16)] ⎜ ↳ TUIO MD 0 Xtst pointer id=17 [slave pointer (15)] ⎣ TUIO MD 0 keyboard id=16 [master keyboard (15)] ↳ TUIO MD 0 Xtst keyboard id=18 [slave keyboard (16)] ⎡ TUIO MD 1 pointer id=19 [master pointer (20)] ⎜ ↳ TUIO MD 1 Xtst pointer id=21 [slave pointer (19)] ⎣ TUIO MD 1 keyboard id=20 [master keyboard (19)] ↳ TUIO MD 1 Xtst keyboard id=22 [slave keyboard (20)] ⎡ TUIO MD 2 pointer id=23 [master pointer (24)] ⎜ ↳ TUIO MD 2 Xtst pointer id=25 [slave pointer (23)] ⎣ TUIO MD 2 keyboard id=24 [master keyboard (23)] ↳ TUIO MD 2 Xtst keyboard id=26 [slave keyboard (24)]
Now 3 new MDs have been created, along with 3 new cursors on the screen. Ignore the Xtst devices, they are automatically created and are used for testing. Now we must reattach the subdevices to their new MDs:
[ryan@namffuh ~]>> xinput reattach 12 15 [ryan@namffuh ~]>> xinput reattach 13 19 [ryan@namffuh ~]>> xinput reattach 14 23 [ryan@namffuh ~]>> xinput list --short ⎡ Virtual core pointer id=2 [master pointer (3)] ⎜ ↳ Virtual core Xtst pointer id=4 [slave pointer (2)] ⎜ ↳ mt0 id=6 [slave pointer (2)] ⎜ ↳ Macintosh mouse button emulation id=8 [slave pointer (2)] ⎜ ↳ TPPS/2 IBM TrackPoint id=9 [slave pointer (2)] ⎣ Virtual core keyboard id=3 [master keyboard (2)] ↳ Virtual core Xtst keyboard id=5 [slave keyboard (3)] ↳ ThinkPad Extra Buttons id=7 [slave keyboard (3)] ↳ AT Translated Set 2 keyboard id=10 [slave keyboard (3)] ↳ Video Bus id=11 [slave keyboard (3)] ⎡ TUIO MD 0 pointer id=15 [master pointer (16)] ⎜ ↳ mt0 subdev 0 id=12 [slave pointer (15)] ⎜ ↳ TUIO MD 0 Xtst pointer id=17 [slave pointer (15)] ⎣ TUIO MD 0 keyboard id=16 [master keyboard (15)] ↳ TUIO MD 0 Xtst keyboard id=18 [slave keyboard (16)] ⎡ TUIO MD 1 pointer id=19 [master pointer (20)] ⎜ ↳ mt0 subdev 1 id=13 [slave pointer (19)] ⎜ ↳ TUIO MD 1 Xtst pointer id=21 [slave pointer (19)] ⎣ TUIO MD 1 keyboard id=20 [master keyboard (19)] ↳ TUIO MD 1 Xtst keyboard id=22 [slave keyboard (20)] ⎡ TUIO MD 2 pointer id=23 [master pointer (24)] ⎜ ↳ mt0 subdev 2 id=14 [slave pointer (23)] ⎜ ↳ TUIO MD 2 Xtst pointer id=25 [slave pointer (23)] ⎣ TUIO MD 2 keyboard id=24 [master keyboard (23)] ↳ TUIO MD 2 Xtst keyboard id=26 [slave keyboard (24)]
And now the subdevices will control individual cursors. If the goal is to control independent cursors, then it would be ideal to write a script/app to make these hierarchy changes automatically, rather than doing it manually every time. This is something I will probably put together in the next few days.
Currently this is where the driver is at. It will route events through the “core” device and the subdevices. If there are more active blobs or objects than there are devices to send events through, then they will have to wait until another blob/object is removed and there is a free device to send events through.
Hopefully this has been making sense – if it hasn’t, please let me know and I can help clear it up! Also, if anyone has any suggestions, I would really appreciate that as well.
You might be wondering what we can actually do at this point with the tuio driver. Without a large list of Xi2 applications to play with, there isn’t much we can do with it out of the box. This week I am going to try to modify several TUIO based applications to receive XI2 input by modifying their input layer – the layer which receives and parses tuio messages. I will make posts as progress is made, and maybe even have a video by the end of this or next week.
Comments are encouraged! Thanks!
Ahh, sorry for the lack of updates, I will *try* to be better in the future. I’m hopefully at a point where I can post updates more frequently – previously it was hard without much to show.
With regards to my last couple posts on here – my proposal was accepted and I’ve been working on the project the past 5-6 weeks, in case you aren’t coming from the nuicode site.
Here is a video to show the current state of the driver:
Ch ch ch changes
First, let me begin by talking a little about the original goals and what has changed, particularly with regards to the driver implementation itself. Here is a snippet from the origianl proposal:
What I am proposing is a TUIO xorg driver (xorg-input-tuio) written in C, that will receive TUIO events from a tracker and send them off as XBlobEvents. It would then only be a matter of writing an application that can receive these special events to enable MT functionality. One possible solution for receiving the TUIO events is to simply open a socket on port 3333 and listen for TUIO UDP packets, all within the driver.
Let’s pick this this apart a little bit.
and send them off as XBlobEvents
Shortly after submitting my proposal I received a reply from Peter Hutterer about several question I had, and was instantly notified that the XBlobEvent branch was orphaned and would not be usable/compatible with the current upstream X. This came as a surprise to me, so I updated my GSoC proposal and reported that I would instead route unique blobs through unique Master Devices (MDs) (you can consider an MD to be an individual cursor). Some things have changed there a bit as well, but I will get to that a little further down.
It would then only be a matter of writing an application that can receive these special events to enable MT functionality.
So, of course, this has changed because these so called “special events” aren’t being used and don’t exist yet. Instead, events are routed through individual Slave Devices (SDs) (which I also talk more about) which should be attached to separate MDs. In terms of writing applications, an application needs to be XI2 aware – Peter Hutterer has been writing on this recently (Peter created MPX and works on XI2).
One possible solution for receiving the TUIO events is to simply open a socket on port 3333 and listen for TUIO UDP packets, all within the driver.
This is about the only thing that hasn’t changed. This is what the driver does now, and fairly soon will support arbitrary devices as well (e.g. /dev/tuio).
Let’s discuss devices a little, and how they come into play in this project.
Without XBlobEvents in play right now we need a way to communicate separate events in X – this has to be done through separate input devices (if your running X, type `xinput list` into a terminal – those are your input devices). To do this we need to get a hold of “dummy” devices that we can use solely to send events through – there are probably several ways to do this, but I know of two, courtesy of Peter Hutterer. The first way is to use libhal to tell HAL to create a new input device. X, so intently listening, will receive the message about a new device and create it (example). The second option is to tell X directly using `NewInputDeviceRequest`, which, is a little more difficult to use because it doesn’t take care of some of the cleanup (example). So, I went the HAL route.
Using the example above, I put together code to create new devices fairly quickly. What it boils down to is creating a new device through HAL each time a new object is found and destroying it when the object is no longer around (I am going to refer to blobs as object from here on out. At the moment I’m using the /tuio/2dcur OSC address space, but later on this may /tuio/blb, so I’ll just say “object”). When a new TUIO device is created, my driver will pick it up and start routing the events for the associated object through that device – or, at least, this is how it should work. I worked with this approach for awhile and it was working when the waters were calm, but as soon as I would start creating/destroying multiple objects many times a second X would freeze. Why? Hmm… attach gdb… Okay… `bt`… dbus??? Communication with HAL is done through DBUS, so that was fine, but it was just sitting there in some function call. Looking at the trace I also saw the function call `dbus_connection_send_with_reply_and_block()`… GREAT. I scoured the internet looking for a way to make asynchronous calls, but with little luck. I believe you can set a callback to make it non-blocking, but I couldn’t find a whole lot out on this, and I wasn’t even sure if it would work – after having it basically crash on me, I was already feeling edgy about using HAL. (If anyone can offer help with that, let me know 🙂 ).
To give some context, the previous paragraph took place only around 1.5-2 weeks ago. Previous to that I had run into various issues getting everything up and running, finally having single device/cursor working after those first weeks.
It was around last Thursday that I thought I had things working, and that I was going to start working on multiple cursors (no, multiple devices doesn’t necessarily mean multiple cursors). Then, Sunday night, I find out that I’m having all these problems and I have no solution. Quickly, I try the second method that Peter referred me too and… seg fault. I quickly diffused that bug (I’m lying, that one actually stole a good chunk of time 🙂 ) and moved on. Okay, things were checking out and everything seemed to be working as before. Add some code to remove the devices as well and… BAM, random errors that don’t seem like they have anything to do with my code. So here I am, it’s Monday some time, and I want to make a post Tuesday/Wednesday that actually shows progress (I did promise it, after all. And, *cough*). So what do I do? I try to fix it, but without a whole lot of an idea on what to do. At that point I took a break, dropped that approach, and tried to fix the HAL-based code. I still think the second option may be better, I just need to figure out what is expected when using those commands. Oh, and, by the way, it’s soon Tuesday and my internet was out from 10am – 5pm. Talk about feeling helpless – kill Google, kill knowledge.
Wednesday (yesterday) rolls around and I’m getting a little down. Everything I’ve tried seems promising, then responds with errors that I just don’t know how to approach. Then it occurred to me, “HAL is my friend until I’m chatting too much with it… Hmm… Why don’t I just have a quick conversation when I first start start up, then ignore it?” So that’s what I did. When the driver first starts it creates some number of devices through HAL, and just reuses them – this is in contrast to constantly adding/removing devices. No more worrying about HAL blocking. Everything seems to work this way, and I haven’t experienced any crashes/freezes yet. Obviously it could block when first initializing, so I will need to fix that soon.
MDs and Cursors
Something I haven’t addressed yet is MDs and having separate cursors. I’m not necessarily going to give a crash course on how MPX/XI2 works, but essentially there are two different types of devices – Masters and Slaves. You would associate the mouse you’re using right now to a Slave device, along with any other physical input device you’re using (mouse, keyboard, tablet, etc.). Each one of these is generally attached to a Master device. Each Master device takes the form of a cursor on your screen. In the general case you have one Master device, with all of your Slave devices attached to it, and all their events are routed through it. So, in order to have multiple cursors you need to create multiple cursors, and to create multiple cursors you need to use the XI2 api to create MDs and attach SDs to them – in particular, create a new MD when a new tuio “dummy” SD is created, and attach the “dummy” SD to it…
More to say on that, but I will have to continue this tomorrow in a “part 2” because I need to get some sleep :).
I will be using this blog to track my experiences with new and innovative ways to interact with computers; in particular, I will be exploring the use of multi-touch (mt) screens to create a more immersive computer experience. I’ve always had an interest in this area of computer interfacing, but it wasn’t until recently that I found a cost-effective way to create and tinker with any of this equipment myself. The Natural User Interface Group is an organization of open source developers creating multi-touch software for cost-effective mt hardware. I urge you to check out their site and read more!
Between work and a hefty load of school work, I have managed to put together an MTmini – basically, a cheap way to create an mt surface using a picture frame, a cardboard box, printer paper, and a webcam (in my case, a Playstation Eye). Using software such as TouchLib, the webcam can pick up shadows on the paper+glass surface, and distinguish them as “blobs.” At this point a software program can do endless things with this information – create cool particle effects at the point of touch, track drags of the finger and translate these to the movement of on-screen objects, etc. – it really is limitless!
The MTmini that I mocked up needs to be refitted into a larger box, but until I get a larger piece of plexiglass this will have to suffice. Here is a picture:
Okay, quit laughing. And yes, that is a Tootsie Pop box. Haha :). I should probably tape the entire box to avoid light leaking in, but I will most likely just replace it with a larger box and a piece of plexiglass pretty soon here anyway. If I can, I would like to put together an FTIR setup (without a projector or lcd) just to get a little experience with it and to have a potentially larger surface to work with without the need for it to be enclosed (although I believe I would need an IR bandpass filter to exclude the need for an enclosure).
So there’s my introduction. Sort of. I guess I didn’t talk about myself, but you can always go to the About Me section to find out about that. Thanks for reading!