MT, Linux, and other stuff

Ryan Huffman

TUIO X Input Driver – Current Progress (Part 1)

with 3 comments

Ahh, sorry for the lack of updates, I will *try* to be better in the future.  I’m hopefully at a point where I can post updates more frequently – previously it was hard without much to show.

With regards to my last couple posts on here – my proposal was accepted and I’ve been working on the project the past 5-6 weeks, in case you aren’t coming from the nuicode site.

Here is a video to show the current state of the driver:

Ch ch ch changes

First, let me begin by talking a little about the original goals and what has changed, particularly with regards to the driver implementation itself.  Here is a snippet from the origianl proposal:

What I am proposing is a TUIO xorg driver (xorg-input-tuio) written in C, that will receive TUIO events from a tracker and send them off as XBlobEvents. It would then only be a matter of writing an application that can receive these special events to enable MT functionality. One possible solution for receiving the TUIO events is to simply open a socket on port 3333 and listen for TUIO UDP packets, all within the driver.

Let’s pick this this apart a little bit.

and send them off as XBlobEvents

Shortly after submitting my proposal I received a reply from Peter Hutterer about several question I had, and was instantly notified that the XBlobEvent branch was orphaned and would not be usable/compatible with the current upstream X.  This came as a surprise to me, so I updated my GSoC proposal and reported that I would instead route unique blobs through unique Master Devices (MDs) (you can consider an MD to be an individual cursor).  Some things have changed there a bit as well, but I will get to that a little further down.

It would then only be a matter of writing an application that can receive these special events to enable MT functionality.

So, of course, this has changed because these so called “special events” aren’t being used and don’t exist yet.  Instead, events are routed through individual Slave Devices (SDs) (which I also talk more about) which should be attached to separate MDs.  In terms of writing applications, an application needs to be XI2 aware – Peter Hutterer has been writing on this recently (Peter created MPX and works on XI2).

One possible solution for receiving the TUIO events is to simply open a socket on port 3333 and listen for TUIO UDP packets, all within the driver.

This is about the only thing that hasn’t changed.  This is what the driver does now, and fairly soon will support arbitrary devices as well (e.g. /dev/tuio).

Devices (SDs)

Let’s discuss devices a little, and how they come into play in this project.

Without XBlobEvents in play right now we need a way to communicate separate events in X – this has to be done through separate input devices (if your running X, type `xinput list` into a terminal – those are your input devices).  To do this we need to get a hold of “dummy” devices that we can use solely to send events through – there are probably several ways to do this, but I know of two, courtesy of Peter Hutterer.  The first way is to use libhal to tell HAL to create a new input device.  X, so intently listening, will receive the message about a new device and create it (example).  The second option is to tell X directly using `NewInputDeviceRequest`, which, is a little more difficult to use because it doesn’t take care of some of the cleanup (example).  So, I went the HAL route.

Blocking?  Eek…

Using the example above, I put together code to create new devices fairly quickly.  What it boils down to is creating a new device through HAL each time a new object is found and destroying it when the object is no longer around (I am going to refer to blobs as object from here on out.  At the moment I’m using the /tuio/2dcur OSC address space, but later on this may /tuio/blb, so I’ll just say “object”).   When a new TUIO device is created, my driver will pick it up and start routing the events for the associated object through that device – or, at least, this is how it should work.  I worked with this approach for awhile and it was working when the waters were calm, but as soon as I would start creating/destroying multiple objects many times a second X would freeze.  Why?  Hmm… attach gdb… Okay… `bt`… dbus???  Communication with HAL is done through DBUS, so that was fine, but it was just sitting there in some function call.  Looking at the trace I also saw the function call `dbus_connection_send_with_reply_and_block()`… GREAT.  I scoured the internet looking for a way to make asynchronous calls, but with little luck.  I believe you can set a callback to make it non-blocking, but I couldn’t find a whole lot out on this, and I wasn’t even sure if it would work – after having it basically crash on me, I was already feeling edgy about using HAL. (If anyone can offer help with that, let me know 🙂 ).

Option #2

To give some context, the previous paragraph took place only around 1.5-2 weeks ago.  Previous to that I had run into various issues getting everything up and running, finally having  single device/cursor working after those first weeks.

It was around last Thursday that I thought I had things working, and that I was going to start working on multiple cursors (no, multiple devices doesn’t necessarily mean multiple cursors).  Then, Sunday night, I find out that I’m having all these problems and I have no solution.  Quickly, I try the second method that Peter referred me too and… seg fault.  I quickly diffused that bug (I’m lying, that one actually stole a good chunk of time 🙂 ) and moved on.  Okay, things were checking out and everything seemed to be working as before.  Add some code to remove the devices as well and… BAM, random errors that don’t seem like they have anything to do with my code.  So here I am, it’s Monday some time, and I want to make a post Tuesday/Wednesday that actually shows progress (I did promise it, after all.  And, *cough*).  So what do I do?  I try to fix it, but without a whole lot of an idea on what to do.  At that point I took a break, dropped that approach, and tried to fix the HAL-based code.  I still think the second option may be better, I just need to figure out what is expected when using those commands.  Oh, and, by the way, it’s soon Tuesday and my internet was out from 10am – 5pm.  Talk about feeling helpless – kill Google, kill knowledge.


Wednesday (yesterday) rolls around and I’m getting a little down.  Everything I’ve tried seems promising, then responds with errors that I just don’t know how to approach.  Then it occurred to me, “HAL is my friend until I’m chatting too much with it… Hmm… Why don’t I just have a quick conversation when I first start start up, then ignore it?”  So that’s what I did.  When the driver first starts it creates some number of devices through HAL, and just reuses them – this is in contrast to constantly adding/removing devices.  No more worrying about HAL blocking.  Everything seems to work this way, and I haven’t experienced any crashes/freezes yet.  Obviously it could block when first initializing, so I will need to fix that soon.

MDs and Cursors

Something I haven’t addressed yet is MDs and having separate cursors.  I’m not necessarily going to give a crash course on how MPX/XI2 works, but essentially there are two different types of devices – Masters and Slaves.  You would associate the mouse you’re using right now to a Slave device, along with any other physical input device you’re using (mouse, keyboard, tablet, etc.).  Each one of these is generally attached to a Master device.  Each Master device takes the form of a cursor on your screen.  In the general case you have one Master device, with all of your Slave devices attached to it, and all their events are routed through it.  So, in order to have multiple cursors you need to create multiple cursors, and to create multiple cursors you need to use the XI2 api to create MDs and attach SDs to them – in particular, create a new MD when a new tuio “dummy” SD is created, and attach the “dummy” SD to it…

More to say on that, but I will have to continue this tomorrow in a “part 2” because I need to get some sleep :).

Written by rhuffman

July 3, 2009 at 9:15 am

Posted in GSoC, MultiTouch

3 Responses

Subscribe to comments with RSS.

  1. Very Nice Ryan!

    Sharath Patali

    July 3, 2009 at 11:34 am

  2. Great post!


    July 6, 2009 at 5:04 pm

  3. […] Progress (Part 2) Posted in Uncategorized by rhuffman on July 6, 2009 This is part 2 of my previous post which discussed my current progress with my GSoC project and also included a […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: