64-bit fixed

So, apparently the xf86-input-tuio driver hasn’t been working with 64-bit versions of X, causing this:

(II) mt0: TUIO UDP Port set to 3333
(II) mt0: FseqThreshold set to 100
(**) mt0: always reports core events

Backtrace:
0: /usr/bin/Xorg (xorg_backtrace+0x28) [0x45a898]
1: /usr/bin/Xorg (0x400000+0x619c9) [0x4619c9]
2: /lib/libpthread.so.0 (0x7fc3819b5000+0xee80) [0x7fc3819c3e80]
3: /lib/libc.so.6 (strlen+0x12) [0x7fc380cbe272]
4: /usr/bin/Xorg (xf86ActivateDevice+0x6b) [0x46d57b]
5: /usr/bin/Xorg (0x400000+0x6d6d7) [0x46d6d7]
6: /usr/bin/Xorg (InitInput+0x58) [0x468ec8]
7: /usr/bin/Xorg (0x400000+0x219af) [0x4219af]
8: /lib/libc.so.6 (__libc_start_main+0xfd) [0x7fc380c61b6d]
9: /usr/bin/Xorg (0x400000+0x21599) [0x421599]
Segmentation fault at address (nil)

Fatal server error:
Caught signal 11 (Segmentation fault). Server aborting

The problem is that there is a disparity in the sizes and offsets of member variables of _LocalDeviceRec – this is what’s passed to xf86ActivateDevice, and what eventually causes the segfault in strlen. The solution was to make sure that config.h was included before anything else – thanks to Peter Hutterer making me aware of this obscure bug. The fix.

X11R7.5 Released and tuio/tmd Packages

As some of you are probably aware, X11R7.5 was released several weeks ago, which means that multi-pointer x will now be much more accessible in the near future depending on which distro you are using.  I use Arch Linux on my laptop and all of the new updates are available on the pacman repositories.  I have put together two PKGBUILD’s for both xf86-input-tuio and the tmd daemon and posted on the AUR, which you can download directly and do a makepkg, or you can simply use yaourt:

yaourt -S tmd-git
yaourt -S xf86-input-tuio-git

In preparation for the release of Fedora 12, which is coming out on Nov 17th, 2009, I will be getting packages together for that as well.

I’m still planning a video, but it’s sort of on hold until I can finish a multitouch panel I’ve been putting together.  I started it before GSoC ended, but I just haven’t had the time to finish it with school keeping me busy and without the necessary tools at my home.  I’m expecting to finish that sometime later this month.

I’ll keep everyone updated on any future advancements.

GSOC: The End

Yesterday (technically 2 days ago, given that it is past midnight) was the official “pencils down” date for GSoC; for me, this means releasing my source code and summing it all up.  This also means that I need to hurry up and release another video :).

You can always grab the latest source at http://repo.or.cz/w/xf86-input-tuio.git, so if you want the latest you should look there.  The code being presented here is more of a snapshot than anything else.

Software

I have tagged version “0.1.0” on the git repo, and is what I will give an update on here.  Mostly a lot of cleanup has been done recently.  I would recommmend checking out the man page (`man 4 tuio`) to read a bit about the driver and the available options.  Honestly, it’s all there, and it’s probably pointless for me to talk about it here (especially because what I write here may soon be outdated).

If you want to grab the latest source, just

git clone git://repo.or.cz/xf86-input-tuio.git
git checkout xf86-input-tuio-0.1.0

if you want version 0.1.0 (there may be a better way to do that).

More exciting is the TUIO Monitor Daemon, though.  As mentioned in a previous post, tmd monitors X events and attempts to create unique master devices for each new TUIO subdevice.  If this is not done, all device events will be tunneled through one master device, and will only control one cursor.  You can always make these device hierarchy changes manually, but it is a pain.

To grab the newest tmd source:

git clone git://repo.or.cz/tmd.git

Disclaimer: The code has not been pushed just quite yet, I’ve made a configuration mistake and need to fix it.  Stay tuned. Fixed

Video

Pending.

Seriously, this will be done soon, I just have one thing pending before this will be completed.  The video will contain several clips of what is currently possible, and I think it will give a good idea of what can be done.

Future

I’m very optimistic about the future.  I’m not sure when the next release of xserver will be, but it should be within the next 2 months, at which point I think a lot of exciting things will happen.  First, I expect we will see a lot of support for XI2/MPX being implemented (it already has been on certain projects, but has quickly become outdated because of the ever changing api/abi), particularly in window managers and later toolkits.  Once there is a stable base for creating multitouch applications, I think everything will really be ready for primetime.  In regards to NUI Group, I think we will start seeing a lot of linux based MT devices being built, and developers developing XI2 applications.  Can’t wait, should be a lot of fun :).

What will I be doing?  Well, I’ll continue to maintain this project, and hopefully make several posts on creating XI2 based applications.

“I have a TUIO based MT setup, should I use this?”

Or, “is this driver of any use to me?”  Without a lot of native multitouch applications or support overall for XI2, it is probably worth waiting until everything is available in a distro such as Ubuntu.  On the other hand, if you are brave and don’t mind getting dirty with the command line (and have extra time), go for it!  If you do, you’d be helping me out quite a bit :).  It isn’t necessarily a trivial task to get your first dev version of X running, but once you do it is easy to maintain.  If you do decide to go down this path, send me an email and let me know about it; I’d really like to know if people are using the driver (r y a n h u f f m a n (at) g m a i l . c o m).

More to come…

Quick Update

I realize I haven’t posted any updates lately, but I am sort of waiting to have one *big* update.  This won’t be that big update, but I’d like to at least give a quick rundown of what I have been up to since the last update.

Driver Update

There isn’t a whole lot to say here, but there is one big one:

Dynamic SubDevice Creation

An option that has been created is for dynamic subdevice creation.  The idea is that because there are a limited number of TUIO subdevices to send events through, there is an inherent limit to the number of unique TUIO cursors or blobs that can be represented or pushed through the X input system at any given time.  This means that if there are 5 subdevices that the TUIO driver is using to send events through, and 6 active blob or cursor ids, only 5 of those will have events being sent through a subdevice; the 6th has to wait for a new subdevice to be available before it can start sending X events.  The dynamic subdevice creation option allows new subdevices to be created when this limit is reached.

Freeglut

In the last post I mentioned changing the input layer of certain multitouch applications so that they would be XI2 compatible; shortly thereafter I had success using the mpx patch posted here.  The patch adds several new callbacks that can that can be registered with freeglut and used to receive multi pointer events.  For the most part they just add an extra device id parameter to the usual mouse_entered, mouse_pressed, etc. events.  Using this patch it was fairly simple to port the popular tuiosmoke application to be XI2 compatible.  I haven’t actually posted the patch for this yet, but I will by the end of this project.  I would also be willing to write a quick post on using the new freeglut callbacks if anyone is interested.

TMD (TUIO Monitor Daemon)

I have written a daemon that will connect to X and listen for any hierarchy events – namely, Master Add and Slave Add events.  Upon receiving a slave add event, the daemon will see if it a TUIO subdevice, and if it is, create a new master pointer and reattach the new slave device to the new master pointer.  This was written to relieve the pain of manually making hierarchy changes so that TUIO subdevices were under their own master pointer.

XInput Manager

More recently I have been working on a GUI version of xinput that will have most of the features that the CLI app has.  Past hierarchy changes, it will be very useful for changing the properties of devices, and I hope it will also make it easier to sift through them.  I am developing this right now – I expect it will be available in beta form mid next week.

Conclusion

So there it is, a *very* quick rundown of what I have been up to.  I haven’t gotten any feedback yet from anyone who has tried any of my work out, so if you have, PLEASE let me know.  If anyone is interested in reading about one certain topic or another, let me know and I can make a post on it – I haven’t gotten any feedback, so it’s difficult to know if there is anything that anyone wants to read about (if anything).

If you aren’t too interested in reading all the development info and want to see results, keep an eye on the project news feed.  Late-August I expect to have a video showcasing the driver and interaction with XI2 applications.

I hope you are all convinced that I have been up to something over here :).

Quick Disclaimer + Update with fseq

One minor disclaimer I’d like to add is that the driver isn’t necessarily stable yet, so don’t do anything important with it loaded. If it seg faults or something similar it will crash X and anything you have open will be lost. I have it running on my system right now while I work so that I can stress test it a bit, but I’m careful about saving frequently.

Because I use MSA Remote as the primary tracker for my testing, I’ve run into TUIO fseq issues. The issue occurs when my tracker has been running, and I later restart it, thus resetting the fseq count back to 0. The driver will throw away any packets containing fseq numbers lower/older than the last accepted packet, causing any packets after an MSA Remote restart to be ignored – that is, until the fseq count catches back up.

For when I’m testing, I use the “CheckFseq” option that I recently added to disable fseq checking.

To use it, add this to the InputDevice section in your xorg.conf

Option "CheckFseq" "false"

The default for this option is “true” and is not necessary to set unless you want to turn fseq checking off.  Once I start adding device properties, I will probably add an option to reset the fseq count at runtime, which would be a more useful alternative.

UPDATE: Instead of ignoring the fseq number, the better solution to this problem is to accept a message if its fseq number is larger than the last received fseq number, OR if the difference between its fseq number and the last received fseq number is larger than a certain threshold value.  This is a much better approach because it takes care of the previously described issue without having to ignore anything, and on the off chance that the fseq ever wraps around (i.e. gets above 2,147,483,648 🙂 ) the fseq number will automatically be fixed without causing any issues.

Thanks goes to Martin Kaltenbrunner for that solution :).

On an unrelated note, I hate how this font displays 0’s exactly the same as o’s (the former is a zero, and the latter is the letter O).

Get/build/install xf86-input-tuio

For those interested in testing the xf86-input-tuio driver I have been working on, this post will provide download/compile/install instructions.

Getting the Source Code

All of my source code is available at http://repo.or.cz/w/xf86-input-tuio.git, which I push my updates to a few times a week, sometimes more depending on what I am pushing.

To grab the source code you can either use git to clone the repository or download a .tar.gz or .zip from the link above.  Using git is probably the easiest method if you plan to pull any updates in the future.

To download/clone the repo, use:

git clone git://repo.or.cz/xf86-input-tuio.git

To update the repo in the future, just do a

git pull

in the directory you downloaded it to.

Requirements

There are a few things you will need before you can build/use this driver, which partly depends on the environment you want to build it on.  If you want to take advantage of multiple cursors (which you have to at the moment – the driver is not backwards compatible as of the time of this writing) then you need to download/compile/install the newest X source from the freedesktop.org git repos.  If you are new to this, I would suggest taking a look at the Modular Developers Guide on the x.org wiki and the git_xorg.sh script.

UPDATE: Alternatively, check out Peter Hutterer’s “Building X.Org from git in 5 steps” in this post on his blog.

Once you have that installed, you can install the driver in a couple of ways.  If you used the method I just mentioned, then it would be easiest to put the driver in the driver/ directory, and add “xf86-input-tiuo” to the build script (util/modular/build.sh).  Then, to install the driver alone, you can just run

./util/modular/build.sh -o driver/xf86-input-tuio -g -s sudo /opt/xorg_debug

to compile/install the driver to the /opt/xorg_debug root directory (this will depend on where you installed the rest of your build).  The -g switch is optional.

If you aren’t using the build script, then you will need to run the autoconfig.sh/make/make install yourself, and set the environment variables up yourself as well.  I won’t get into this, but if you want help with that method then send me an email or talk to me in #nuigroup on Freenode (I’m rarely there, though).

UPDATE: The driver depends on liblo, which it uses for OSC parsing.  You will need to install that before compiling the driver.

Using the Driver

Xorg.conf

Because nothing is being hotplugged at the moment, the tuio device will need to be defined in your xorg.conf.  If you don’t want to use /etc/X11/xorg.conf, you can define your own in startx by using the “-config” flag (more on that down below).

Section "InputDevice"
    Identifier      "mt0"
    Driver          "tuio"
    Option          "SubDevices"      "4
EndSection

This is pretty straightforward.  There are two options that are currently availabe: SubDevices and Port.  Subdevices determines the number of subdevices that will be created to send object events through, and because the “core” device itself will be used for events, this gives a maximum of SubDevices + 1 objects at one time that can send events.  The Port options just determines the UDP port that is used to listen on for TUIO data.  The defaults are 5 SubDevices and 3333 for the Port.

UPDATE:

Martin Kaltenbrunner brought it to my attention that I was missing information on adding the device to the “ScreenLayout” section of xorg.conf, which is required.  You will need to add the input device to your ServerLayout section like this:

Section "ServerLayout"
    ...
    InputDevice "mt0"   "CorePointer"
    ...
EndSection

Starting X

At this point I will assume that you’ve installed everything to /opt/xorg_debug/ (substitute your directory where necessary).  To run everything, I use a small script that sets my environment vars up for me and calls startx:

export PKG_CONFIG_PATH=/opt/xorg_debug/lib/pkgconfig:${PKG_CONFIG_PATH}
export LD_LIBRARY_PATH=/opt/xorg_debug/lib:${LD_LIBRARY_PATH}
export LD_RUN_PATH=/opt/xorg_debug/lib:${LD_RUN_PATH}
export PATH=/opt/xorg_debug/bin:${PATH}
startx -- /opt/xorg_debug/bin/Xorg -verbose -config xorgdev.conf

If everything loads correctly (which, hopefully it will), then you be able to send TUIO data to your computer and it should respond accordingly.  If you want to use multiple cursors then you will need to use xinput to add new master devices and reattach the subdevices, which was described in my last post.

Bugs

If you run into any problems or have any feedback, create a new task at http://nuicode.com/projects/gsoc-mpx/issues/new.

TUIO X Input Driver – Current Progress (Part 2)

This is part 2 of my previous post which discussed the current progress of my GSoC project and also included a video that showed the driver in action.

MDs, continued

In part 1 I ended off by talking about MDs and cursors.  To create new MDs or cursors you need to use the XI2 api, which can’t (or, atleast, shouldn’t) be done in the driver.  From a users point of view, this makes sense.  If I plug a mouse in, I would be pretty pissed if the driver decided that it wanted to make a new MD and attach itself to it without me telling it to do that.  Granted, this driver is a special case (especially because we don’t have xblobevents yet), but I still think it is important to use everything as it was meant to be used.  If we start making exceptions everywhere, we will end up with a bunch of special purpose applications, which is what we were trying to avoid in the first place.

To this end, I have written the driver such that it leaves MDs untouched.  In the video, though, the second half shows multiple cursors moving around at once.  To do this, I was required to manually move the subdevices that the driver creates to their own master devices.  Let me give you quick example.

When starting X, it will parse my xorg.conf, and come across this section:

Section "InputDevice"
    Identifier      "mt0"
    Driver          "tuio"
    Option          "SubDevices"      "3"
EndSection

When reaching this, it will create a new device that uses the tuio driver, and set the “SubDevices” option for the device.  Once this device is started, it will create the number of devices specified by “SubDevices” to be used to route events through (it also routes events through itself).

Once the tuio driver is loaded and its subdevices are created, the output of `xinput list –short` will look something like this:

[ryan@namffuh ~]>> xinput list --short
⎡ Virtual core pointer                    	id=2	[master pointer  (3)]
⎜   ↳ Virtual core Xtst pointer               	id=4	[slave  pointer  (2)]
⎜   ↳ mt0                                     	id=6	[slave  pointer  (2)]
⎜   ↳ Macintosh mouse button emulation        	id=8	[slave  pointer  (2)]
⎜   ↳ TPPS/2 IBM TrackPoint                   	id=9	[slave  pointer  (2)]
⎜   ↳ mt0 subdev 0                            	id=12	[slave  pointer  (2)]
⎜   ↳ mt0 subdev 1                            	id=13	[slave  pointer  (2)]
⎜   ↳ mt0 subdev 2                            	id=14	[slave  pointer  (2)]
⎣ Virtual core keyboard                   	id=3	[master keyboard (2)]
    ↳ Virtual core Xtst keyboard              	id=5	[slave  keyboard (3)]
    ↳ ThinkPad Extra Buttons                  	id=7	[slave  keyboard (3)]
    ↳ AT Translated Set 2 keyboard            	id=10	[slave  keyboard (3)]
    ↳ Video Bus                               	id=11	[slave  keyboard (3)]

“mt0” is the device as specified in the xorg.conf, and the “mt0 subdev #” devices are the devices create by the “mt0” device.  At this point all input events will be routed through the “Virtual core pointer” MD and thus through the one pointer that is active.  To create new cursors/MDs, xinput can be used:

[ryan@namffuh ~]>> xinput create-master "TUIO MD 0"
[ryan@namffuh ~]>> xinput create-master "TUIO MD 1"
[ryan@namffuh ~]>> xinput create-master "TUIO MD 2"
[ryan@namffuh ~]>> xinput list --short             
⎡ Virtual core pointer                    	id=2	[master pointer  (3)]
⎜   ↳ Virtual core Xtst pointer               	id=4	[slave  pointer  (2)]
⎜   ↳ mt0                                     	id=6	[slave  pointer  (2)]
⎜   ↳ Macintosh mouse button emulation        	id=8	[slave  pointer  (2)]
⎜   ↳ TPPS/2 IBM TrackPoint                   	id=9	[slave  pointer  (2)]
⎜   ↳ mt0 subdev 0                            	id=12	[slave  pointer  (2)]
⎜   ↳ mt0 subdev 1                            	id=13	[slave  pointer  (2)]
⎜   ↳ mt0 subdev 2                            	id=14	[slave  pointer  (2)]
⎣ Virtual core keyboard                   	id=3	[master keyboard (2)]
    ↳ Virtual core Xtst keyboard              	id=5	[slave  keyboard (3)]
    ↳ ThinkPad Extra Buttons                  	id=7	[slave  keyboard (3)]
    ↳ AT Translated Set 2 keyboard            	id=10	[slave  keyboard (3)]
    ↳ Video Bus                               	id=11	[slave  keyboard (3)]
⎡ TUIO MD 0 pointer                       	id=15	[master pointer  (16)]
⎜   ↳ TUIO MD 0 Xtst pointer                  	id=17	[slave  pointer  (15)]
⎣ TUIO MD 0 keyboard                      	id=16	[master keyboard (15)]
    ↳ TUIO MD 0 Xtst keyboard                 	id=18	[slave  keyboard (16)]
⎡ TUIO MD 1 pointer                       	id=19	[master pointer  (20)]
⎜   ↳ TUIO MD 1 Xtst pointer                  	id=21	[slave  pointer  (19)]
⎣ TUIO MD 1 keyboard                      	id=20	[master keyboard (19)]
    ↳ TUIO MD 1 Xtst keyboard                 	id=22	[slave  keyboard (20)]
⎡ TUIO MD 2 pointer                       	id=23	[master pointer  (24)]
⎜   ↳ TUIO MD 2 Xtst pointer                  	id=25	[slave  pointer  (23)]
⎣ TUIO MD 2 keyboard                      	id=24	[master keyboard (23)]
    ↳ TUIO MD 2 Xtst keyboard                 	id=26	[slave  keyboard (24)]

Now 3 new MDs have been created, along with 3 new cursors on the screen.  Ignore the Xtst devices, they are automatically created and are used for testing.  Now we must reattach the subdevices to their new MDs:

[ryan@namffuh ~]>> xinput reattach 12 15
[ryan@namffuh ~]>> xinput reattach 13 19
[ryan@namffuh ~]>> xinput reattach 14 23
[ryan@namffuh ~]>> xinput list --short
⎡ Virtual core pointer                    	id=2	[master pointer  (3)]
⎜   ↳ Virtual core Xtst pointer               	id=4	[slave  pointer  (2)]
⎜   ↳ mt0                                     	id=6	[slave  pointer  (2)]
⎜   ↳ Macintosh mouse button emulation        	id=8	[slave  pointer  (2)]
⎜   ↳ TPPS/2 IBM TrackPoint                   	id=9	[slave  pointer  (2)]
⎣ Virtual core keyboard                   	id=3	[master keyboard (2)]
    ↳ Virtual core Xtst keyboard              	id=5	[slave  keyboard (3)]
    ↳ ThinkPad Extra Buttons                  	id=7	[slave  keyboard (3)]
    ↳ AT Translated Set 2 keyboard            	id=10	[slave  keyboard (3)]
    ↳ Video Bus                               	id=11	[slave  keyboard (3)]
⎡ TUIO MD 0 pointer                       	id=15	[master pointer  (16)]
⎜   ↳ mt0 subdev 0                            	id=12	[slave  pointer  (15)]
⎜   ↳ TUIO MD 0 Xtst pointer                  	id=17	[slave  pointer  (15)]
⎣ TUIO MD 0 keyboard                      	id=16	[master keyboard (15)]
    ↳ TUIO MD 0 Xtst keyboard                 	id=18	[slave  keyboard (16)]
⎡ TUIO MD 1 pointer                       	id=19	[master pointer  (20)]
⎜   ↳ mt0 subdev 1                            	id=13	[slave  pointer  (19)]
⎜   ↳ TUIO MD 1 Xtst pointer                  	id=21	[slave  pointer  (19)]
⎣ TUIO MD 1 keyboard                      	id=20	[master keyboard (19)]
    ↳ TUIO MD 1 Xtst keyboard                 	id=22	[slave  keyboard (20)]
⎡ TUIO MD 2 pointer                       	id=23	[master pointer  (24)]
⎜   ↳ mt0 subdev 2                            	id=14	[slave  pointer  (23)]
⎜   ↳ TUIO MD 2 Xtst pointer                  	id=25	[slave  pointer  (23)]
⎣ TUIO MD 2 keyboard                      	id=24	[master keyboard (23)]
    ↳ TUIO MD 2 Xtst keyboard                 	id=26	[slave  keyboard (24)]

And now the subdevices will control individual cursors.  If the goal is to control independent cursors, then it would be ideal to write a script/app to make these hierarchy changes automatically, rather than doing it manually every time.  This is something I will probably put together in the next few days.

Currently this is where the driver is at.  It will route events through the “core” device and the subdevices.  If there are more active blobs or objects than there are devices to send events through, then they will have to wait until another blob/object is removed and there is a free device to send events through.

Hopefully this has been making sense – if it hasn’t, please let me know and I can help clear it up!  Also, if anyone has any suggestions, I would really appreciate that as well.

You might be wondering what we can actually do at this point with the tuio driver.  Without a large list of Xi2 applications to play with, there isn’t much we can do with it out of the box.  This week I am going to try to modify several TUIO based applications to receive XI2 input by modifying their input layer – the layer which receives and parses tuio messages.  I will make posts as progress is made, and maybe even have a video by the end of this or next week.

Comments are encouraged!  Thanks!

⎡ Virtual core pointer                        id=2    [master pointer  (3)]
⎜   ↳ Virtual core Xtst pointer                   id=4    [slave  pointer  (2)]
⎜   ↳ mt0                                         id=6    [slave  pointer  (2)]
⎜   ↳ Macintosh mouse button emulation            id=8    [slave  pointer  (2)]
⎜   ↳ TPPS/2 IBM TrackPoint                       id=9    [slave  pointer  (2)]
⎣ Virtual core keyboard                       id=3    [master keyboard (2)]
↳ Virtual core Xtst keyboard                  id=5    [slave  keyboard (3)]
↳ ThinkPad Extra Buttons                      id=7    [slave  keyboard (3)]
↳ AT Translated Set 2 keyboard                id=10    [slave  keyboard (3)]
↳ Video Bus                                   id=11    [slave  keyboard (3)]
⎡ TUIO MD 0 pointer                           id=15    [master pointer  (16)]
⎜   ↳ mt0 subdev 0                                id=12    [slave  pointer  (15)]
⎜   ↳ TUIO MD 0 Xtst pointer                      id=17    [slave  pointer  (15)]
⎣ TUIO MD 0 keyboard                          id=16    [master keyboard (15)]
↳ TUIO MD 0 Xtst keyboard                     id=18    [slave  keyboard (16)]
⎡ TUIO MD 1 pointer                           id=19    [master pointer  (20)]
⎜   ↳ mt0 subdev 1                                id=13    [slave  pointer  (19)]
⎜   ↳ TUIO MD 1 Xtst pointer                      id=21    [slave  pointer  (19)]
⎣ TUIO MD 1 keyboard                          id=20    [master keyboard (19)]
↳ TUIO MD 1 Xtst keyboard                     id=22    [slave  keyboard (20)]
⎡ TUIO MD 2 pointer                           id=23    [master pointer  (24)]
⎜   ↳ mt0 subdev 2                                id=14    [slave  pointer  (23)]
⎜   ↳ TUIO MD 2 Xtst pointer                      id=25    [slave  pointer  (23)]
⎣ TUIO MD 2 keyboard                          id=24    [master keyboard (23)]
↳ TUIO MD 2 Xtst keyboard                     id=26    [slave  keyboard (24)]

TUIO X Input Driver – Current Progress (Part 1)

Ahh, sorry for the lack of updates, I will *try* to be better in the future.  I’m hopefully at a point where I can post updates more frequently – previously it was hard without much to show.

With regards to my last couple posts on here – my proposal was accepted and I’ve been working on the project the past 5-6 weeks, in case you aren’t coming from the nuicode site.

Here is a video to show the current state of the driver:

Ch ch ch changes

First, let me begin by talking a little about the original goals and what has changed, particularly with regards to the driver implementation itself.  Here is a snippet from the origianl proposal:

What I am proposing is a TUIO xorg driver (xorg-input-tuio) written in C, that will receive TUIO events from a tracker and send them off as XBlobEvents. It would then only be a matter of writing an application that can receive these special events to enable MT functionality. One possible solution for receiving the TUIO events is to simply open a socket on port 3333 and listen for TUIO UDP packets, all within the driver.

Let’s pick this this apart a little bit.

and send them off as XBlobEvents

Shortly after submitting my proposal I received a reply from Peter Hutterer about several question I had, and was instantly notified that the XBlobEvent branch was orphaned and would not be usable/compatible with the current upstream X.  This came as a surprise to me, so I updated my GSoC proposal and reported that I would instead route unique blobs through unique Master Devices (MDs) (you can consider an MD to be an individual cursor).  Some things have changed there a bit as well, but I will get to that a little further down.

It would then only be a matter of writing an application that can receive these special events to enable MT functionality.

So, of course, this has changed because these so called “special events” aren’t being used and don’t exist yet.  Instead, events are routed through individual Slave Devices (SDs) (which I also talk more about) which should be attached to separate MDs.  In terms of writing applications, an application needs to be XI2 aware – Peter Hutterer has been writing on this recently (Peter created MPX and works on XI2).

One possible solution for receiving the TUIO events is to simply open a socket on port 3333 and listen for TUIO UDP packets, all within the driver.

This is about the only thing that hasn’t changed.  This is what the driver does now, and fairly soon will support arbitrary devices as well (e.g. /dev/tuio).

Devices (SDs)

Let’s discuss devices a little, and how they come into play in this project.

Without XBlobEvents in play right now we need a way to communicate separate events in X – this has to be done through separate input devices (if your running X, type `xinput list` into a terminal – those are your input devices).  To do this we need to get a hold of “dummy” devices that we can use solely to send events through – there are probably several ways to do this, but I know of two, courtesy of Peter Hutterer.  The first way is to use libhal to tell HAL to create a new input device.  X, so intently listening, will receive the message about a new device and create it (example).  The second option is to tell X directly using `NewInputDeviceRequest`, which, is a little more difficult to use because it doesn’t take care of some of the cleanup (example).  So, I went the HAL route.

Blocking?  Eek…

Using the example above, I put together code to create new devices fairly quickly.  What it boils down to is creating a new device through HAL each time a new object is found and destroying it when the object is no longer around (I am going to refer to blobs as object from here on out.  At the moment I’m using the /tuio/2dcur OSC address space, but later on this may /tuio/blb, so I’ll just say “object”).   When a new TUIO device is created, my driver will pick it up and start routing the events for the associated object through that device – or, at least, this is how it should work.  I worked with this approach for awhile and it was working when the waters were calm, but as soon as I would start creating/destroying multiple objects many times a second X would freeze.  Why?  Hmm… attach gdb… Okay… `bt`… dbus???  Communication with HAL is done through DBUS, so that was fine, but it was just sitting there in some function call.  Looking at the trace I also saw the function call `dbus_connection_send_with_reply_and_block()`… GREAT.  I scoured the internet looking for a way to make asynchronous calls, but with little luck.  I believe you can set a callback to make it non-blocking, but I couldn’t find a whole lot out on this, and I wasn’t even sure if it would work – after having it basically crash on me, I was already feeling edgy about using HAL. (If anyone can offer help with that, let me know 🙂 ).

Option #2

To give some context, the previous paragraph took place only around 1.5-2 weeks ago.  Previous to that I had run into various issues getting everything up and running, finally having  single device/cursor working after those first weeks.

It was around last Thursday that I thought I had things working, and that I was going to start working on multiple cursors (no, multiple devices doesn’t necessarily mean multiple cursors).  Then, Sunday night, I find out that I’m having all these problems and I have no solution.  Quickly, I try the second method that Peter referred me too and… seg fault.  I quickly diffused that bug (I’m lying, that one actually stole a good chunk of time 🙂 ) and moved on.  Okay, things were checking out and everything seemed to be working as before.  Add some code to remove the devices as well and… BAM, random errors that don’t seem like they have anything to do with my code.  So here I am, it’s Monday some time, and I want to make a post Tuesday/Wednesday that actually shows progress (I did promise it, after all.  And, *cough*).  So what do I do?  I try to fix it, but without a whole lot of an idea on what to do.  At that point I took a break, dropped that approach, and tried to fix the HAL-based code.  I still think the second option may be better, I just need to figure out what is expected when using those commands.  Oh, and, by the way, it’s soon Tuesday and my internet was out from 10am – 5pm.  Talk about feeling helpless – kill Google, kill knowledge.

Compromises

Wednesday (yesterday) rolls around and I’m getting a little down.  Everything I’ve tried seems promising, then responds with errors that I just don’t know how to approach.  Then it occurred to me, “HAL is my friend until I’m chatting too much with it… Hmm… Why don’t I just have a quick conversation when I first start start up, then ignore it?”  So that’s what I did.  When the driver first starts it creates some number of devices through HAL, and just reuses them – this is in contrast to constantly adding/removing devices.  No more worrying about HAL blocking.  Everything seems to work this way, and I haven’t experienced any crashes/freezes yet.  Obviously it could block when first initializing, so I will need to fix that soon.

MDs and Cursors

Something I haven’t addressed yet is MDs and having separate cursors.  I’m not necessarily going to give a crash course on how MPX/XI2 works, but essentially there are two different types of devices – Masters and Slaves.  You would associate the mouse you’re using right now to a Slave device, along with any other physical input device you’re using (mouse, keyboard, tablet, etc.).  Each one of these is generally attached to a Master device.  Each Master device takes the form of a cursor on your screen.  In the general case you have one Master device, with all of your Slave devices attached to it, and all their events are routed through it.  So, in order to have multiple cursors you need to create multiple cursors, and to create multiple cursors you need to use the XI2 api to create MDs and attach SDs to them – in particular, create a new MD when a new tuio “dummy” SD is created, and attach the “dummy” SD to it…

More to say on that, but I will have to continue this tomorrow in a “part 2” because I need to get some sleep :).

NUI GSoC Proposal

I submitted my proposal GSoC to work with the NUI Group this summer, so we’ll see what happens! 🙂

If you are interested, or someone reviewing my proposal, here is a link to the pdf version of the proposal:  MPX TUIO Driver Proposal

I wasn’t able to put in as much time as I would have liked into the proposal, having been extremely busy lately.

It’s going to be a long wait until the 20th…

Hello! (Introduction)

I will be using this blog to track my experiences with new and innovative ways to interact with computers; in particular, I will be exploring the use of multi-touch (mt) screens to create a more immersive computer experience.  I’ve always had an interest in this area of computer interfacing, but it wasn’t until recently that I found a cost-effective way to create and tinker with any of this equipment myself. The Natural User Interface Group is an organization of open source developers creating multi-touch software for cost-effective mt hardware.  I urge you to check out their site and read more!

Between work and a hefty load of school work, I have managed to put together an MTmini – basically, a cheap way to create an mt surface using a picture frame, a cardboard box, printer paper,  and a webcam (in my case, a Playstation Eye).  Using software such as TouchLib, the webcam can pick up shadows on the paper+glass surface, and distinguish them as “blobs.”  At this point a software program can do endless things with this information – create cool particle effects at the point of touch, track drags of the finger and translate these to the movement of on-screen objects, etc. – it really is limitless!

The MTmini that I mocked up needs to be refitted into a larger box, but until I get a larger piece of plexiglass this will have to suffice.  Here is a picture:

Okay, quit laughing.  And yes, that is a Tootsie Pop box.  Haha :).  I should probably tape the entire box to avoid light leaking in, but I will  most likely just replace it with a larger box and a piece of plexiglass pretty soon here anyway.  If I can, I would like to put together an FTIR setup (without a projector or lcd) just to get a little experience with it and to have a potentially larger surface to work with without the need for it to be enclosed (although I believe I would need an IR bandpass filter to exclude the need for an enclosure).

So there’s my introduction.  Sort of.  I guess I didn’t talk about myself, but you can always go to the About Me section to find out about that.  Thanks for reading!