Multi-Touch Reactive Extensions

Link: Reactive Extensions for .NET

It's been a while since I've played with MultiTouch functionality so I decided it was time to dust off the old TouchSmart and put it to good use.  The purpose wasn't to play with MultiTouch functionality but more to see how we could simplify managing isolated events using RX.  The MultiTouch API is good for this because by its very nature MultiTouch requires us to collect multiple events simultaneously and wire them up to create gestures.  Unfortunately the current iteration of MultiTouch API wrappers for Silverlight and WPF (touch, MIRIA etc) are quite basic and when working with custom gestures you are forced to introduce a lot of global code such as flags etc.  Not pretty!

Imagine the following scenario.

• 2 touch points on screen (let call them HoldRight and HoldLeft).  Both expose 2 events Hold and Release
• When a person presses and hold a finger over BOTH points trigger an event
• If only one is being pressed and held do not fire the event
• If the user lifts their finger or fingers off the points and repeats the process the event should fire again.

This is a fairly simple example but actually wiring this up requires the use of nasty global variables and a bit of repetition.  Here is a very quick implementation of this.

public partial class MainPage : UserControl
{
    private bool _rightHeld = false;
    private bool _leftHeld = false;

    public MainPage()
    {
        HoldLeft.Hold += HoldLeft_Hold;
        HoldRight.Hold += HoldRight_Hold;
        HoldLeft.Release += HoldLeft_Release;
        HoldRight.Release += HoldRight_Release;
    }

    private void HoldRight_Release(object sender)
    {
        _rightHeld = false;
        DoCheck();
    }

    private void HoldLeft_Release(object sender)
    {
        _leftHeld = false;
        DoCheck();
    }

    private void HoldRight_Hold(
        object sender,
        GestureHoldEventArgs e)
    {
        _rightHeld = true;
        DoCheck();
    }

    private void HoldLeft_Hold(
        object sender,
        GestureHoldEventArgs e)
    {
        _leftHeld = true;
        DoCheck();
    }

    private void DoCheck()
    {
        if (_rightHeld && _leftHeld)
        {
            // Fire the event
        }
    }
}

As you can see I have made use of 2 global variables a DoCheck method that checks if the 2 canvases are currently in a "held" state and fires the event if they are.  All the event handlers do nearly the same thing.  Sure I could refactor this to use maybe 2 event handlers and inspect the sender but that starts getting messy.

Reactive Extensions to the Rescue!

Reactive extensions let us "compose" events so we can pass them around and filter them like first class citizens.  Because of this we can actually create custom events by combining.  Tackling the same issue as above we can achieve the same functionality without the need for dodgy global vars and boilerplate code.  Lets look at the code first.

public partial class MainPage : UserControl
{
    public MainPage()
    {
        // compose events
        var leftHold = Observable.FromEvent<GestureHoldEventArgs>(
            HoldLeft, "Hold");
        var rightHold = Observable.FromEvent<GestureHoldEventArgs>(
            HoldRight, "Hold");
        var rightRelease = ObservableEx.FromMultiTouchReleaseEvent(HoldRight);
        var leftRelease = ObservableEx.FromMultiTouchReleaseEvent(HoldLeft);

        // subscribe to dual hold event
        leftHold.Zip(rightHold, (l, r) => l)
            // listen until either release triggered
            .TakeUntil(leftRelease.Amb(rightRelease))
            // trigger the event
            .Subscribe(_ => { /* trigger event */ });
    }
}

public static class ObservableEx
{
    public static IObservable<IEvent<EventArgs>> FromMultiTouchReleaseEvent(
        TCanvas canvas) {
        return Observable.FromEvent<TCanvas.ReleaseHandler, EventArgs>(
                h => new TCanvas.ReleaseHandler(e => { }),
                h => canvas.Release += h,
                h => canvas.Release += h
            );
    }
}

First things first for some random reason the implementers of MIRIA decided to create en event the actually passed NO event args - this is not normal and could easily be considered and anti-pattern.  Because of this I had to use the long winded Observable.FromEvent which is created as a helper method.

So what are we doing here?

1. Compose our events into first class citizens - Observables
2. Zip left and right hold events.  Zipping basically combines 2 Observables and publishes or emits a value when both observables have emitted a value.  The second argument in Zip is the transform function that converts the 2 emitted values into 1 value.  We don't care about this value so just return some arbitrary value.
3. Take the Zip generated observable and keep publishing it's values until Either the left or right release events are triggers (Amb = most ambitious - publishes first value to appear)
4. Subscribe to this super composed event.  When this happens we can fire our event

What actually happens on the front end is irrelevant (the solution looks like this and will spin when the two Thumb areas are pressed and held)

[[posterous-content:ECxarBiHmvIvJutnbJaJ]]

But the fact we have managed to combine 4 isolated events without having to use boilerplate is very nice.  This gives us a lot of power to create and control complex gestures - not limited to MultiTouch but any sort of UI interactions (Mouse events, Web service calls etc).

* I'll keep saying this.  RX doesn't do anything new or solve any unsolvable problems but what it does do is allow us to do things in a neater way.  Now there is a bit of learning curve in it - you really need to start thinking in RX but once you wrap your head around RX is a nice tool to have on your tool belt.  No it's not complexity for complexity's sake - I honestly believe the code above, when used in the real world, will hep reduce complexity and make maintenance easier.

Good Fortune Awaits!