Rx: Event composition – single-valued

In a first blog post on the topic „event composition with Rx“ I presented operators that take several event input streams and create a single output streams, whose elements contain values from each input event („multi-valued“ event composition operators). This second blog post instead treats “single-valued” event composition. Multiple streams go into those composition operators and one single stream comes out, where each element of the output stream matches exactly one event/value from any input stream!

Example solution, PPT slides and general information

In general, all introductory information on composition, description of the „multi-valued“ event composition operators and more bits-and-pieces can be found in my first blog post.

I’ve created a SL3 demo project where you can experiment with each composition operator. You can find the sourcecode (SL3 VS2008 project) here and the live demo on Windows Azure here.

The PPT slides that contain all diagrams of the composition operators can be downloaded here.
I would be glad if you could leave my name and/or the blog URL (www.minddriven.de) on every diagram.

Merge()

The operator Merge does serialization of all input event streams by firing an event everytime one of the input streams fires. The event/value of the output stream will be the same as the event/value on the input stream that has fired.

The following code shows the simple usage of Merge:

    var composition = 
        Observable.Merge(stream1, stream2, stream3);

And the following example flow diagram shows the behavior of the operator under this code. Note how the incoming events are written to the output stream in exactly the same order as they come in:
Rx Event Composition - Merge

Amb()

Amb comes from ambiguous and has the following semantics. If you combine several input event streams with Amb, the output stream will only contain the events from the input stream that fires first. The other input streams are not taken into account afterwards. That’s a kind of winner-takes-all-strategy and could be valuable in some situations.

Amb can be used as follows:

    var composition = 
       stream1.Amb(stream2).Amb(stream3);

And the following diagram shows an example composition situation. Note that stream 1 fires first and consequently Amb only forwards events from this stream:
Rx Event Composition - Amb

Concat()

Concat does (the name says it all) concatenation of the input event streams. But wait: how can this work? In fact Concat only works when you have finite event streams! On infinite streams, Concat watches the very first stream forever and the other streams don’t come to their turn. Thus use Concat on finite event streams, when serialization of the input streams in the right order is crucial.

The following code shows the use of Concat on three finite input streams:

    var composition = 
        stream1.Concat(stream2).Concat(stream3);

And here’s the concatenation done by this code in an example situation (the square brackets illustrate finite streams):
Rx Event Composition - Concat

Until()

The operator name says it all: Until forwards events from an input stream to the output stream until an event on a second stream occurs.

The following code shows the usage:

    var composition = 
        stream1.Until(stream2);

And the following diagram shows the behavior of Until on 2 streams based on this code:
Rx Event Composition - Until

WaitUntil()

Again the operator name speaks for itself: WaitUntil does not forward events from an input stream to an output stream until an event on a second stream occurs. From then on every event of the first input stream will be forwarded.

Here’s some code that shows how WaitUntil works:

    var composition = 
        stream1.WaitUntil(stream2);

And the following diagram shows WaitUntil’s behavior based on this code in an example situation:

Rx Event Composition - WaitUntil

That’s it again. I hope you finde this information useful. As in the blog post on multi-valued event composition, I want to encourage you to give me active feedback and to improve the examples and descriptions of the composition operators above. Thanks!

kick it on DotNetKicks.com

Rx: Event composition – multi-valued

In preparation of some presentation on the Reactive Extensions (Rx) last week I’ve investigated the composition methods for several event streams on the Observable class. What methods are existing and what are they doing? Let’s take a look…

This first blog post treats „multi-valued“ event composition in terms of multiple streams going in and one stream comes out, but each element of the output stream is composed by sub-elements from each input stream! There is a second blog post that treats „single-valued“ event composition as well.

Example solution

I’ve written a little SL3 application where you can discover the behavior of those composition operators. It builds on example code that first has been published by Paul Batum some months ago and extends his code with some more possibilities. I’ve hosted the SL3 on Windows Azure where you can test it directly:

Rx Composition Demo

You can download the sourcecode of the Visual Studio 2008 solution as well: [Event Composition Sourcecode (ZIP)]
All you need to run the solution is the latest Rx for SL3 build, Visual Studio 2008 and the Silverlight 3 SDK.

Powerpoint Sources

If you want to use my diagrams in your presentations or somewhere else, here is the source PPT file: [Rx Event Composition PPT]
I would be glad if you could leave my name and/or the blog URL (www.minddriven.de) on every diagram.

How to read the diagrams and code

The general visualization concept is taken from Eric Meijer’s PDC session about Rx. Thanks Eric for your great work!

Rx Event Composition - General Visualization

The visualization shows an Rx composing operator in the middle. On the left there are event input streams, on the right is the composed output stream. The order of the events on a timeline is from right to left: the most right event goes first into the operator and comes first out in composed form. This is visualized by the t1, t2, t3 and t1‚, t2‚, t3‚ points.

The sourcecode snippets below are kind of abstract. They only show how the operators are used. They are based on 3 streams stream1, stream2 and stream3. Those streams can be seen as general abstract event streams, no matter which datatype they’re really using.

Now let’s come to our multi-valued composition operators…

SelectMany()

The SelectMany operator should be known from LINQ on IEnumerable or others and is reflected by the from keyword in LINQ queries as well. It projects a foreach loop or several cascaded foreach loops, if you execute multiple SelectMany calls after another. Thus the order in which SelectMany is executed is relevant!

The same is true for SelectMany on IObservable. Take the following code:

var composition = 
    from element1 in stream1
    from element2 in stream2
    from element3 in stream3
    select new[] { element1, element2, element3 };

This is equal to the following direct use of SelectMany:

var composition = 
    stream1.SelectMany(
        element1 => stream2.SelectMany(
        element2 => stream3.SelectMany(
        element3 => Observable.Return(new[] { element1, element2, element3 }))));

With this code an example composition scenario could be:

Rx Event Composition - SelectMany

Note that the event order is crucial. The first event on every input stream is in the right order, thus yielding to a first composed event. The second event on input stream 2 first fires when the third event on stream 3 is fired. The resulting event in this case contains the first event from stream 1, since it retains the right event order.

CombineLatest()

CombineLatest is a very useful operator. It fires everytime when one input stream fires. Then the corresponding output stream contains the latest values/events from each input stream.

Here’s the code how to use the operator:

var composition = 
    stream1.CombineLatest(
        stream2, (element1, element2) => new[] { element1, element2 })
    .CombineLatest(
        stream3, (elements12, element3) => new[] { elements12[0], elements12[1], element3 }));

And an example composition follows:

Rx Event Composition - CombineLatest

Zip()

In comparison to CombineLatest and others the Zip operator doesn’t use any results from an event stream multiple times. However it builds a queue of the events on every input stream. When every input stream queue contains at least one event, the Zip composition operator fires. Thus it synchronizes events, whereas every event on an input stream is waiting for an event on the other input streams.

The following code shows the usage of Zip():

var composition = 
    stream1
        .Zip(stream2, (element1, element2) => new[] { element1, element2 })
        .Zip(stream3, (elements12, element3) => new[] { elements12[0], elements12[1], element3 });

And the following diagram shows an example composition. Note that no event is used twice in the output stream and compare this with CombineLatest and the other operators:

Rx Event Composition - Zip

Note that the last event on stream 3 has no „partner“ on the other streams and thus it doesn’t appear on the output stream in this example.

Also notice that Zip() is equal to Join() if used with a conjunction of all input event streams as parameter:

var composition = 
    Observable.Join(
        stream1.And(stream2).And(stream3)
            .Then((element1, element2, element3) => new[] { element1, element2, element3 }));

Join() – 2 out of 3

The Join operator can be used in many cases and is very flexible. The method Join() can take several event compositions as parameters. Everytime one composition fires, the whole operator fires. That said, if you create a conjunction of all your event streams, then the composed output stream will be equal to Zip().

But Join gives you more power and flexibility. One example is a „2 out of 3“-situation. Imagine you have 3 input streams and want Join to fire everytime when 2 input streams contain an event. This can be done by passing Join the appropriate partial compositions as shown in the following code:

var composition = 
    Observable.Join(
        stream1.And(stream2)
            .Then((element1, element2) => new[] { element1, element2 }),
        stream1.And(stream3)
            .Then((element1, element3) => new[] { element1, element3 }),
        stream2.And(stream3)
            .Then((element2, element3) => new[] { element2, element3 }));

This yields to the following resulting composition for example input streams:

Rx Event Composition - Join 2 out of 3

Again in this example, the last event on stream 3 has no partner and doesn’t appear on the output stream (due to an odd amount of events over all input event streams).

ForkJoin()

ForkJoin can be used if you want a composition to fire exactly once. When all input event streams contain at least one event/value, the ForkJoin operator fires with the first event on every stream. Afterwards it doesn’t react to any changes/event on the input streams again.

Here’s the code how ForkJoin can be used:

var composition = 
    Observable.ForkJoin(stream1, stream2, stream3);

And here’s the diagram of a composition example:

Rx Event Composition - ForkJoin

That’s it for now, guys. I hope it’s valuable for you. Since these are just my initial ideas on this topic, I’m reliant on your feedback. Together we could make all those descriptions even more useful.

kick it on DotNetKicks.com

Rx: Bing Translate Example

In his Rx session at PDC09 Eric Meijer gave a nice example named Bing Translate. It was connecting to Bing via the Bing Web Service API and translated some term given by the user. It was quite interesting because Eric used the Reactive Framework for handling the translation and did some composition of the incoming results. He took just the first 2 translations and discarded the others. This didn’t make much sense, but it was ok for demonstration purposes. As followup Somasegar described the example in his blog.

But there was a lack of the whole sourcecode (btw: Eric Meijer posted it now on the Rx forum, but not as VS solution) and so I implemented it on my own with some modifications and now I want to share the whole Visual Studio solution with you.

Live demo

I’ve put my Bing Translate example as SL3 application on Windows Azure at http://bingtranslate.cloudapp.net, where you can directly test it:

Rx Bing Translate Example - Reactive Extensions - Reactive Framework

Sourcecode requirements

Since the Bing Translate example uses the Bing Web Service API, you need to get a free AppId here. Additional information on the Bing API can be found on the Bing Developer Center. When you’ve obtained your AppId, insert it into the BingService.cs file (field AppId of class BingService) of the sourcecode below. Now you should be able to run the solution.

What you furthermore need to open and run the solution:

  • Visual Studio 2008
  • Silverlight 3 SDK
  • Reactive Extensions for SL3 (latest release)

Now give me the code!

Ok, you want it, you get it 😉

[Visual Studio Solution for Silverlight 3 (ZIP)]

Notes on the code:

  • Contrary to Eric’s solution, mine will not synchronize the translation events when using the SetUpTranslationService() method in the MainPage.xaml.cs file. Use the SetUpTranslationServiceWithJoin() method if you want to have some synchronization. The main advantage of Rx is the composition of event streams, so this synchronization shows how simple things could be.
  • The code’s style is not expressing my real coding skills. Since this is just an example project, I did not make it stylish and bullet-proof.

That’s it! Have fun and please feel free to give me feedback.

kick it on DotNetKicks.com

News on Rx (Reactive Extensions for .NET)

Rx Logo - Reactive Framework - Reactive Extensions for .NETAs written before the Reactive Extensions for .NET (Rx, aka „Reactive Framework“) have been published as pre-release on their own DevLabs-site. The Rx-team is asking for feedback from you as developers and will include improvements into timely released development packages, until they’ll ship the final pieces with .NET 4.0. Check out the MSDN forum to give your feedback or ask for support.

So what’s new in Rx compared to the early DLL from the Silverlight 3 toolkit sources?

In general, many improvements were made on the API and functionality: methods on the Observable class have been rearranged and extended to give you full flexibility in many use cases. One method that I really love is Throttle(). Imagine that you have a textbox and everytime the user makes an input (without pressing Return) a data source is queried for items that match this input in some way (e.g. filtering by name or something). Now you discover that you’ve got performance issues. The search is done as the user types the first character and that’s not what you want. You only want to react to user’s input if he isn’t typing anything for -let’s say- 1 second. Throttle() solves this for you by one method call, you do not need anything else or do something manually. That’s great! Try to develop this with common event-based programming and you know what I mean…

Then the Rx-team intends to converge the functionality of IEnumerable and IObservable. IObservable has much of the functionality of IEnumerable and LINQ operators make composing, filtering and transformation of event streams very simple and intuitive. And in the DevLabs-release of Rx the Rx-team is moving some functionality from IObservable (such as the Amb() method) to IEnumerable as well. If you need it or not: this is leveraging the dualism of both concepts for developers and makes both interfaces more intuitively usable.

Furthermore the ground base of Rx is now Px (or PFx), the Parallel Extensions for .NET that will be part of .NET 4.0 as well. For every concurrent operation Px is taken and does the whole work (by creating a Task<T>). As a side effect of supporting Rx for .NET 3.5 SP1, the Rx-team has back-ported Px from .NET 4.0 to .NET 3.5 SP1 and made it usable for this platform as well. Great job!

One thing very cool and worth to mention are Subjects as new components in Rx. Subjects implement the type ISubject<S,T> and are kind of hybrids between IObserver<S> and IObservable<T>. A Subject implements both IObservable and IObserver and can be seen as „channel“ or „transforming storage“. The channel behavior is that you can push values as input into the subject via the IObserver methods (OnNext(), OnCompleted() and OnError()) and that you can subscribe to the Subject via the IObservable.Subscribe() method to get the output from the Subject. The Subject itself can do some transformation with the values that are pushed into it, can store the values and is able to take responsibility of what happens if someone subscribes to it. Subjects act as storage as well. When values are flowing into the Subject it can store them (or only the last or some transformed value, …) and when someone subscribes it can return all stored values (or only the last etc.). That means that e.g. an asynchronous operation needs not to be executed multiple times when using Subjects. The return value is stored in the Subject and can be extracted everytime a client subsribes to the Subject (independent of the moment of the subscription).
Dependent on the intent of the Subject, Rx comes with several implementations that differ in their transformation and output behavior:

  • AsyncSubject<T>: This subject is used for asynchronous operations where only one value is expected as result. Thus the AsyncSubject remembers only the last value.
  • BehaviorSubject<T>: This subject always has a current value and fires everytime the value changes. Thus it’s kind of the INotifyPropertyChanged event.
  • EventSubject<T>: This subject corresponds to an event stream and thus will never invoke OnError() or OnCompleted().
  • ReplaySubject<T>: This subject is like a tape/video recorder. It records all events that are pushed into it and can be replayed. On replay it will return all events that have been recorded in exactly the same order and time interval in which they occurred.

Another really nice development is an Add-In for Visual Studio: Reactive Extensions Generator. It overcomes the problem that all types in the .NET Framework rely on .NET events and don’t offer methods or properties that return corresponding instances of IObservable. This tool allows you to choose an assembly and it generates extension methods for all .NET events that it finds on the components of the assembly. Those methods will return the Observable pendants for those events instead. Really impressing stuff and very timely! Single drawback: it can be used only with  Visual Studio 2010 and .NET 4.0.

So to sum up again: I think Rx is valuable in many situations. First it’s a better pattern for reactive programming that unifies asynchronous programming (that normally returns just one value) and event-based programming (that typically has an infinite event-stream) and second it’s very strong when you want to compose (+ filter and transform) several event streams and when you want to build up and react to very special and complex events. Stay tuned for more information and examples on that in upcoming blog posts.

kick it on DotNetKicks.com

Reactive Framework: it’s here!

Short announcement: Microsoft has published the Reactive Framework as Reactive Extensions for .NET (Rx) on its own DevLabs page (see release notes). Check it out now!

Rx
(Have you noticed the former Volta logo? ;-))

New Channel9 video are available, too: Rx in 15 minutes, Reactive Extensions intro, first Rx app).

And they’ve got a new Rx Team Blog.

I will give it a look at this weekend.

Reactive Framework (Rx) – first look

During the last weeks I took some time to have a first look at the Reactive Framework. Here I want to share some impressions and information with you.

Rx: What is it about?

The Reactive Framework (Rx) has been developed at the Microsoft Live Labs and is another creation by Erik Meijer, the father of LINQ. Rx will be part of the upcoming .NET Framework 4.0, but is already included as preview DLL (System.Reactive.dll) in the sources of the current Silverlight 3 Toolkit. It’s used there to get some smoother Unit Tests to work.

The aim of Rx is to simplify/unify complex event-based and asynchronous programming by providing a new programming model for those scenarios. That’s very important in a world which becomes more and more async (Web, AJAX, Silverlight, Azure/Cloud, concurrency, many-core, …) and where present approaches (event-based programming with .NET events, Begin/Invoke) hit the wall. Program logic gets hidden in a sea of callback methods which leads to nasty spaghetti code. Those of you who are creating larger async programs know what I mean 😉

Basic ideas

The main concept of Rx is reactive programming and that’s not new at all. It involves a distinction of data processing methods into push and pull scenarios:

  • In a pull-scenario your code/program is interactively asking for new data from the environment. In this case the actor is your program while it pulls new data from the environment.
  • In a push-scenario your program is reactive. The environment in pushing data into your program which has to react on every new incoming data item (see illustration below). Reactive programs are ubiquitous: every event-handler (e.g. for UI events) is an example for a reaction in a push-scenario.

Reactive Programming

The guys at Microsoft got a very important insight which builds the fundament of Rx: there’s a dualism between the Iterator and the Observer pattern! This means that both patterns are essentially the same, if you dismiss their practical intentions for a moment.
With the Iterator pattern you get a sequence of data which is iterated in the pull-mode: the Iterator is the actor and asks the iterable object for data.
The Observer pattern can be interpreted equivalently: the environment pushes data into your program (e.g. through events), which has to react. As result with repeatedly fired events you get a sequence of data, which makes the relationship to the Iterator pattern obvious.

With this, an Observer in the push-scenario is equal to an Iterator in the pull-scenario and that’s the key insight, which has led to the development of the Reactive Framework. 14 years had to pass from the publication of the GoF Design Patterns book until now to figure out this intense relationship of the Iterator and the Observer pattern.

LINQ to Events

Rx realizes the Observer pattern with the two interfaces IObserver and IObservable which are the pendants for IEnumerator/IEnumerable of the Iterator pattern. IObservable is a sort of „asynchronous collection“, which represents the occurrence of a certain event at different points in time. Thus it expresses a sequence of data, on which you can run arbitrary actions. In fact IObservable for push-scenarios is essentially the same as IEnumerable for pull-scenarios.

The interplay of IObservable and IObserver is the following: on an IObservable you can register (Subscribe()) an IObserver (Note the dualism to IEnumerable/IEnumerator here!). The IObserver has three methods OnNext(), OnCompleted() and OnError(). These methods are invoked when the IObservable gets new data, completes the data sequence or results in an error. Thus your IObserver implementations take care of the data processing which replaces your custom callback methods.

The real impressing thing with Rx is that with events as data sequences you can use LINQ syntax to process and transform these data sequences (that’s why it’s sometimes called LINQ to Events). To create instances of IObservable and for providing LINQ methods, Rx comes with the class Observable. If you want for example take the first 10 click events on a button but skip the first click, you could easily write:

Observable.FromEvent(button1, "Click")
    .Skip(1)
    .Take(9)
    .Subscribe(ev => MyEventAction(ev));

Subscribe() has some overloads. In this example I’ve directly specified an action which is executed when an event (= a piece of data) comes in. Instead you could give it e.g. your custom implementations of IObserver as well.
LINQ gives you real power in processing, composing and transforming your events as data stream. Even in the simple example above, if you would implement it by event-based programming without Rx, you would need some additional effort. You would have to use temporary variables to skip certain elements, take others and so forth. In general, more complex cases require complete finite state machines which can be circumvented with Rx.

Composing events with LINQ is another typical and impressing example for Rx. Thus, a dragging event on a textblock can be implemented as:

IObservable<Event<MouseEventArgs>> draggingEvent =
    from mouseLeftDownEvent in Observable.FromEvent(textBlock1, "MouseLeftButtonDown")
    from mouseMoveEvent in Observable.FromEvent(textBlock1, "MouseMove")
        .Until(Observable.FromEvent(textBlock1, "MouseLeftButtonUp"))
    select mouseMoveEvent;
draggingEvent.Subscribe(...);

In this example, with the from keyword the two events "MouseLeftButtonDown" and "MouseMove" are chained together. Everytime the left mouse button is pressed on the textbox, the „gate“ opens to react on mouse move events. The „gate“ closes again when the left mouse button is released. So the composed stream includes all events where the left mouse button is pressed and the mouse is moved – the typical dragging event!

There are other possibilities to compose events, which for example consider the order of events differently. Those build the interesting parts of the Reactive Framework and clarify the use cases, where Rx has real advantages over present event-based .NET programming.

Bottom line

Personally I like the new way of thinking of events as data sequence! The dualism of the Iterator and the Observer pattern has nothing artificially constructed, but some sort of purity and mathematical elegance for me. But one big question remains: do we need this in our everyday’s developer life and which advantages come with it?

Erik Meijer himself says about Rx:

Of all the work I’ve done in my career so far, this is the most exciting. […] I know it’s a bold statement, but I really believe that the problem of asynchronous programming events has been solved.

That’s a really strong message and it has some important weight since Erik is the creator of LINQ and has a big expertise in language design. Does Rx hold what he promises?

In my opinion Rx is really beautiful and valuable when you do complex asynchronous operations, where you have to compose and orchestrate events/data from several sources. With Rx you’re able to chain and filter events to easily create the concrete event, to which you want to react. That’s the power of LINQ, which allows transforming the resulting data as well. LINQ gives you the whole flexibility of processing the data stream that you already know from LINQ to Objects (on IEnumerable) etc.

Beyond that at the moment I don’t see a real big impact in many current development scenarios. If you have to handle simple events or do simple async computations, you are still fine with the present event-based programming model. Of course with Rx you can write your own observers to prevent a sea of callback methods, but encapsulation of handler logic can be done without that as well. Thus at the end I like the concepts of Rx (Observer, LINQ, …) and I’m curious to see how this whole thing develops after the release of .NET 4.0, but I’m not over-enthusiastic at the moment.

Links

Finally here is a list of links to some interesting websites regarding the Reactive Framework:

kick it on DotNetKicks.com