LightBlue: An alternative to the Azure Emulator

For certain projects, my team uses a componentized and distributed system architecture based on CQRS and Event Sourcing. We find that for complex domains that add significant business value, this sort of architecture is worth the extra investment as it helps ensure that:

  • The system scales.
  • The system is easier to maintain.
  • If the business change their mind or introduce new requirements, it is easier to deal with.
  • When things do go wrong, the system degrades instead of going offline completely.
  • DR environments are relatively easy to build.

We do all this using Microsoft’s PaaS offering as we like to run each component on its own stack, and Windows Azure allows us to easily deploy, update & maintain everything. However… for our team members to build & test components in this environment, there needs to be a way for them to run the system locally. Given the architecture, this might involving running many roles and instances (processes) that are all communicating with each other via a messaging system built on top of Azure Blob Storage. Unfortunately we’ve found the Azure Storage Emulator has significant short comings in this type of environment. It doesn’t scale very well, utilizes lots of system resources and suffers from a number of stability issues. This has ultimately been chewing up lots of developer man hours in maintenance and troubleshooting.

Introducing LightBlue

LightBlue is a development framework that abstracts our dependency on the Azure platform. It provides a light-weight hosting mechanism allowing both worker & web roles to be deployed (without packaging) in a dedicated process. This approach has yielded the following advantages:

  • Faster build times. We found the packaging step required to deploy to the Azure Emulator is very slow. LightBlue does not require packaging.
  • Much faster blob storage access. Under low/moderate load, we’ve found the Azure Storage Emulator consistently takes ~1 second and regularly takes up to 10 seconds to be retrieve a blob of less than 1 KB.
  • Slow & buggy compute emulator UI. It’s just gone.
  • No limits. The Azure emulator limits the number of roles, deployments you are running.
  • Reduced Hard Drive usage. The Azure Emulator seems to use a lot of disk space.
  • Reduced CPU usage. We found that the Azure Storage Emulator used a lot (or all) of the CPU resources available on our developers machines.
  • More Stable. The Azure Storage Emulator regularly falls over under load.

*Warning*

  • We consider LightBlue to be in the “experimental quality band” (it has been in development for 9 days).
  • We do not advise moving your projects over to it at this stage, however we do encourage you to try it out and let us know what you think.
  • Do not expect future versions of the API and emulator to offer any level of compatibility with this early release.
  • LightBlue is currently dependent on Autofac (IoC Container), I’m advocating a different approach here.
  • Azure Queues / Topics are not yet supported (coming!).
  • The “Getting Started” documentation is a bit sparse, we’re working on it!

I’ll keep you up to date as development progresses. Kudos to Colin Scott for all his hard work and late nights on the project.

LightBlue Resources

Solving Puzzles in C#: Poker Hands

I particularly like the solution to this puzzle as it can be easily adapted to any problem involving a deck of cards & the probability of being dealt a particular hand.

In this post I’ll be linking each code snippet to “LINQPad instant share” so you can literally execute the code as we go!

The Question

(courtesy of Mitch Wheat)

“How many five-card hands from a standard (52 card) deck of cards contain at least one card in each suit?”

LINQPad Solution

So as you might have guessed, we’re going to use a brute force approach. Before we can tackle the specific problem, we need to come up with a way of generating all possible hands. I figured the simplest (and reasonably efficient) way to represent a deck of cards is with a sequence of integers, each representing a card in the deck.

Enumerable.Range(0,52)

What’s nice about this, is we can quickly assign each integer a suit & the respective card values using the following query.

Generating Cards (http://share.linqpad.net/rkqnan.linq)

from i in Enumerable.Range(0,52)
let suit = i / 13
let card = (i % 13)
select new { card, suit }

query results: my 4 suits

This step is not necessary, but you might want to assign each card & suit an enum value. I didn’t actually do this, but if you don’t understand what’s going on here, it might help!

Generating Gold Plated Cards (http://share.linqpad.net/5usvcx.linq)

enum Suit { Hearts, Clubs, Diamonds, Spades }

enum Card { Two, Three, Four, Five, Six, Seven, Eight, Nine, Ten, Jack, Queen, King, Ace }

void Main()
{
    var query = 
        from i in Enumerable.Range(0,52)
        let suit = (Suit)(i / 13)
        let card = (Card)(i % 13)    
        select new { card, suit };
    
    query.Dump();
}

OK, now that we have a deck of cards, let’s see how we can generate all possible 5 card hands. This is probably the crux of the puzzle & it is a little bit tricky: ordering is not important with a hand of cards & in addition to this, we’re “sampling with out replacement”; a card can only appear in a hand once.

First step is to decide on a method name, I’m going with GetAllHands.

Next we should nail down the interface (I’ve opted for a recursive solution). Remember our deck of cards will be represented as a sequence of integers, so our method will need to accept an IEnumerable<int>. Likewise, each hand can also be represented in this manner (as a sequence of integers) except this method will generate all possible (many) hands, so we can return an IEnumerable<IEnumerable<int>>.

Method Signature

IEnumerable<IEnumerable<int>> GetAllHands(IEnumerable<int> deck)

The only thing I’d like to add here, is a parameter that tells our function the size of the hand. This is important as my recursive approach will need this and we might want to generate all possible 2 card hands later (Texas hold’em).

IEnumerable<IEnumerable<int>> GetAllHands(IEnumerable<int> deck, int handSize)

With these types of problems, I generally find it easiest to work on the non-recursive (first level of depth) solution first. Imagine we just need to return all possible 1 card hands.

Generate All One Card Hands (http://share.linqpad.net/ebh299.linq)

void Main()
{
    var deck = Enumerable.Range(0,52);
    GetAllHands(deck,1).Dump();
}

IEnumerable<IEnumerable<int>> GetAllHands(IEnumerable<int> deck, int handSize)
{
    foreach(var card in deck)
    {
        if(handSize == 1) yield return new[]{card};
    }
}

Easy right? OK, it’s now time to add a sprinkling of recursion! For two card hands, we just need to call our function again, and concatenate the hands together.

Adding Recursion (http://share.linqpad.net/fii9wv.linq)

IEnumerable<IEnumerable<int>> GetAllHands(IEnumerable<int> deck, int handSize)
{
    foreach(var card in deck)
    {
        if(handSize == 1) yield return new[]{card};
        else foreach(var hand in GetAllHands(deck, handSize-1))
            yield return new[]{card}.Concat(hand);
    }
}

Looking at our output, there are two problems.

  1. We are producing the same hand in different orders: {1,0} & {0,1}
  2. We are sampling with replacement, we can’t have the same card in the hand twice: {0, 0}

To solve this problem, we just need to remove a card from the deck once we’ve generated all the hands it’s involved in.

Killing Two Birds With One Stone (http://share.linqpad.net/futqdc.linq)

IEnumerable<IEnumerable<int>> GetAllHands(IEnumerable<int> deck, int handSize)
{
    var count = 0;
    foreach(var card in deck)
    {
        if(handSize == 1) yield return new[]{card};
        else foreach(var hand in GetAllHands(deck.Skip(++count), handSize-1))
            yield return new[]{card}.Concat(hand);
    }
}

OK, so now this is where this solution really comes into its own. Not only can we compute how many 5 card hands there are, we have a sequence of all 2598960 hands meaning we can query it with LINQ!

If we think back to the puzzle that started all this, we’re interested in finding all the hands that contain at least one card from each suit. We can filter out any hand that doesn’t meet this criteria with a simple where clause!

The Solution (http://share.linqpad.net/u9q7aq.linq)

from hand in GetAllHands(deck, 5)
let suits = hand.Select(x => (Suit)(x / 13))
where suits.Contains(Suit.Hearts)
where suits.Contains(Suit.Clubs)
where suits.Contains(Suit.Diamonds)
where suits.Contains(Suit.Spades)
select hand

What’s interesting is how easy it is to adapt this to other poker hands. Here are some examples:

All Flushes (http://share.linqpad.net/349fer.linq)

from hand in GetAllHands(deck, 5)
let suits = hand.Select(x => (Suit)(x / 13))
where suits.All(x => x == Suit.Hearts)
    || suits.All(x => x == Suit.Clubs)
    || suits.All(x => x == Suit.Diamonds)
    || suits.All(x => x == Suit.Spades)
select hand

PS. When verifying this answer I found some poker websites incorrectly compute this one! They should have used LINQPad.

Three Fours (http://share.linqpad.net/n63fsf.linq)

from hand in GetAllHands(deck, 5)
let cards = hand.Select(x => (Card)(x % 13))
where cards.Count(x => x == Card.Four) == 3
select hand

Pretty cool huh?! I think that just about wraps it up: Another puzzle solved in C#!

Stay tuned for an interesting alternative solution.

Record Types & Pattern-Matching Coming in C# 6?

A developer from the Roslyn (.NET Compiler Platform) team recently published a *draft* specification for Records & Pattern-Matching in C#. It seems that the proposed language specification is an attempt to neatly integrate these two concepts (borrowed from F#, Haskell and friends) into the C# language. For some background see:

Pattern Matching in F#
Records in F#
Pattern Matching in Haskell

Record Types

Similar to records in F#, the proposed record type (referred to as a record class) is a new type of class that makes it easier to define & maintain read-only (immutable by default) POCOs, with the compiler handling generation of:

  1. Private Backing Fields
  2. Properties (field accessors)
  3. Equals (override equality operator)
  4. Is (required by the new pattern-matching operator)
  5. GetHashCode
  6. ToString

As an example, a record type defined as follows:

public record class Student(int age: Age, string name: Name);

Would instruct the compiler to generate the following:

public class Student 
{ 
    private readonly int _age;
    private readonly string _name; 
    
    public Student(int age, string name) 
    {
        _age = age;
        _name = name;
    }
    
    public int Age { get { return _age; } }
    public string Name { get { return _name; } } 

    // new "is" operator required for "pattern-matching"
    public static bool operator is( Student student, out int age, out string name) 
    {
        age = student.Age;
        name = student.Name;
        return true; 
    } 

    public override bool Equals(object obj)
    {    
        var o = obj as Student;
        return !ReferenceEquals(o, null)
            && object.Equals(_age, o.Age) 
            && object.Equals(_name, o.Name); 
    } 

    public override int GetHashCode() 
    {    
        int v = 1203787; 
        v = (v * 28341) + Age?.GetHashCode().GetValueOrDefault(); 
        v = (v * 28341) + Name?.GetHashCode().GetValueOrDefault();
        return v;
    } 

    public override string ToString() 
    {
        return new System.Text.StringBuilder()
            .Append("Student(Age: ")
            .Append(_age)
            .Append(", Name: ")
            .Append(_name)
            .Append(")")
            .ToString(); 
    } 
} 

While all of this code generation and syntactic sugar will no doubt make our lives easier, the really interesting thing here is the new “is” operator. From what I’ve learnt, the proposal states that any type implementing this operator (not just record types) will be compatible with pattern-matching expressions and overriding this with user-defined code will allow the programmer to extend the pattern-matching capabilities.

Patterns

This part of the proposed feature will allow developers to control program flow, using familiar if-is or switch statements, by expressing the “shape” data must be matched to including:

  • Types
  • Constants
  • Variables
  • Wildcards
  • Recursive Patterns

As an example, consider how we’d currently control program flow based on the type of a variable.

var button = control as Button;
if(button != null) button.Click();

Using a “Type Pattern” we’d be able to simplify this:

if(control is Button button) button.Click();

While this is a fairly trivial example, complex pattern matching, including switch statements & recursive patterns may revolutionize the way we write some C# programs. I understand the team is planning to publish a prototype “in a few weeks”, so I’ll post some real world examples of how the different types of patterns might be used.

In the meantime, if you’re interested in learning more, the draft specification is available here.

Posted in Uncategorized. Tags: , . 1 Comment »

Solving Puzzles with C#: Coins in a Bag

This is the first post in my new blog series focusing on solving puzzles in C# & LINQPad!

In the team I work with, it has become common practice for people to taunt their fellow developers by proposing interesting and sometimes mind bending puzzles. Aside from being a bit of fun, this often results in some interesting snippets of code and I thought it would be a shame not to share this with the wider community.

Coins in a Bag

(courtesy of Mitch Wheat)

“Two indistinguishable coins are placed in a black bag. One coin is biased towards heads – it comes up heads with probability 0.6

The other coin is biased towards tails – it comes up heads with probability 0.4. For both coins, the outcomes of successive flips are independent.

You select a coin at random from the bag and flip it 5 times. It comes up heads 3 times – what is the probability that it was the coin that is biased towards tails?”

LINQPad Solution

Although it is likely that the author of the puzzle intended us to use real maths, this problem is remarkably easy to simulate with a program.

Firstly, we need to write two functions that simulates flipping the coins (you can do this with one function, however this is easier to follow).

static Random r = new Random();

// Flip the heads weighted coin
public static string Heads()
{
    return r.Next(1000) >= 400 ? "Heads" : "Tails";
}

// Flip the tails weighted coin
public static string Tails()
{
    return r.Next(1000) >= 400 ?  "Tails" : "Heads";
}

OK, now that we’ve modelled the two different coins, we can write a LINQ query that simulates the exact scenario outlined in the puzzle definition.

    var query = 
        // run the test 1 million times
        from e in Enumerable.Range(1,1000000) 
        // take a coin from the bag
        let isHeads = r.Next() % 2 == 0
        // flip the coin 5 times
        let results = isHeads 
            // calling our weighted to heads function
            ? Enumerable.Range(0,5).Select(_ => Heads()).ToArray()
            // or call our weighted to tails function
            : Enumerable.Range(0,5).Select(_ => Tails()).ToArray()
        // we're only results where we managed to get 3x heads in a row
        where results.Count(x => x == "Heads") == 3
        select isHeads;

If you’d like to play with this, I’ve uploaded the solution to "LINQPad Instant Share".

Enjoy!

Asynchronous Commands in Metro, WPF & Silverlight

I’ve seen quite a few examples demonstrating the new async/await language features (C# 5 & VB next) with button click events;

private async void button1_Click(object sender, RoutedEventArgs e)
{
    string url = "http://reedcopsey.com";
    string content = await new WebClient().DownloadStringTaskAsync(url);
    this.textBox1.Text = string.Format("Page {0} supports XHTML 1.0: {1}",
      url, content.Contains("XHTML 1.0"));
}

If you are using an architectural pattern like MVVM it’s unlikely that you’re writing code like this. In WPF, Silverlight & Metro you can bind buttons directly to an object implementing the ICommand interface.

// WPF ICommand interface
public interface ICommand
{
    /// <summary>
    /// Defines the method to be called when the command is invoked.
    /// </summary>
    /// <param name="parameter">Data used by the command.
    /// If the command does not require data to be passed, this object can be set to null.</param>
    void Execute(object parameter);

    /// <summary>
    /// Defines the method that determines whether the command can execute in its current state.
    /// </summary>
    /// 
    /// <returns>
    /// true if this command can be executed; otherwise, false.
    /// </returns>
    /// <param name="parameter">Data used by the command.
    /// If the command does not require data to be passed, this object can be set to null.</param>
    bool CanExecute(object parameter);

    /// <summary>
    /// Occurs when changes occur that affect whether or not the command should execute.
    /// </summary>
    event EventHandler CanExecuteChanged;
}

The nice thing about commands vs. a simple click event is that they encapsulate the logic informing the button wether or not it can be executed. This is particularly useful when we start talking about asynchronous operations as we might like to disable the button while the asynchronous request is in flight.

Example

    public bool CanExecute(object parameter)
    {
        return !isExecuting;
    }

    public async void Execute(object parameter)
    {
        isExecuting = true;
        OnCanExecuteChanged();
        try
        {
            // await some asynchronous operation
        }
        finally
        {
            isExecuting = false;
            OnCanExecuteChanged();
        }
    }

What About Errors?

Note that commands are generally executed by the UI frameworks message loop, meaning that any unhandled exceptions will be posted onto the relevant synchronisation context.

AsyncCommand

This pattern is easily captured in a reusable object that we can use to build all our asynchronous commands.

    // a reusable asynchronous command
    public class AsyncCommand : ICommand
    {
        private readonly Func<Task> execute;
        private readonly Func<bool> canExecute;
        private bool isExecuting;

        public AsyncCommand(Func<Task> execute) : this(execute, () => true) { }

        public AsyncCommand(Func<Task> execute, Func<bool> canExecute)
        {
            this.execute = execute;
            this.canExecute = canExecute;
        }

        public bool CanExecute(object parameter)
        {
            // if the command is not executing, execute the users' can execute logic
            return !isExecuting && canExecute();
        }

        public event EventHandler CanExecuteChanged;

        public async void Execute(object parameter)
        {
            // tell the button that we're now executing...
            isExecuting = true;
            OnCanExecuteChanged();
            try
            {
                // execute user code
                await execute();
            }
            finally
            {
                // tell the button we're done
                isExecuting = false;
                OnCanExecuteChanged();
            }
        }

        protected virtual void OnCanExecuteChanged()
        {
            if (CanExecuteChanged != null) CanExecuteChanged(this, new EventArgs());
        }
    }

Usage

In your view model you can now create asynchronous commands like this;

// example command, simulate an operation that takes 2 seconds.
new AsyncCommand(() => TaskEx.Delay(2000));

// example command, with some custom can execute logic
new AsyncCommand(() => TaskEx.Delay(2000), () => IsValidInput());

 

Memory Leaks

A word of warning… If your command object’s lifetime extends beyond that of the UI element (Button) that is subscribing to the CanExecuted event you should implement a weak event pattern in here. I think that is outside the scope of this article. I’ll follow up shortly.

In Conclusion

This is a great example of why async void methods are required in C#. Commands are like a bridge between synchronous UI elements like buttons and your view models asynchronous operations like web requests. Enjoy!

Does my cluster have quorum

I’m finally returning to my blog after a 2 month hiatus. In a previous series of posts, I was looking into various heart beating patterns in Rx. I generally don’t like “multi part blog series” as there is a pressure to finish them off and I have a tendency to get distracted.

1. http://enumeratethis.com/2010/10/19/heart-beats-keep-alives-rx/
2. http://enumeratethis.com/2010/10/21/refining-the-heat-beat-timeout/
3. http://enumeratethis.com/2010/10/22/monitoring-cluster-nodes-in-rx/
4. Does my cluster have quorum. (You’re looking at it)

Last time we ended up with a query that tells us which nodes in a cluster are currently sending us heart beats;

 

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace ConsoleApplication1
{
    class Program
    {
        static void Main()
        {
            //..

            // query tells us which nodes are up or down.
            var query =
                from nodeId in source
                group nodeId by nodeId into grp
                from isConnected in IsConnected(grp, timeout)
                select new { nodeId = grp.Key, isConnected };

            //...
        }

        // takes a stream of heart beats & a timeout period.
        // transforms into an observable bool that tells us if we are connected or not.
        static IObservable<bool> IsConnected<T>(IObservable<T> heartbeats, TimeSpan timeout)
        {
            var connected = Observable.Return(true);
            var disconnected = Observable.Return(false).Delay(timeout);

            return Observable.Switch
            (
                from hb in heartbeats
                select connected.Concat(disconnected)
            ).DistinctUntilChanged();
        }
    }
}

 

 

However in my case, what I’m actually interested in is the clusters “quorum”;

Quorum – “The minimal number of officers and members of a committee or organization, usually a majority, who must be present for valid transaction of business.”

Let us assume that quorum >= 1/2 the nodes in the cluster. Really, we just need a reactive “counter”, that is incremented & decremented as nodes go on & offline. Scan is perfect for reactive counters.

query.Scan(0, (a, v) => v.isConnected ? a + 1 : a - 1)

 

We can then transform this into a reactive Boolean telling us if the cluster has quorum or not.

var hasQuorum =
    from count in query.Scan(0, (a, v) => v.isConnected ? a + 1 : a - 1)
    select count >= 2;

I hope you find this useful. Here is a complete sample program that covers everything discussed in this series. As a TODO I’d like to revisit all of this with the new join/group join/window operators in Rx, it’s possible they could be applied here.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace ConsoleApplication1
{
    class Program
    {
        static void Main()
        {
            // timeout period
            var timeout = TimeSpan.FromSeconds(1);
            // test source
            var source = new Subject<string>();

            // query tells us which nodes are up or down.
            var query =
                from nodeId in source
                group nodeId by nodeId into grp
                from isConnected in IsConnected(grp, timeout)
                select new { nodeId = grp.Key, isConnected };

            query.Subscribe(Console.WriteLine);

            // query tells us if the cluster has quorum.
            var hasQuorum =
                from count in query.Scan(0, (a, v) => v.isConnected ? a + 1 : a - 1)
                select count >= 2;

            hasQuorum
                .DistinctUntilChanged()
                .Subscribe(b => Console.WriteLine("has quorum: " + b));

            source.OnNext("martyn");
            source.OnNext("warne");
            source.OnNext("ponting");

            Console.ReadLine();
        }

        // takes a stream of heart beats & a timeout period.
        // transforms into an observable bool that tells us if we are connected or not.
        static IObservable<bool> IsConnected<T>(IObservable<T> heartbeats, TimeSpan timeout)
        {
            var connected = Observable.Return(true);
            var disconnected = Observable.Return(false).Delay(timeout);

            return Observable.Switch
            (
                from hb in heartbeats
                select connected.Concat(disconnected)
            ).DistinctUntilChanged();
        }
    }
}

 

Posted in .NET, C#, LINQ, Observable, Rx. Tags: , , . Leave a Comment »
Follow

Get every new post delivered to your Inbox.